Listening test

Evaluation of expressivity and singing style modelisation for singing voice synthesis
(Not active anymore) contact : luc.ardaillon@ircam.fr

Aller à la version française

Thank you for your time ! Please read carefully the following information, even though you are used to doing such tests !

This listening test aims at evaluating the expressivity and modelisation of singing styles using the pitch and phonemes durations, for singing voice synthesis.
We do not want to assess the overall quality of the synthesis or the used techniques for processing the waveforms.
However, the sounds below may contain artefacts which are not related to this modelisation (e.g. unatural timbre, distortions, ...).
Thus, please try to ignore these artefacts and focus only on the pitch variations and phonemes durations.

You will be presented 2 tests below. The first one aims at evaluating the expressivity (or liveliness, musicality) of singing voice synthesis using various settings.
The second one aims at evaluating the modelisation of the singing style.
Detailed explanations on the evaluation procedure are given below for each test.

Recommendations

If there is any technical problem (e.g. sound not playing) with one sound, select Prob. (please note that it may take a few seconds for all sounds to load)
If you perceive a difference of loudness such as you can't properly evaluate a pair of sounds, select also Prob.
Do the test in a quiet place.
Use absolutely headphones or earphones.
Verify that the sound level is loud enough to hear the sound details properly.
Please, take the time to listen !
No need to try to find a repetition or order among the sounds. There is none. Their order is randomized.

Test I

For each pair of recordings below (each line) select one button depending on your preference about the expressivity of the two interpretations.
By the term "expressive", we mean an interpretation that sounds lively, with musical intentions, as opposed to a more mechanical or static interpretation.

If the left sound is much better than the right one, select the most left button (+3)
If the left sound is better than the right one, select the second button (+2)
If the left sound is slightly better than the right one, select the third button (+1)
If you do not hear any difference or you have no preference, select the middle button (0)

... and the same on the other way.

There is no "correct" answer. It is only about your subjective preference.

Pair	File1	+3	+2	+1	0	+1	+2	+3	File2	Prob
1
Pair	File1	+3	+2	+1	0	+1	+2	+3	File2	Prob
2
Pair	File1	+3	+2	+1	0	+1	+2	+3	File2	Prob
3
Pair	File1	+3	+2	+1	0	+1	+2	+3	File2	Prob
4

Pair	File1	+3	+2	+1	0	+1	+2	+3	File2	Prob
5
Pair	File1	+3	+2	+1	0	+1	+2	+3	File2	Prob
6
Pair	File1	+3	+2	+1	0	+1	+2	+3	File2	Prob
7
Pair	File1	+3	+2	+1	0	+1	+2	+3	File2	Prob
8

Test II

For each line below, first listen to the 2 sounds in columns "style A" and "style B". Then listen to the 3^rd sound in column "style X", and select "A" or "B" according to wether you think that the expressivity and singing style is more similar to the sound in "style A" or "style B" column.
Please, note that many parameters related to the singing style, like intensity, timbre, tempo, and variations on the score itself (rhythm and notes) are not modelled in the synthesis and thus should not be considered in this evaluation.
The score used for the synthesis is "stylistically neutral" and differs from the interpretations of the original recordings in columns "style A" and "style B", which have different rhythms and pitches which shouldn't be considered in the evaluation.
Your choice should be based only on the relative pitch variations (attacks, transitions, vibrato, ...) and phonemes durations.