Previous Contents Next

2   Summary of Activities

The state in May 2002 was the following: The mathematical model for score following, that had been developed during the year 2001 at Ircam, had been implemented in two objects for jMax, version 2.5.3: suiviaudio and suivimidi. Both objects rely on the sequence editor, an object for the representation of musical events, which is used to create the internal representation of the score to be followed (this is called score parsing). For each object, a number of examples and help patches had been created.

Even if the model can deal with monophonic and polyphonic scores, insofar the two objects were able to follow performances only with single notes, because the score parser still had to be completely debugged and extended to handle polyphonic scores.

A number of tests with both audio and MIDI input have been carried out; they are described in section 2.1. The tested pieces were:

Section 2.2 reports the state of software development of the suivi package. The two internships (stages) carried out are described in section 2.3. Finally, section 2.4 lists the chronology of past dates around score following, i.e. the visits of Nicola Orio, the tests, meetings, and releases.

To aid communication, the mailing list score-recognition1 has been revived and attracted more subscribers (over 30 at the end of 2002), and regular progress meetings have been installed to keep anyone interested in score following up-to-date of the current state of the project.

2.1   Tests

The tests were done beginning of June and July 2002 with version 1.0 of the score follower.

The test data collected during these tests and recordings, and also before, is used to evaluate and train suiviaudio and suivimidi. It is centralised on the internally accessible file server of the ATR team lichtung. A list of the audio and Midi performances and Midi scores available is given in appendix B.

During the tests, the evaluation method described in section 7.1 helped in quantifying the performance of the follower.

2.1.1   En Écho Audio

The object suiviaudio has been tested in a real environment on July 3. Tests have been carried out live with the singer Donatienne Michel--Dansac on four parts of En Echo. Different takes of each part have been recorded during the tests, in order to create a database of performances to be used for the training of the model parameters. In particular, the score follower has been tested on difficult passages, the ones that usually gave problems with previous score followers --- namely, Explode and F9.

The score follower succeeded in finding the correct position on the score for most parts of the performances, with only local mismatches that never affected the overall following (in other words, the follower never got lost). These local mismatches normally happen when the same note is repeated, this behavior is already known and a novel approach to overcome them has already been found and implemented (see section 4.2.1). It should be remarked that the score follower behavior is fairly good also with passages that usually gave problems with previous followers.

The singer was also asked to sing out of tune, mostly singing wrong notes. The global alignment showed to be robust to these errors, even if this behavior heavily depends on the number of errors. Due the small amount of tests on wrong performances, these results are encouraging but clearly not definitive.

At the end of the session, a number of additional passages have been recorded. Passages included: changes in the sung vowel with a stable pitch, fricatives, and liquid consonants followed by a vowel. These recordings will be used to create a statistics of the sung voice in order to improve the performance of the score follower and avoid local mismatches.

2.1.2   Explosante--fixe Audio

The objectsuiviaudiohas also been tested with recorded flute performances of Explosante--fixe by Pierre Boulez, played by flutist Emmanuelle Ophèle and supervised by Andrew Gerzso on June 10. Only a part of the piece has been tested, due to the lack of a suitable electronic version of the score. It has to be noted that the lack of a musical score tends to be a major problem: The follower needs to know the score as played by the musician, but most of the available scores are edited in order to improve the performances of previous versions of score followers, resulting in big differences between what is represented and what is actually played.

In any case, results with these excerpts are very encouraging. Also in this case the score follower succeeded to find the correct position on the score, with local mismatches affecting only few notes. It is interesting to note that the excerpt contained many passages with repeated notes, and some notes were played using the flutter-tongue technique, which were all correctly recognized.

These tests have been made during the preparation of a performance of the piece in Lyon on June 15. (The preparations were necessary to do the necessary adjustments to get the old score follower Explode on the NeXT working correctly.) It was arranged that the flute was recorded in the concert, which makes for realistic test data with the orchestra and electronic part bleeding thru to the flute microphone used for following.

2.1.3   Explosante--fixe Midi

The tests of Explosante--fixe with the output of the Midi flute revealed the inappropriateness of the Midi data: even after a long setup and calibration phase, the flutter-tongue notes gave rise to a hail of notes, some with zero-length. Because of other inherent inconveniences of the Midi flute (capturing of the note-on by a microphone, resulting in wrong triggers from noises of the orchestra, breathing of the flutist, or turning pages of the score; capturing of the Midi-note by the keys, which means that no octave information is transmitted; and, last but not least, the excess weight of the captors and the manipulations that the flute has to support), and because of the successful experience with the audio-follower, it was decided to abandon Midi-following for Explosante--fixe.

However, the Midi version of the follower was tested, just for debugging purposes, using one sequence from the recorded Midi-data. The sequence was used both as the reference score and as the performance, and the following was perfect (every note was recognised immediately, except one, which had a delay of 7 ms!).

2.1.4   Pluton Midi

The tests of Philippe Manoury's piece Pluton for Midi piano, played by Manoury, on July 2, revealed problems with suivimidi: First of all, there is an ambiguity with long sustained notes, that can be played using the sustain pedal (or the third, selective, sustain pedal) or by holding down the keys, according to the performer. In principle, this can be normalised at the Midi input stage, but Serge LeMouton made the remark that in most of the Midi pianos, the foot pedals are not midified (even if they could), so that there is no way to know the actual end of the notes! Another possible problem is the use of silent notes in Pluton, that serve to open strings to resonante. These notes still generate note-on/note-off Midi messages but are probably played less precisely in time, and should therefore be not actually followed. The last problem shown in the test was that high polyphony was actually not yet handled by suivimidi. This has been remedied now (see section 6.2).

The tests were recorded on a Midi sequencer, and we now have two executions of section 1 and one of section 2 available. The tests with the professional pianist Andy Russo scheduled for January 23, 2003 will add more test data.

During the tests Philippe Manoury remarked that while composing for score following it is often necessary to play a section, pause to make some adjustments, and then resume playing --- all this with the score follower following. It has to be tested if the follower accomodates this automatically. If it doesn't, a special ``composer mode'' could be envisaged, where long pauses are possible without disturbing the follower.

Also, during rehearsal, often the tempo is slowed down considerably, so the follower must not be disturbed, which might give rise to a global speed control input being necessary.

2.2   Development

The development between May 2002 up to July is gathered in a first internal release of the package suivi with version number 1.0 from July 5, 2002. The releases are available on the shared document space of the mailing list2, together with a release notes document (given in appendix A.1).

The next milestone was version 1.1, from September 18 (the release notes are given in appendix A.2), which incorporated the reorganised handling of score and performance for suivimidi and suiviaudio, and allows setting the parameters of the probability density functions. These parameters are calculated off-line on example recordings by a Matlab program, developed between September and November (see section 5).

Version 1.2 from December 20, 2002 is the first ``production'' release (see notes in appendix A.3). It eradicates the last bugs in the Midi and audio input, includes improvements for voice following explained in section 4, and allows truly polyphonic Midi following.

Version 1.2 will be the last release under jMax-2.5. The port of the suivi package to jMax-4 is in progress and due to be finished end of January 2003. Changes are mainly due to a new organisation of jMax's sequence editor internal note and event representation. It allows a better and more flexible graphical presentation.

2.3   Internships (Stages)

Two stagiaires were working on score following during the year under the supervision of Diemo Schwarz and François Déchelle: Gilles Mathieu and Vincent Goudard, both from the electrical engineering and computer science school INSA, Lyon.

2.3.1   Gilles Mathieu

Gilles Mathieu mainly worked on the extension of the score parsing to true polyphony between Mai 5 and August 23. His report is [Mat02], from which we quote:

Premiers objectifs: Les premiers objectifs, définis avec François Déchelle, sont les suivants :

Objectifs "revus": Au cours de ma période de stage, en fonction des différentes attentes vis-à-vis du suivi et des nouveaux problèmes soulevés, les objectifs initiaux ont étés revus. Il s'est ajouté:

Les objectifs plus liés au suivi audio sont peu à peu sortis de mon champ d'action pour entrer dans les objectifs principaux du stage de Vincent Goudard.


2.3.2   Vincent Goudard

Vincent Goudard's stage took place from June 24 to September 20, and is described in [Gou02] as follows:

Le système du suivi de partition était déjà lancé au moment où je commençais mon stage; les objectifs de mon stage sont donc liés au buts finaux, ainsi qu'à l'état du projet en cours; le but final étant de pouvoir suivre une performance musciale jouée par un (plusieurs) musicien(s), grâce à une représentation de la partition jouée.

Il en découlait plusieurs objectifs, relatifs à l'état d'alors:

The two internships yielded very satisfactory results: advancments in the software architecture and the documentation, helper objects convenient for the users, and analysis objects that lay the ground for advancing research. All the objects come with complete documentation (which can be reused for the Forum release of the software) and help patches. Some interesting paths of heuristics for better Midi-following were explored although they were later abandoned to keep closer to the mathematical model.

2.4   Journal of Past Events

2002
Mai -- August Stage Gilles Mathieu
7 Mai Start-up meeting score following
15 Mai Meeting Midi score following
10 June Recording session Explosante--fixe
11 June Meeting score following
27 June Meeting score following
June -- September Stage Vincent Goudard
1--5 July Work visit of Nicola Orio, meetings
2 July Recording session Pluton Midi piano
3 July Recording session En Echo voice
5 July Release of suivi version 1.0
29 August Meeting score following
29 August Release of suivi version 1.1 beta
3 September Meeting ``Réorganisation Temps Réel''
18 September Release of suivi version 1.1
14--20 October Work visit of Nicola Orio
19--20 October Open Doors at Résonances, presentation of score following by Diemo Schwarz
20 October Weekend New Instruments at Résonances, talk and discussion by Nicola Orio
7 November Progress meeting 1
28 November Progress meeting 2
4 December Meeting ``Objectifs de Développement''
20 December Release of suivi version 1.2

Previous Contents Next