Figure 1: The PACOM console
Marcelo M. Wanderley
Marc Battier, Philippe Depalle, Shlomo Dubnov,
Vincent Hayward, Francisco Iovino, Véronique Larcher,
Mikhail Malt, Patrice Pierrot, Joseph Butch Rovan, and Christophe Vergez
Ircam
1, Place Igor Stravinsky
75004 - Paris - France
Marcelo.Wanderley@ircam.fr
During the last few years there has been, in France and abroad, an apparent increase in the number of conferences/workshops devoted to human-computer interaction in music, specifically in the area of gestural control. Examples of these are: ``Les Nouveaux Gestes de la Musique - Colloque International'', held in Marseille [Wan97] and ``KANSEI - The Technology of Emotion Workshop'' [Cam97] [Wan98], held in Genova, Italy. Conversely, some 12 papers/posters/reports related to gestural capture/control have been presented in ICMC'97 in Thessaloniki, Greece 1 2.
Research in subjects related to gesture applied to music at the Institut de Recherche et Coordination Acoustique/Musique - Ircam - Paris, extends over the last decades. Different experiments have been carried out: real-time control of sound synthesis with the 4X machine [GK81] via the PACOM gestural controller interface, developed by Starkier and Prévot [ST86]; and on the development of alternate musical controllers such as the "MIDI Flute" [Pou92] - by X. Chabot, M. Starkier and the late flutist L. Beauregard - that uses an instrumented acoustic flute to capture and transmit the key positions via MIDI. This flute was used in the piece "Explosante-fixe...", by P. Boulez.
This report intends to survey briefly some of the previous work on gestural capture/control at Ircam and focus on current activities related to this area. For the sake of clarity, we will divide it into four parts: Previous work, Pedagogy, Research and the transversal Discussion Group on musical gestures. 3
Since the advent of real-time digital synthesis, researchers have been considering alternate means to control these machines - ways to express musical intention other than with computer keyboards or keys/buttons. In the late 70's, Max Mathews devised his sequential drum to control the first generation of digital synthesizers [MB78]. In the 80's, this question was again considered at Ircam, leading to the development of the PACOM system 4 5.
Figure 1: The PACOM consoleAs mentioned above, the MIDI Flute was developed during approximately the same period. 6
During the 90's, G. Dubost wrote a master's thesis (D.E.A.) on the theme: ``Sensor Technology and their Musical Applications'' [Dub93]. As a result of this work, a sonar type sensor was built. Used initially by Chabot [Cha93] and Evans, it has recently been used by composer S. Goto during public presentations as part of a setup for demonstrating some issues on gestural control. Among other researchers/composers, A. Tanaka and A. Boulard have also taken part in related projects, such as a children workshop in a previous ``Journées Portes Ouvertes''. Some simple sensors were used to draw children's attention to the question of interactivity.
Many activities 7 have been carried out by the Pedagogy Department, under the directorship of J.-B. Barrière and where Marie-Helène Serra is currently the acting director. These consist of support to composers interested in this area, courses proposed for different audiences and the coordination of trainees working closely in relation with the composers.
Young composers are accepted each year as part of the ``Cursus de Composition'', a one year course on computer music. During the last two years, three compositions have been created using different sensors and/or gestural interfaces: Suguru Goto(95/96) - ``Virtual AERI'', Lucia Ronchetti(96/97) - ``Eluvion-Etude'', and Roland Auzet (96/97) - ``OROC.PAT''.
These three pieces used different technologies for gesture/movement acquisition, such as ultrasound, FSR's (pressure sensors), accelerometers, among others. The signals output by the sensors are typically converted into MIDI messages by means of custom interfaces (S. Goto) or commercial units, such as the I-Cube or the Sensorlab.
M. Bibes and E. Flety developed an on-line sensor classification, where sensors were classified according to the need of physical contact and the resolution provided 8. The goal is to provide composers a tool for choosing the technology that best fits their specific needs. The classification encompasses information on nearly two dozen devices - sensor characteristics, description, technical data, price, and availability.
Also regarding the Pedagogy department, a regular course on sensors and gestural capture were given. The course deals with different subjects, such as interface design, sensor choice, interaction techniques, and so on. Lecturers include C. Cadoz, M. Waisvisz, and B. Rovan. Also, a specialized ten-week course on "Installations sonores" was given, to present artists the technology necessary to design interactive installations, including real-time gestural control of graphical media such as Quick Time video, etc.
As for hardware developments/products related to the activities of Pedagogy/Production department, P. Pierrot and A. Terrier carried on the development of different gestural interfaces. In cooperation with T. Coduys, a vest (gilet, in French) was built where 5 pressure sensors (FSR's) used as switches were employed, as well as 2 mini-keyboards, each with 2 keys. These 9 sensors provide signals to a small MIDI interface powered by a 9V battery. Currently, the sensor signal output is provided by a cable, but a radio link is being considered. The piece ``Tapottages'' by M. Monnet performed by percussionist J.-P. Drouet featuring this device was performed in January 1997 at Centre Georges Pompidou, in Paris.
Another gestural interface was developed by Pierrot and Terrier, this time for composer/performer S. Goto [PT97] - the SuperPolm violin controller, used in his piece ``Virtual AERI''. Seven sensors integrated in a violin-like controller provide 11 continuous output variables.
Associated to the sensors is an 8-key keyboard that allows the direct choice of MIDI messages. The ensemble thus provides an interface that is able to control different synthesis parameters departing from a gesticulation similar to that of a violin player. Pushing further in this direction, the same group has designed another interface to take account of body gestures related to a violin player, more specifically, a diving suit containing 12 flexion sensors - 6 for the arms and 6 for the legs. The idea is to translate the violinist's movements into electrical signals.
Finally, P. Pierrot is currently working on a MIDI interface that handles up to 32 inputs from piezoelectric sensors placed on cow-bells. This is a project by composer James Woods for a production at Ircam, with musical assistantship by Carl H. Faia.
Many projects focusing on or related to gestural control are currently being developed in the Research Department, headed by Hugues Vinet. These projects relate to different synthesis methods - physical and signal models, haptic devices, interface design, among others.
One of these is the research led by Ch. Vergez and X. Rodet on the the real-time control of a physical model of a trumpet [RV96], [VR97], [VR97b]. The control interface was developed using a MIDI wind controller (Yamaha WX7) for the information concerning mouth's pression and piston position. The lips resonance frequency in the model is controlled by a MIDI pedal or by the reed information from the WX7. Another pedal is used to modify the value of the viscous damping in the lip model. This interface permits the control of the model within a short learning period and the resulting interaction between the player and the model closely simulates the one experienced by a performer and his instrument.
In the same direction, the control of a physical model of a sliding clarinet pipe has been developed by G. Peeters and X. Rodet. A common MIDI keyboard has been used to control the different parameters of the model, such as pitch-bender controlling the length of the sliding pipe, and velocity and aftertouch controlling the parameters of the non-linear exciter (acting like the mouthpiece).
Research on Haptic devices applied to computer music has been started at Ircam from last July by V. Hayward, a professor from McGill University - Canada on a sabbatical leave.
Haptic devices [Hay98], [HA96], [Hay95], [HCLR94] [RH94] are devices that can achieve computer controlled stimulation of the tactile and kinesthesic sense. One motivation for their use in music is the observation that the production of gestures by humans (motor control) is linked in many ways and at various levels with mechanical perception. There is a lot of work on the control of production of computer generated music via gestures, but for the most part, the result of these gestures is acoustic (or visual) and deprived from mechanical feedback 9. Hence, the availability of computer generated mechanical feedback affords the possibility to couple the human with the machine in even more meaningful ways.
One project under way is the development of an environment whereby gestures are recorded and mechanically "re-presented" to a musician. The musician is then in a position to experience, modify, update, or otherwise "edit" his own gestures or those of someone else. A gesture then exists as an "object" to control a music generating process, either in real-time or offline. 10
Another project consists of using simple tactile stimulators to assist musicians in the use the existing gestural controllers, which, as far as we can tell, are all deprived of programmable mechanical feedback. A component of this project is the robust extraction of key events in a gesture such as reversals, stops, whips, etc. from noisy signals.
Gestural control of the physical modelling environment ``Modalys'' is another research axis, led by researchers R. Caussé, N. Misdariis and programmer F. Iovino [ICD97]. Physical modeling methods impose a direct relationship between the performer gesture, the instrument and the sound result; in Modalys, the instrument is an assembly of basic physical structures (strings, plates, membranes, fingers, hammers, ...) that have been selected and coupled with different types of interaction (bow, breath, adhere, ...) by the user himself. It is thus logical to use gestural data to send it to the connection inputs that couple the components of the assembly (bow pressure, bow position, finger position, ....)
At present time, the Modalys team is experimenting two concrete situations where gestural control is called for: First, in a real-time use of Modalys, the gestural data is mapped (via a linear convergence) to the connection input. For non real-time applications, the team is interested in capturing a gesture for further edition and/or transformation before being linked with the connection input, via break-point-functions or midi files.
Within the context of Ircam's Spatialisateur project [Jot97], a commercial 3D mouse 11 was interfaced with the O2 12 version of FTS, the real time platform on which the software Spat was developed. This communication occurs through a serial port of the computer and allows position coordinates (x, y and z) and orientation coordinates (yaw, pitch and roll) of the 3D mouse to be transmitted. The serial object was developed in C language as an FTS module by L. Ghys, after another object conceived by W. Ritsch and available via ftp 13.
The Spatialisateur uses this module in its binaural rendering mode, where a tri-dimensional sound scene is reproduced over headphones [Jot95]. By attaching the 3D mouse to the headphone set, the software can compensate for the rotations of the listener's head. This required the real-time implementation of conversion matrixes from (yaw,pitch,roll) to (azimuth,elevation) coordinates, which was achieved by V. Larcher. The fidelity of the binaural simulation is thereby noticeably increased, since headtracking reduces the probability of front-back confusions and makes the auditory scene independent of the listener's movements.
The development of a real-time synthesis system with applications to gestural control, based on j-MAX by N. Schnell, B. Rovan and M. Wanderley has been carried out from the end of 1997. The system is intended to provide a flexible environment for sound synthesis and a basis for experimentation in human-computer interaction in music [WSR98]. It is built as independent modules that may be replaced according to the synthesis method used (initially additive synthesis), and to the type (level) of interaction desired. Initial demonstrations have been done using MIDI controllers, such as the Yamaha WX7, where the mapping of controller's outputs to ESCHER input variables may be changed to produce different levels of interaction with the system - expert mode (acoustic instrument control), direct mode (normal one-to-one), etc. In addition, an infrared location sensor was used to track the movement of the WX7; the data generated was used for controlling the morphing parameter between different additive models.
The discussion on the use of sensors and gestural interfaces has been carried out by M. Wanderley and B. Rovan. Part of this work will be presented as a paper at the SEAMUS'98 conference, held at Dartmouth College, Hanover, N.H. [RW98]. B. Rovan has also developed a data-glove [DTW97] based on FSR's and an accelerometer, flexible enough to be used during an instrumental (saxophone/clarinet) performance, providing additional degrees of control to the performer.
Figure 5: The data-glove used while playing a clarinet. Photo by Myr Muratet.
A study of mapping strategies and the real-time control of additive synthesis using a timbral space built by the analysis of clarinet samples from the Studio-on-Line project [Fin96] has been developed by B. Rovan, M. Wanderley, S. Dubnov and P. Depalle[RWDD97] and was presented at the Kansei workshop [Cam97] in Genoa, Italy. The main object in this work was to study the influence of the choice of mapping between controller outputs and synthesis inputs as a determinant for expressivity in an additive synthesis environment, using off-the-shelf MIDI controllers.
Still regarding research topics, M. Wanderley with Ph. Depalle and X. Rodet has been developing research on gesture capture [DTW97], modelling and application to the control of sound synthesis, mainly signal models. Part of this research is based on the modelling of instrumental gestures and the development of a clarinet model taking into account the influence of non-trivial gestures such as performer's movements in the produced sound. It has been observed that, in the case of an acoustic clarinet, body movements may influence the produced sound due to radiation characteristics, as well as room response [WRDR98]. Research is being developed in order to devise a simplified model of amplitude variations of sound partials that may be useful as a means of providing another degree of expressivity to synthesis. Results of this work will be implemented as parts of the ESCHER architecture, as described above.
One other approach taken is the research on expressive synthesis using signal models. Post doctoral researcher S. Dubnov (now at the Hebrew University - Israel) has developed algorithms and signal analysis methods in order to obtain better quality sound synthesis (additive synthesis) by means of gestural control and different mapping strategies [RWDD97], [DR97].
The final goal of this study is to provide both quantitative and qualitative representation of the expressive variations in sound and thus define a suitable parameter model. This expressive model will be further used in conjunction with the input controller, so as to achieve a proper mapping between the human controls and the synthesis parameters.
A. Terrier, P. Pierrot and X. Rodet collaborated with on the development of JERRY, a four dimensional mouse - a normal computer mouse equipped with two pressure sensors, allowing the continuous control of 4 parameters [RTP97].
Several applications are envisaged for such device, including its use to control real-time sound synthesis (signal or physical models). For example, on the simulation of a violin bow, where X and Y positions from the normal mouse are used to simulate the bow position respective to the string, the first pressure sensor as the bow pressure and the second one as the amount of bow-hair coming into contact with the string. Other applications include its use as a replacement for a graphic tablet (for computer design) or for data processing (multi-dimensional databases).
S. Serafin is pursuing a master's thesis (DEA ATIAM) on real-time control of a physical model of the violin, under the supervision of X. Rodet. The use of input devices such as tablets and 3D mouses is envisaged for simulating the violinist gestures.
The Groupe de Discussion sur le Geste Musical (GDGM) was created in 1997 by composer B. Rovan, and researchers M. Wanderley and S. Dubnov. The idea behind the group is to discuss and exchange experience among researchers/composers/performers working in gesture related disciplines. Initially devised as an internal group, a home-page was created where information about the group's activities is available, as well as information on related events. The group's main activities, apart from the maintenance of the site, relate to the development of a bibliography on the subject and the organisation of meetings (usually once a month).
These meetings may be either organised as discussion meetings on a pre-determined theme or may consist of an invited lecture by a researcher in this field. Previous lectures (1997) have been presented by David Wessel (CNMAT), Axel Mulder (Infusion Systems) and Mark Goldstein (Buchla and Associates). In 1998 a lecture on Haptic devices was presented by Christophe Ramstein (Haptic Technologies, Canada) and another on building simulated environments, by Dinesh Pai (University of British Columbia, Canada); other lectures are planned.
From October 1997, the group expanded to an ``open'' group, and a second site was created to provide means for a general discussion on the subject. 14 It contains basic texts on several topics related to gestural capture/control followed by links to current work on gestures and music. This page is intended as a basic reference to other people's work in this area 15. Many researchers have shown interest in this idea and there are currently a dozen links to different laboratories and institutes.
Finally, in February a mailing list on gestures and music has been created 16, and we expect it can serve as a means for further developing the area.
Three invited presentations from members of the GDGM have been presented in 1997 - the first two were talks to the Journée d'Études held by the SFA, AES-France and CNSMP on november, 20th. S. Goto spoke about his projects at Ircam, followed by the second talk, where M. Wanderley and B. Rovan presented results of their research on gestural capture. A demonstration was made by Rovan with violinist B. Goris using alternate controllers and sensors placed on the violin to control real-time synthesis via analysis of different performance techniques. Rovan also demonstrated real-time control of sound processing on a clarinet, controlled by an infrared location sensor.
The third event consisted of a concert by Butch Rovan, preceeded by a presentation about Ircam and the activities of the GDGM by M. Wanderley. These events took place in Genoa, Italy, from an invitation of Antonio Camurri (DIST - University of Genova) and the Teatro Carlo Felice. The concert consisted of two pieces by B. Rovan - CONTINUITIES I for glove controller and interactive electronics, and CONTINUITIES II for clarinet, gestural controller, and interactive electronics - and where performed live by the composer during the presentation. This event was part of a series of concerts/presentations on new music and was organized by Camurri and the Carlo Felice Theater.
In this article we reviewed the past work developed in the area of gestural capture/control related to music at Ircam, and presented the current developments and researches on this field. The Groupe de Discussion sur le Geste Musical was introduced and its different activities presented.
The authors would like to acknowledge : N. Schnell, M.-H. Serra, H. Vinet, E. Flety, S. Goto, X. Rodet, G. Peeters, R. Caussé, N. Misdariis, J.-M. Jot, G. Bertrand, A. Terrier, M. Bibes, S. Serafin, A. Mihalic, T. Coduys, R. Woehrmann, J. Baboni-Schilingi, and all other composers, researchers and performers that collaborated on this project.
Thanks to Guillaume Dimanche for the photograph of the PACOM interface.
Gestural Research at IRCAM: A Progress Report
This document was generated using the LaTeX2HTML translator Version 96.1-f (May 31, 1996) Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
The command line arguments were:
latex2html -split 0 jim98_final.tex.
The translation was initiated by Marcelo Wanderley on Tue Apr 28 18:08:55 MET DST 1998
3D mouse and Head Tracker