|
|
Équipe Analyse/Synthèse |
A. Camurri and his group at University of Genoa have been developing research on gestures for more than 10 years, usually on the area of dance-music interfaces [3] and artificial intelligence applied to music [4]. His group presented several articles at the workshop. We will focus here mostly on the paper: Towards KANSEI information processing in Music/Dance interactive multimodal environments (page 74 of the proceedings). Abstracts of other papers can be found through the GDGM home-page.
In this paper the authors focus on movement and gesture analysis and machine communication to humans. In particular, on the first topic, the interest is mainly on recognition of non-symbolic, expressive data from human movement and gesture and their relation to music performance.
The first part of this paper presents the general framework of the work: human-computer communication in interactive Multimodal Environments (ME). The authors consider an ME as an active space populated by agents allowing one or more users to communicate by means of different modalities, such as full-body movement, gesture, voice. Users get feedback from the ME in real-time in terms of sound, music, visual media, lights control, and actuators (e.g. mobile scenography, on-stage robots). The interest in kansei information processing derives from the fact that high-level interaction requires, according to the authors, ME agents to be capable of changing their ``character'' and social interaction over time.
The article follows by discussing human gesture taxonomies, mainly by Coutaz [5] and Cadoz [6], kansei and movement analysis, where the authors are concerned about extracting high-level, whole body features, gesture gestalts, from the observation of a dancer or a performer.
Systems used for movement and gesture detection include camera based ones, where a preprocessing phase tries to recognise the posture of the human figure and match it to one of a few stereotypical postures or clusters, by extracting a small number of parameters and by sending them as inputs for a self-organizing neural network. The second phase consists of a possible segmentation on the stream of frames and application of different analysis algorithms to extract features from the different video segments. Other systems are the V-Scope, that uses ultrasound and infrared technologies, developed by Lipman Ltd. and DressWare, wearable piezo-resistive fabrics, developed by De Rossi et al. [7].
The data acquisition system used is called DanceWeb and is able to handle simultaneously up to 16 digital inputs and 40 ultrasonic sensors, configured in 8 groups of 5 sensors, each sensor individually enabled. Sample time can be set via software and varies between 15ms and 10 s. The system communicates via an RS 232 to Win32 compatible applications.
Finally, data processing is done by a software developed at DIST, called Mummia, an environment for the development and supervision of real-time, dance- and gesture-driven performances. Material provided by the composer is dynamically rearranged according to composer's rules and external inputs, typically dancers' movements. ``Mummia'' is based on two notions: virtual sensors and MIDI agents. A virtual sensor can be: a physical sensor or an elaboration of it (e.g., its derivative), or the fusion of data from different sensors. A MIDI agent is an agent able to produce a MIDI output according to its internal state, to continuous parameters and triggers that modify its behavior (typically, a single agent only controls a small portion of the global score/performance, for a given duration of time. Conversely, more than one agent can control a single musical parameter).
The second part of the paper deals with the possibilities to control the movement of physical objects, including robots and in general effectors. The authors discuss the robot control system and briefly the development of a robot's emotional component. These developments have been implemented in the project called ``La Cita dei Bambini'', a permanent exposition in Genova.
More information on Camurri's previous and related work can be found at [8], [3], [4], [9], [10] and [11].