In this thesis we have proposed an integrated environment for music composition and synthesis. Our system integrates a high-level Scheme interpreter, a real-time synthesizer, and a rich framework for temporal composition and sound synthesis. These are some of the reasons why we found it necessary to develop such an environment:

The integration of timbre and the control of sound synthesis into a music composition: In our environment the description and control of the sound synthesis becomes an integral part of the composition. There is no longer a need for two programs: one for the composition and one for the synthesis. This integration allows the manipulation of timbre as one of the musical elements in a piece.
A rich time model: Most synthesizers have a very limited notion of time - in the general case, they only know the current time. In our environment, however, the synthesis processes and controllers have the full knowledge of the time context of the composition. This knowledge is necessary if we want time manipulations, such as a stretch, to behave correctly in all situations.
Real-time playback: We want music pieces to be played in real-time from within the composition environment. This should not only make the work process more efficient, but should also allow instant feedback and control of the composition.
A fully dynamic environment: Most synthesizers proceed in two steps: programming the synthesis patch and executing it. We want a fully dynamic environment in which the composer can act upon the synthesis during the execution. This runtime programming is important to make adjustments to a composition that can only be verified during runtime. Interactive pieces can use this feature to manipulate the piece being played following user actions.
An open system: We cannot impose any musical style upon the composer. Instead, we want a highly customizable environment. The system should offer a rich set of structures for composition and synthesis, combined with a high-level programming environment to express the musical ideas of the composer.
Complex interactivity: The interactivity should not be reduced to the sequencing of pre-existing parts. Instead, it should be possible to generate the contents and structure of a piece on the fly according to a high-level description. In an interactive piece the user should be able to manipulate the structure of the piece using high-level tools. In addition, the use of events should not be limited to the setting of a control value. Indeed, events can provoke a complex response on the side of the system.

The environment we proposed in the text is entirely written in the Java programming language. In addition, it uses an embedded Scheme interpreter. This approach furnishes us with a dynamic programming environment that can be personalized and extended. The Scheme interpreter is written in Java, and the Scheme objects are implemented as Java objects. This allows a transparent use of Scheme primitives in the environment. In particular, we will use the Scheme procedures to describe the complex behavior of the system. In the text we called these functional objects programs. Programs are used in two basic elements of the environment:

Events: they consist of an evaluation time, a program, and the arguments for the program. At the specified time, the program is evaluated with its arguments.
Activities: they are the basic element of temporal composition and represent any activity in time. They consist of a location in time and a duration. How the activity is to be started is described by a start program; how it is to be stopped by a stop program.

In the text we also discussed the basic elements of sound synthesis:

Synthesis processes: they are the basic signal processing unit.
Control functions: they describe continuous varying parameter values in time. They describe, for example, the amplitude envelop of a sound. We defined control functions as continuous, multi-dimensional, reentrant time functions.

Synthesis processes can be created using a meta-class approach: synthesis techniques create new synthesis voices, synthesis voices create new synthesis processes. We have discussed additional structures to organize activities in a musical piece:

Patterns: they are basic element of organization. A pattern groups a set of activities and the time relations between them. The pattern is also in charge of the reorganization of the activities when modifications are made. These changes may trickle down the musical structure and provoke an incremental update. This approach is inspired on the graphical layout managers found in the Java Abstract Window Toolkit. It can be extended to the technique of constraint propagation found in graphical interfaces.
Motifs: they can be considered as a handle to a pattern. They allow a pattern to be inserted at several occasions in a music piece.

Additionally, pattern modifiers can be used to make variations of a pattern or to extend the organization of a pattern to other elements than time. When activities of unknown duration are used, causal relations can be used to express their temporal organization. These operators reorganize the start and stop programs of activities to assure a correct sequencing.

The environment also offers a rich, layered time model. This time model associates a local time axis to each level of the hierarchy in which a musical piece is organized. Additionally we have the following two objects:

Time models: they map the time axis of one level of the hierarchy onto the axis of the next level. They can be used to deform time.
Time context: they bundle all the time information of one activity in the composition. This information is used by the synthesis processes and control functions to map to global time of the synthesizer to the local time of the activity.

In the text we discussed the architecture of the system. The environment handles three tasks concurrently:

Synthesis: A synthesizer object calls all the active synthesis processes for the sound synthesis. Sound can be output directly to the audio device.
Event handling: an event thread handles all events that are sent to the system. These events can be sent from any source, including the synthesis processes.
User interaction: an interface thread handles the interaction with the user thru a Scheme shell.

The presence of a real-time task (the synthesizer) and a garbage collector (the Scheme interpreter) in a multi-threaded environment poses a technical problem to guarantee hard real-time conditions. In the text we examined the possible interactions between the two tasks. We imposed the constraint that the synthesis processes cannot allocate new objects during the synthesis. Under these conditions the only synchronization between the two tasks is the root scanning (an atomic operation) and the write barrier (additional instructions for every pointer write operation). If the synthesis processes are developed using traditional real-time design techniques - they generally implement straightforward, easy to analyze signal processing techniques - the problem of the garbage collector reduces to the scheduling problem of concurrent tasks. Thus, real-time performance should be possible.

To use the environment in a distributed application, events can be send using the UDP/IP protocol. On the server side these events are handled by programs set by the user, similar to the handling of local events. Low-level, distributed events thus can use the full richness of the high-level programming environment. Interaction with the Scheme shell is realized over a TCP/IP connection. Real-time sound output over the network is not available as yet. For this we will move the sound output on top of the Real-Time Streaming Protocol (RTSP).

Future work

Although the environment, in its current state, represents a rich playground for music composition and sound synthesis, there are many issues that require additional study and development:

Logical constraint propagation: Patterns organize activities according to a set of relations. These patterns verify the relations after each time manipulation. This approach directs us to the techniques of logical constraint propagation used in graphical interfaces. Further study should allow us to extend the structures for temporal composition and integrate the full power of constraint propagation techniques into the composition environment. Possible starting points are [BMSX97,FB93,FJLV98]
Real-time behavior: The current prototype offers good time performances. For critical situations, however, better real-time guarantees are needed. This requires a more detailed study of the behavior of the system, in particular, the influence of the garbage collector. We have found it hard to come to a conclusion on the real-time capabilities of such a dynamic environment solely on the basis of the discussions found in the literature. A theoretical study should therefore be accompanied by a thorough evaluation of a carefully designed implementation of the garbage collector and the Java virtual machine.
Formalization: We have presented a formal model of the environment. This model served as a guide-line for the implementation. Additional study must allow a more rigid formalization of the architecture of the system. We can refer to Hoare's communicating sequential processes, Petri-nets, Milner's PI-calculus [Mil91], Pratt's work on the duality of time and information [Pra92], or scheduling algebras (see for example [vGR98]).
Graphical editors: The current Scheme interface offers a very flexible programming environment. However, from experience we know that many composers are hesitant to pick up the Scheme programming. It will therefore be useful to develop graphical interfaces for particular programming tasks (the creation of synthesis patches, for example).
Libraries: The current environment lacks a rich set of functions for algorithmic composition and modules for sound synthesis. In the future we hope to use the environment in artistic projects and collect the gained expertise in additional libraries.