In this chapter we have discussed how multimedia pieces, and in particular music pieces, can be structured. The basic element of a composition is called an activity. It is defined very generally as an object that has a location and duration in time. How the ``contents'' of the activity is played and stopped is described by the start and stop program of the activity. In the case of sound activities the start program creates and adds one or more synthesis processes with the help of a synthesis voice. The stop program simply kills the synthesis process.

Composition is possible in various ways:

• Composition algorithms can be expressed in the Scheme language. When the activity is called for the scheduling the algorithm constructs its content with the help of the algorithm.

• A structure can be defined progressively thru the use of a pattern. A pattern groups a number of activities and defines the relations between them. After every modification to a contained activity the relations are verified. A pattern is not used immediately into a composition. They are used thru the intermediary motifs. Motifs are a ``handle'' to a pattern and define the context in which the pattern should be organized.

• The organization of activities whose duration is determined during runtime by user interactions is possible with the use of ``causal'' relations. The start and stop programs of related activities are recombined to create the desired dependencies.

The synthesis processes that are created have a local time space. The local time is mapped onto the ``absolute'' time thru the use of a hierarchy of time models. These time models can be defined in the composition. The hierarchy of the time models corresponds the structural hierarchy of patterns and motifs. All the time information necessary for the time conversions is passed on to the synthesis process in an object called ``time context.'' This rich, layered time model provides the means for complex, non-linear time deformations.

In future projects we would like to extend the current model. First, we want to explore the technique of local constraint propagation for use in the description of patterns. Second, it would be interesting to include an activity into several patterns simultaneously. This is more than just a feature. Musical organization has always been characterized by a complex network of relations. The type of relations are various: structural, harmonic, dynamic, timbral, and so on. Several layers of organization are always in play with the piece. It would be interesting, both from a compositional as from a computer science point of view, to develop a representation scheme for this complex problem.

Thus far we have discussed the architecture of the environment and its principal classes. We have also given examples of the interface to the Scheme interpreter. We have seen how to define synthesis techniques and how to construct music pieces. One question remains: Can such a complex environment really work in real-time? We will analyze this in the next chapter.