Carlos Agon, Gérard Assayag, Mikael Laurson, Camilo Rueda.

Ircam, Paris. {agon,assayag,rueda}@ircam.fr

Sibelius Academy, Helsinski. laurson@siba.fi

Introduction

Although one may consider music composition to be an important issue in any computer music research or development, the term " Computer Assisted Composition " (CAC) has obtained a particular meaning during the past years, at least at Ircam. As opposed to the generation and processing of audio signal by the means of DSP hardware or software technologies, CAC systems focus on the formal structure of music. They generaly suppose symbolic computation techniques with data structures like trees, graphs, sets, collections, associative memory, etc. and algorithms relevant to discrete mathematics. These are adapted to handle the complex structures and relations that are needed in order to model (parts of) musical pieces.

It is well known that this approach has been envisioned by the famous Ada Lovelace in the end of the 19th century when she realized that the unachieved computing machine designed by Babbage should be able to manipulate symbols as well as numbers, and, by computing relations between symbols, could become someday a composing machine for several artistic disciplines including music. Other imaginary machines were decsribed in the 30's by french writer Raymond Roussel who envisioned machines for composing music on the formal and sonic levels.

After the invention of the computer, the first musical applications by Hiller, Xenakis, Barbaud, (Hiller58, Xenakis63, Barbaud68) were essentially combinatorial and formal. Sets of rules, often drawn from academical treatises, were implemented. Search space exploration, finite automata and graph exploration techniques were used at that time.

The interest of the computer music community then shifted massively to the synthesis of sound, specially when high level specialized languages became available (Mathews 69).

Taking advantage of the evolution in computer sciences domains of languages, software technologies, (e.g. object oriented environments), graphical user interfaces and personal computing, the CAC approach seemed to awake in the eighties. In centers such as Ircam (Assayag 93)and Stanford's CCRMA (Taube 91), a significant amount of work has been achieved.

At Ircam, a series of tools were designed, experimented with, and actually used in music production throughout the 80's.

The Formes environment (Rodet 84), although devoted to the high level control of sound synthesis, was written in an object oriented way using the Le_Lisp language. It included a set of classes intended to help the description of processes evolving in time and connected by recursive hierarchical relations. For example, a parent process could distribute data over time to child processes, while receiving information computed by its children during their execution. This model favored the construction of complex sound textures while minimizing the programming task. The output was a set of control signals for software synthesizers.

The Crime environment (Assayag 85,86) was the first attempt at Ircam to realize a general CAC environment where the user could handle musical abstractions and easily program musical formalisms. As a significant innovation, the output was no longer a set of signals, but rather a symbolic description of a musical score which could also be printed in common music notation. Specialized formal sublanguages were made available to composers which could then be used to define rhythmic patterns, harmonic progressions, polyphonic interrelations and so on.

Crime was written in Le_Lisp on a digital Vax minicomputer and conceived by G. Assayag and composer C. Malherbe in the former Music Research Department headed by J.B. Barrière.

After the Crime experience, an important advance was made possible by the recent availability of the Macintosh computer, involving the provision of a graphical user interface and easy connection to small MIDI systems. Several software prototypes (e.g. Preform, Esquisse, see Barrière 90 ) finally resulted, in the beginning of the 90's, in the PatchWork project, by M. Laurson, J. Duthen and C. Rueda, which introduced an original visual programming concept (Laurson 89, 96). The combination of programming simplicity, a visual interface, and low-cost personal hardware created a real infatuation for this environment among european composers with highly diverse musical and aesthetic backgrounds, including Antoine Bonnet, Brian Ferneyhough, Gérard Grisey, Paavo Heininen, Magnus Lindberg, Claudy Malherbe, Tristan Murail, Kaija Saariaho. Developped during the same period by Francis Courtot, CARLA was an attempt to use logic programming in a visual way (Balaban 92).

PatchWork is a visual interface to the Lisp Language. As such, it is a purely functional language, except that some functions may retain a local state and provide a graphical editor for inspecting and editing this state. OpenMusic, by G. Assayag and C. Agon, is the latest Ircam CAC environment in date. It is a visual interface to CLOS, the Common Lisp Object System (Steele 90). Aside from being a superset of PatchWork, it opens new territories by allowing the composer to visually design sophisticated musical object classes. It introduces the Maquette concept which enables high level control of musical material over time and revises the PatchWork visual language in a modern way.

PatchWork

PatchWork (PW) is a tool for computer-assisted composition, although it can be used also as a general purpose programming tool. Writeen in Digitool's Macintosh Common Lisp (MCL), PW provides a graphical interface to the Lisp language. Any Lisp function can be translated to a graphically operable box. The programming syntax consists of making connections between boxes.

The kernel part of PW contains an extensive library of predefined boxes. PW-kernel is musically neutral in the sense that it does not make assumptions about what kind of music or musical raw material is to be produced or analyzed with it. The main aim is to give the user basic tools for visual programming and to provide a straightforward correspondence with the base language, Common Lisp.

The visual model of PatchWork

Much like in most graphical environments, PatchWork's interface is event-driven. It consists of windows in which different kinds of graphical items can be positioned, moved and edited in various ways according to actions triggered by external events such as mouse clicking or key pressing. Of all windows open at one time, one (and only one) is always distinguished to be active. The active window is responsible for mapping external events to specific actions. The PW interface, written in CLOS, takes advantage of predefined graphical classes and methods available MCL. All PatchWork windows are defined as a particular subclass of the MCL window class.

PatchWork is a visual language based on the notion of patch. A patch is a layout of visual elements (or views ) in a window. The views in a PatchWork patch are box frames enclosing one or more input rectangles and exactly one output rectangle. An output rectangle can be linked to one or more inputs of different boxes by connecting lines. The box frame also contains some text naming the box. This text is used as a documentation of the box's computational meaning. Two types of text are also associated with each input rectangle. An editable text defining a default value for that input, and a documentation text. A mouse click on the box name switches back and forth between documentation and value of its inputs. Each box represents the invocation of a particular function. Lines linking boxes define functional composition. A mouse click on an output rectangle launches the invocation of the function represented by the box containing the output. The process of box evaluation recursively launches evaluation of all boxes linked to its inputs, if any.

Figure 1

Boxes are implemented as CLOS objects. In most cases these are instances of a class defined roughly as follows:

(defclass patch-box (view)

((input-boxes) (input-rectangles) (output-rectangle)

(meaning-function)) )

The input-boxes slot contains the list of boxes connected to the box's inputs, if any. Theinput-retangles and output-rectangle slots contain the views defining the input rectangles and the single output rectangle of the box. Finally, the meaning-function slot refers to a Lisp function defining the computational meaning of the box. Since patch boxes are subclasses of the standard MCL class view they can be simply added as subviews of a PW window (itself a subclass of the standard MCL window class) to benefit from all the graphical functionalities of MCL windows. In fact, input-rectangles and output-rectangle are also subclasses of a MCL view having their own event handling fucntionalities for editing the default value of each input and for transforming click events into a box evaluation process. PW windows are approximately defined as

(defclass pw-window (window) ((boxes)) ) ,

where the boxes slot contains the list of all patch boxes in the window. Drawing a patch, for example, is then simply controlled by standard MCL event handlers that are redefined for the PW window class:

(defmethod view-draw-contents :before ((self pw-window)

(tell (boxes self) `view-draw-contents) )

A visual patch is intended to define a very simple model of computation in which boxes take the role of function calls. Procedural abstraction is added by allowing a whole patch to be visually collapsed into a single abstraction box with suitably defined inputs and output. The reverse action, that is, expanding an abstraction box into its patch window, is achieved by a double-click mouse event. Abstraction boxes belong to a subclass of a the standard patch-box with two added slots, one for holding a pointer to the patch window associated with the abstraction, and one containing a reference to the box defining the output of this patch. Instead of looking at the meaning-function to compute its result, the abstraction box simply asks the output box of its associated patch to compute its result. Abstractions are thus a sort of mirror deflecting evaluation messages. This property allows changes made to an abstracted patch to be immediately reflected in its abstraction box.

Boxes which have windows attached to them are not restricted to abstractions. In fact, this idea is generalized in PatchWork to link suitable visual representations of complex musical structures to the box that computes them, as explained further below.

The basic computational model of PatchWork is augmented with two important features, namely :

* state -preserving boxes, and

* data structure interpretation, visualization and editing

A PW box can be defined to be state -preserving in that the result of the invocation of its associated function depends on the result of a previous invocation. Such boxes are identified visually by the presence of alock. A locked box acts as a constant function returning the last value computed before the lock was set. Locking and unlocking of boxes is handled through mouse events.

Data in state-preserving boxes can be interpreted in various forms, notably as musical structures. Each musical structure can be given a visual representation adapted to its compositional use. This representation is ensured by window objects associated with the patch box in much the same way as is done for abstractions. In most cases the window object is not only used for visualization but also for editing. There is, in this case, an additional relation between the data object computed by the patch box and its edited visual interpretation, namely, they must be at any given moment equivalent , in the sense that a transformation between one and the other should be always possible. This means that locked state-preserving boxes always return a data object which is the equivalent of the currently edited visual interpretation. Figure 2 shows a box that interprets a list of integers as notes which are displayed in common musical notation.

Figure 2.

Each box that interprets data as some musical object is defined as a subclass of the standard patch-box class. A box defining a note, for example can be defined as follows:

(defclass note-box (patch-box)

((midic) (velocity) (duration) (channel)) )

where slots define pitch, dynamics, duration and instrument of the note. Associating a note editor to this object simply requires changing in the definition of its superclass from patch-box to a standard box-with-editor class which in addition to the structure of a standard patch-box supplies a window representing an editor object. The box-with-editor class also handles basic behaviour such as opening the associated window editor in response to mouse double-click events. Editor objects have a very simple structure:

(defclass mn-editor (window)

((menu-bar) (patch-box) (view-controller)) )

An editor thus has three slots : the menu bar for that editor, a pointer to its associated patch box and a view controller. The view controller handles edition of parameters affecting globally the outlook of the window (such as zooming). In addition to a set of buttons for giving user access to this global editing (e.g. the dyn button for displaying the note dynamics in figure 2), a view controller contains a number of panels . A panel object is a view that handles specific editing actions within a precise region of the window (such as changing the pitch of a note). Any subview of an editor (view controller or panel) can handle mouse events occurring inside its associated region. This simple hierarchical structure gives a lot of flexibility to the visualization and editing of computed musical data. view-controller and panel can be easily subclassed to accommodate particular editing capabilities of any object computed by a state-preserving patch box. Supplied standard PW boxes have a variety of editors for visualizing notes, chords, chord sequences or rhythms in various ways. This scheme can be used to visualize the same computed data as a break point function, musical notation or editable cell arrays (see figure 3).

Figure 3

Finally, there is also a static typing scheme in PW. Two PW boxes can be connected when input and output types are compatible. Each box defines the set of types it admits for each of its inputs and the set of types of the results it computes. PW patch-box objects store information about their output types whereas input-rectangle objects keep a list of possible input types. The graphical editor of a patch window enforces the typing scheme by allowing connections only if the set of output types in the source box intersects with the set of input types in the target box.

The semantic model

The functional behaviour of a patch in PW has a direct correspondence in the behaviour of a Common Lisp form. More precisely, a PW patch induces a patch graph. This is a connected acyclic graph P=<B, E, C, F, O >, whereB is a set of nodes called boxes, E is a set of nodes called entries, C is a set of directed arcs called connections, F is a set of node labels called values and O is a set of arc labels called orderings. C is a subset of . Patch graphs are in one to one correspondence to PW patches. Figure 4 shows the graph corresponding to the patch in figure 1. The formal semantics of a program in PW is defined by a valuation function mapping each graph P to a well formed Common Lisp expression. We say V[P] is the meaning of the patch P. In what follows, we describe this mapping and explain how the actions occurring in PW when a patch evaluation is triggered correspond to the Lisp form given by the valuation mapping.

Figure 4

A PW box is evaluated by a mouse click event on its output rectangle. This results in the evaluation of the Lisp expression:

(patch-value (view-container self) (view-container self))

where the argument self corresponds to the output-rectangle slot of the box. The expression (view-container self) returns the patch box whose evaluation has been requested. Evaluation of a PW box thus invokes its patch-value method. For standard PW boxes this is defined as follows:

(defmethod patch-value ((self patch-box))

(apply (meaning-function self)

(ask-all (input-boxes self) 'patch-value)))

The PW macro ask-all sends the method given in its second argument to the list of objects given in its first argument. In the above code the results of invoking patch-value for each input-boxes are collected into a list. Then the Lisp function associated with the box (i.e. the meaning-function slot) is applied to that list.

The meaning of a standard patch box in a patch is thus defined by the valuation mapping of the equivalent patch graph rooted at the box. This valuation basically translates into Lisp the behaviour described above:

V [BOX.patch-box] =

(value(BOX.patch-box)

V [connection (1,BOX.patch-box)]

V [connection (2,BOX.patch-box)]

...

V [connection (n,BOX.patch-box)] )

where connection (i, BOX.patch-box) represents the target of the arc of ordering i going out from the node BOX.patch-box. The expression value (node) maps box and entry nodes to their labels . The valuation of an entry node is defined as

V [ENTRY] = (QUOTE value(ENTRY))

The valuation of the patch graph of figure 4 at node g+ is thus

(g+ (quote 25)

(g* (const (quote (60 64 67)) )

(quote 100))

This kind of translation semantics is well defined only if there are no cycles in the patch graph. The PW graphical interface enforces this condition by refusing any recursive connection between boxes.

Since most PW boxes are direct instances of the class patch-box , the above valuation accounts for the bulk of patch graphs. However, PW constructs, such as state preserving boxes or loops, require a more complex treatment.

Figure 5

The patch and corresponding graph in figure 5 specifies a loop that constructs a list of note objects with pitches D#, G, A . The valuation of this patch is given by

V [BOX.pw-map] =

(MAPCAR

(FUNCTION (LAMBDA (V [connection (1,BOX.pw-map)] )

V[connection (2,BOX.pw-map)] ))

V[connection (1,connection (1, BOX.pw-map) )] ) )

and

V [BOX.enum] = *AN-ENUM-VARIABLE*

which results in the Lisp code

(mapcar

(function (lambda (*AN-ENUM-VARIABLE*)

(mk-note *AN-ENUM-VARIABLE*

(quote 75) (quote 100) (quote 1))) )

(const (quote (6300 6700 6900)))))

State preserving boxes are valuated into closure expressions. These are Lisp forms containing a local state. Details about how this is actually expressed in the valuation function are beyond the scope of this paper.

From the above and the previously mentioned type checking scheme, it follows that the valuation of a PW patch always leads to a well formed Common Lisp expression. In this sense, a patch is always "correct". This is one of the major advantages in visual programming, and probably one of the reason why composers have taken on PatchWork so easily. There are, of course, patches that do not behave according to the user's expectations (not to mention Lisp's expectations!) but then the exact behaviour of the patch is equivalent to that of the valuation of its patch graph in the semantics of Common Lisp.

There is then a close correspondence between PW patches and Lisp programs. This fact is a central design issue for two reasons. On the one hand, it provides an easy way to save, load and optimize patches by compiling them into Lisp and on the other, it allows a complete two-way interaction between Lisp and PW. The latter is underlined by a scheme that gives the option of transforming any user defined Lisp function into its visual box counterpart. In this way different levels of computer programming abilities may smoothly coexist using the same graphical environment while taking full advantage of the built-in musical structure construction and editing capabilities of PW.

PW Libraries

As has been mentioned, PW can be easily extended to accommodate different ways of conceiving and manipulating musical structures. Users can package patch boxes into libraries. A set of tools are provided for helping the musician in setting up a hierarchy of menus where the addition of her/his boxes to a PW window can be done by simply selecting the box name in the menu. Over the years, a rich set of libraries have been constructed by composers and programmers. Several of these libraries have been integrated into the PW environment thus providing a very comprehensive spectrum of musical applications. These include :

* Esquisse, conceived by the composer Tristan Murail, for spectral music operations.

* Chaos, by Mikhail Malt, for the definition of musical structures through stochastic manipulations and dynamic models.

* Kant, by Gerard Assayag, Carlos Agon and Camilo Rueda, for rhythm quantizing (Assayag 94),

* PWConstraints, by Mikael Laurson, for rule base programming,

* Situation, by Antoine Bonnet and Camilo Rueda, for constraint based generation of harmonic and rhythmic structures,

* Repmus by G. Assayag and C. Malherbe for harmonic and rhythmic structure generation and processing.

Some of these are described below.

Constraints

Constraint programming is nowadays a very active research topic that can have far reaching applications in computer assisted composition. We describe two systems, PWConstraints (Laurson 93) and Situation (Bonnet 98), both based on the notion of programming as searching for sets of elements satisfying a certain number of given properties or constraints. PWConstraints is a PatchWork library and Situation an OpenMusic library (a version is also available for PatchWork). Both systems offer constraint satisfaction capabilities for constructing musical structures. They differ significantly in the strategies and algorithms used and also in their relative " knowledge " about the application domain. Much like PatchWork, PWConstraints is neutral in the sense that it makes no attempt to favour applications that fit into a particular notion of a musical object. In fact, as is also the spirit of PatchWork, objects are " musical " only if the user chooses to interpret them as such. The elements that the search engine of PWConstraints tries to find does not necessarily have any structure, although complex PatchWork musical structures can also be involved in the exploration. Situation on the other hand, defines a very general structure for the objects it deals with. They should be expressible as sets of points and distances in a certain space. This structure is very general since it can easily accommodate musical objects such as chords, chord sequences, voices, rhythmic patterns, etc. But the fact that Situation " knows " already the structure of its objects allows it to greatly optimize solution searching in many practical applications. Obviously, not every problem can be conveniently expressed as that kind of structure, so Situation has a way to consider unstructured objects but in this case the optimizations do not apply. We describe each system in the following.

PWConstraints

PWConstraints is a rule-based programming language for solving complex musical problems where the user describes the end result from many different points of view. Writing counterpoint is a good example. The result is described with the help of rules controlling melodic lines, harmony, voice-leading, etc.

When using PWConstraints we first define a search space. Then a backtrack search engine looks for solutions by systematically producing potential results. In general not all possible results are looked for, but only those satisfying a set of rules or constraints.

A search space is defined as a set of variables. Each variable has a domain containing a set of values. PWConstraints provides a protocol for writing rules. A pattern-matching language is used to extract relevant information from a potential solution. This information is given to a Lisp test function that either accepts or rejects the current choice made by the search engine. The rules are compiled into computationally efficient Lisp functions. The user can also define preferences by heuristic rules.

In addition to these basic tools PWConstraints contains several extensions. The most important and complex one allows the solving of polyphonic search problems. As in a traditional polyphonic score, we operate with several layers (parts, voices) of events (notes). The user gives in advance the rhythmic structure of a search problem in a standard PW rhythm-editor. This input score (which can be arbitrary complex) is given as an argument to the search engine. The search, in turn, aims at filling the input score with pitch information according to the given rules.

Figure 6

Situation allows the construction of objects out of two notions: That of point and that of distance. An object contains one or more points, spaced according to certain distances which are evaluated in a given unit of measure. Distances can be internal or external depending on whether they involve points contained in only one object or in several different objects. Intervals in a chord, for example, are internal distances whereas melodic intervals are external. The unit of measure can be a semitone, an eighth tone or any other. The objects could also be rhythmic plans concerning a set of articulation points spaced according to given distances. An eighth note, quintuplet or other could be the unit of measure in this case.

The specification of a problem for Situation consists in supplying the number of objects wanted, the space of possibilities (region) for points, the number of distances in each object and the set of possibilities for those distances. Built-in constraints establish the allowed configurations of each object and also that of object combinations. Any point or distance of any object or set of objects can be related by a constraint. Built in constraints conveniently define : general profiles that should follow the objects ; patterns that should match a given set of distances ; fixed points that should belong to every solution ; equality or difference of points or distances (or simultaneity, in the case of rhythms), etc. An example of Situation will be given in the OpenMusic section below.

OpenMusic

OpenMusic is a highly visual environment for the composer on the Macintosh. It is intended as a superset of PatchWork. While drawing benefit from the huge amount of knowledge and experience gathered around the PatchWork project, OpenMusic implements a set of new features that makes it a second generation Ircam compositional software.

Constructed on top of Digitool MCL, OpenMusic provides a visual programming interface to Lisp programming as well as to CLOS. Thus OpenMusic is an Object Oriented visual programming environment. Objects are symbolized by icons that may be dragged and dropped all around. Most operations are then performed by dragging an icon from a particular place and dropping it in an other. These places include the OpenMusic Workspace as well as the Macintosh finder itself. A lot of classes implementing musical data and behaviour are provided. They are associated with graphical editors and may be visually subclassed by the user to meet specific needs. Different representations of a musical process are handled, which include common notation, MIDI piano-roll, sound waveforms, break-point functions. High level in-time organisation of the music material is available through the maquette concept.

The music notation in OpenMusic is handled by CMN, a portable Common Lisp library designed by Bill Schottstaedt at CCRMA. CMN has been extended for Quickdraw compatibility and some interactive editing features have been added. The MIDI environment used is MidiShare by Yann Orlarey and his team at GRAME. Much expertise on musical and user interface issues have been provided by Joshua Fineberg, Claudy Malherbe, Mikhail Malt, Tristan Murail

Workspace

The main interface to OpenMusic is the Workspace window (figure 7). The workspace is a icon-oriented browser based on the Macintosh Finder metaphor. Icons represent OpenMusic objects, or folders where objects (or subfolders) may be stored. Every OpenMusic operation starts by browsing the workspace, selecting objects, dragging them from one place to the other, opening them, or interconnecting them.

Several independent Workspaces may be maintained simultaneously on a single machine so several users can have their own personalized environment, or a single user may work simultanuously on several projects.

Objects in the OpenMusic workspace are persistent. This implies that the user does not have to worry about file management. Objects no longer required may be trashed.

figure 7

Packages, classes, methods

Classes are a common concept in OO languages. They are templates from which object instances are derived. Classes inherit from one another so it is easy to define graphically a new class that adds specifical data and behaviour to an existing class. It is possible to create new, specialized, musical objects by subclassing the standard classes provided in OpenMusic. A method is a function whose behaviour changes depending on the classes its arguments belong to. For example, one could decide that the '+' method, if called with 2 numbers, should compute their sum, and if called with 2 musical structures should concatenate them.

The packages folder is a special folder in the workspace and the main access to classes and methods. These are dragged from this folder to a patch window in order to become active, that is to become actual invocations. Packages are inspired from the Java language and provide a place where related classes and functionalities are stored together. Packages can have subpackages.

figure 8a

figure 8b

figure9

The OpenMusic class hierarchies may be browsed in the package view or as trees (figure 8). Subclassing is done graphically (figure 9) by dropping an alias of the superclass in the target package of your choice, creating a new class icon and connecting it to the superclass icon, then adding slots to the new class and giving them a type and a default value. Initialization methods (equivalent to CLOS initialize-instance) can be programmed visually as ordinary methods.

Patches

As in PatchWork, a patch is a place where objects --functions, classes, instances, subpatches or maquettes -- will be interconnected in order to specify musical algorithms. Functions are generic, so when they are invoked in a patch window, a particular method belonging to the function will be called depending on the kind of objects connected to its inputs. Another difference with PatchWork is that functions are multivalued thus they can have as many outputs as needed.

When a class icon is dragged from a package to a patch window, it becomes a class factory. A factory looks like a function call, whose result is a new instance of the class it represents. Factories have as many inputs and outputs as there are public slots in the associated class, plus one. Inputs allow the initialization of slots, outputs allow the direct reading of slot values. The supplementary, leftmost output, delivers the new instance as a whole. The leftmost input is useful when we want to create an object from another object, by cloning it or converting it, rather than by slot initalization. If, for example, an instance of class measure is connected to a voice factory through its object input, a single-measure voice will be created. This mechanism is extensible through a special kernel ObjFromObjs method, so any automatic type conversion may be added.

A factory is almost always linked with a graphical editor where the last instance created is made visible. This instance is kept as a local, lockable, state of the factory.

Icons in a patch may be in one of three states :

* The locked state means that evaluation yields the local state, which is always preserved. This is very useful if the result has been modified in an associated editor.

* The eval-once state means that evaluation occurs only once during a global evaluation of the patch, which is useful to avoid multiple computation of the same.

* The lambda state means that instead of giving back a computed value, the object is asked to deliver itself as a function. This is how a patch or a function can be passed to another one that waits for a functional argument.

The patch abstraction mechanism is very straightforward. When a patch icon is dropped into a patch window, it appears as a patch invocation icon, with its inputs and oututs as small round inlets and outlets attached to its icon. Note that a patch can be recursively called within itself.

figure 10

figure 11

In figure 10, two voice factories are merged into a third one. Depending on tests (whose conditions are hidden into subpatches) either the resulting voice or a midifile or a soundfile (these files are just dragged from the finder to the patch window) are played.

In figure 11 the Situation constraint solver (see above) is used to harmonize a melodic line. Constraints expressed symbolically in the upper left determine lists of authorized intervals as well as an interpolation pattern for the authorized ambitus and density of chords. Graphical constraints in the upper right impose that the lower pitch line in the resulting chord sequence must be monotonically decreasing and that the upper line is to be taken from a midifile (in this example, the beginning of Syrinx by Debussy). In the voice factory in the bottom, the rhythm is entered symbolically in the corresponding slot, and the pitches come from the constraint solver. Its first output on the left is the resulting voice object itself, connected to the output of the patch. Outputs on the right are the available voice instance slots (e.g. the rhythm tree structure and a list of chord instances). They correspond one-to-one to the inputs. Figure 12 shows the voice object opened in its editor window, where it can be interactively edited.

figure 12

Genericity

Functions in OpenMusic are generic and multivalued. As many methods as wanted may be added to the method set of a function. In figure 13a we have opened the inverse generic function by double-clicking on its icon. This function contains two methods. One is defined for the class list and the other for the class voice. We would like to state that the inverse of a voice instance is a new instance with the same rhythm as the first one but a reversed list of chords. Figure 13b shows the content of the method specialized for class voice opened in a patch-like window. Its input class has been set to voice just by dragging and dropping the icon of that class above it. Once the input is associated with a class, it shows its public slots as small outlets. Here, all we have to do is take the rhythm (second outlet from the left) and plug it into the rhythm inlet of the voice factory below. The third outlet on the left is the entering voice's list of chords. It is provided as an argument to a recursive invocation of the inverse function (which, this time will dispatch to the method specialized for class list). Once the list of chords has been reversed, it is sent to the chords slot of the voice factory, which is connected to the output of the method and thus considered to be its result value.

figure 13a

figure 13b

Maquette

A maquette is a kind of patch with an horizontal time dimension. It helps organize the musical materials computed in patches or built in the editors into higher level, time oriented structure. One may drag and position in time any object into a maquette window providing this object has an (explicit or computable) duration. Maquette-compatible objects are : an instance of a musical class, like voice or chord ; a patch that computes an instance of a musical class ; a midifile or a soundfile dragged from the finder ; or another maquette. A maquette can be played through audio and MIDI. As with other OpenMusic objects, a maquette can be viewed as an icon or opened in an editor window. As an icon, it can be stored into a folder or dropped into a patch window where it may be connected to other modules.

As was mentioned before, a key feature of PatchWork was the possibility to have both a visual representation of a musical structure in an editor and also of the patch leading to it. However, these two forms of notation were really completely separate. The Maquette idea in OpenMusic is a step towards their integration.

In figure 14a a maquette editor can be seen. The icon representing the patch in figure 11 has been dragged to the maquette window. It appears as a light gray rectangle in the upper right. The lower left block has been added directly in the maquette. It is shown opened in a patch-like window in fig. 14b. This block is asked to fill itself by taking the value present at its input and reversing it, using the inverse generic function defined in fig. 13. The two other (dark gray) blocks are a midifile and a soundfile directly dragged from the finder. Figure 14c shows the maquette after evaluation.

figure 14a

figure 14b

figure 14c

A maquette can be embedded into another maquette. It may also be algorithmically constructed by using a maquette icon in a patch window and providing it some content through its inlets. Note that such a patch containing a maquette as a part may again be dropped into another maquette, and so on. Such a flexible set of possibilities makes OpenMusic a powerful tool for designing sophisticated hierarchical/temporal musical structures. Another advandced way to control dynamically the temporal structure is the use of the self object that can be seen in fig. 14b. This object stands for the maquette block itself. Its outlets represent, from left to right, the object as a whole, then its public slots, including its time position and duration. By using these slots as values (i.e drawing links from them), the block can parametrize the computation of its musical content with regards to its time position or scaling coefficient. By acting directly on the self object (i.e. using the left most outlet) and setting its slots using some specialized methods not detailed here, the position and duration of the block can be changed from the outside. One can thus imagine a maquette where the evaluation of a particular block involves the re-positioning and scaling of some other blocks, and even the creation of new blocks.

How is OpenMusic done ?

Rather than a graphical interface to CLOS, OpenMusic can be regarded as an extension of CLOS with metaprogramming techniques. This involves basically the possibility of subclassing metaclasses (such as standard-class, the class whose instances are themselves classes) and then defining new methods for the subclasses. Figure 15 shows the hierarchy of OM classes. CLOS classes are represented by a bold framed rectangle.

figure 15

Since there is no defined CLOS protocol for Lisp simple functions (i.e. non-generic functions) and simple instances (i.e. instances that are not classes), the technique of delegation, rather than multiple inheritance, has been used in these case, for the classes OmLispFun and OmInstance. All other CLOS objects have been orthogonally integrated in the visual formalism. The left side of the tree in figure 15 defines OM metaclasses whereas the right side (rooted at OMFrame) implements the visual paradigm. It can be seen that the class OMFrame inherits from class OMBasicObject. The reason for this is that the visual paradigm itself is part of the language. This means that (very) advanced users could control the visual behaviour itself by visual programming.

Every OM object has a name and it can be visualized in one of two ways, either as a simple view or as an Editor. An Editor is an instance of class OMEditor and a simple view is an instance of the class OMSimpleView. These classes are subclasses of OMFrame, which contains an object slot pointing back to the object it visually represents.

There is a protocol of generic functions which is applicable to every instance of OMBasicObject. In this model an object is composed by a set of elements together with (optionally) a relation over them. An OMClass object, for example, is composed of an ordered list of slots, a generic function contains a set of methods, a patch contains a list of boxes, an editor a list of frames, etc. A fundamental function in the system is thus get-element which returns for any OMBasicObject the list of its elements.

Every object is first visually represented as a SimpleView. This representation is obtained from any OMBasicObject by the function get-simple-view. Similarly get-editor gives the Editor of an OMBasicObject. A double-click event opens the editor of an object by calling OpenEditorObject with the object slot of the simpleView as an argument. OpenEditorObject calls the function get-simple-view for each element of the object. An OMEditor is then constructed with all these simple views as elements.

Functions add-element and remove-element are supplied for the user. These functions and the mechanism of delegation allow editing of any basic object.

Drag&drop is the main tool in OM. This is defined as an action between a dragged OMsimpleView and a target OMEditor. A set of allow-drop predicates determine whether the object slot in the source can be added to the object slot in the target. The drop operation is performed only when those predicates hold. This results in the invocation of the add-element function with the dragged and target frames as its arguments in order to visualize the operation, together with the delegation of the message to the corresponding object slots in the dragged and target frames, so as to launch the edition of the relevant OMBasicObject. Other OM operations such as move, select, eval follow a similar mechanism.

Future work

PatchWork libraries are progressively ported to OpenMusic. As OpenMusic is a superset of PatchWork, there should be some day a compatibility engine to convert PW patches into OM patches.

Constraints have proven to be a very attractive paradigm for composers. Until now, they have been added as external libraries in PW and OM. A more powerful approach would be the integration of a visual notation for fine grained constraint programming. There should then be a way to consider patches as predicates expressing, through particular links, logical relations between objects or object slots. This would avoid the user to be subjected to predefined constraints or complex textual rule specification.

Information about ongoing work is available at:

http://www.ircam.fr/equipes/repmus.

Aknowledgements

Thanks to Andrew Brown for kind proof-reading.

References

G. Assayag, M. Castellengo, C. Malherbe, Functional integration of complex instrumental sounds in music writing. Proceedings of the ICMC, Vancouver, 1985.

E. Amiot, G. Assayag, C. Malherbe, A. Riotte, Duration structure generation and recognition in music writing. Proceedings of the ICMC, La Haye, 1986.

Assayag G., Agon C., Fineberg, J., Hanappe P. An Object Oriented Visual Environment For Musical Composition, Proceedings of ICMC, 364-367, Thessaloniki, 1997.

Assayag G., Rueda C., The Music Representation Project at IRCAM, Proceedings of the ICMC 93, 206-209, Tokyo, 1993

Assayag, G., Agon, C., Fineberg, J., Rueda, C. Kant: a Critique of Pure Quantification, Proceedings of the ICMC, Aarhus, 1994.

Barrière J.B., Devenir de l'écriture musicale assistée par ordinateur : formalisme, forme, aide à la composition, Analyse musicale, ndeg. 20, 52-68, 1990.

Balaban, M. K. Ebcioglu, and O. Laske, eds. 1992. Understanding Music with AI: Perspectives on Music Cognition. MIT Press.

Barbaud, P. La musique discipline scientifique, Paris, Dunod, 1968

Bonnet A., Rueda A., Situation: Un Langage Visuel Basé sur les contraintes pour la Composition Musicale, in Recherches et applications en informatique musicale, M. Chemillier and F. Pachet (Ed.) , Hermès, Paris, to be published, 1998.

Hiller L.A., Isaacson L.M., Experimental Music, McGraw-Hill, 1958.

Laurson M., Duthen J., PatchWork, a graphical language in PreForm, Proceedings of the International Computer Music Conference, San Francisco, 172-175, 1989.

Laurson M., PWConstraints, In Haus G. and Pighi I. (eds), X Colloquio di

Informatica Musicale, 332-335, Associazione di Informatica Musicale Italiana, . Milano 1993.

Laurson M. PATCHWORK: A Visual Programming Language and some Musical Applications. Helsinki: Sibelius Academy, 1996.

Mathews, M. V., The Technology of Computer Music" Mit Press, 1969.

Steele, G.L., Common Lisp the Language, second edition, Digital Press, 1990.

Taube H., Common Music: A Music Composition Language in Common Lisp and CLOS , Computer Music Journal, vol. 15, ndeg. 2, 21-32, 1991.

Rodet, X., and P. Cointe. FORMES: Composition and Scheduling of Processes. CMJ 8(3):32-48. 1984.

Xenakis I., Musiques formelles, Richard-Masse, 1963.