A playful application which came to my mind and which was easy to implement is the conversion between spectral envelopes over time (spectrograms) and bitmap images. A spectrogram is a two-dimensional presentation in the time-frequency plane of the spectrum of a sound over time. Here, of course, it displays the course of the spectral envelope over time. Usually, time runs from left to right, frequency runs from bottom to top, and amplitude is coded by brightness (on screen) or blackness (on paper). I.e. one column xin the image corresponds to one time-frame t=x', and the brightness or blackness values for a pixel (x, y) in that column are read from the spectral envelope vt(f) at time t and frequency f=y', with the tick operator ' being some mapping from image coordinates to the time-frequency plane.
The generation of the bitmap is trivial, and for input/output in an image file format, the PBMPLUS library was used. This library generates an intermediate file format, which can be subsequently converted to and from all other existing image file formats.
There is already an interest in converting spectral envelopes to images, namely visualization of spectral envelopes. In figure 8.1, the upper left image shows the spectral envelopes of a mongolian chant over time (the blacker the colour, the higher the amplitude). Sometimes it is interesting to see how the spectral envelope changes over time, how the formants move, etc. But, the other way seems even more interesting to me: reading a bitmap image and converting it to a spectral envelope. This works analogously, just that the brightness of a pixel (x, y) of the image is converted to the amplitude of the spectral envelope: vt(f) at time t=x' and frequency f=y'. The resulting spectral envelope is then applied to some sound's partial structure and an additive synthesis of the resulting sound is performed.
=
Now that we have both directions of conversion, we can apply the vast manipulation techniques developed for graphic design and visual arts to images of spectral envelopes, and listen to the results. Two examples of such manipulations can be found in figure 8.1. Interesting enough, the visually more spectacular twirl in the upper right image sounds rather dull (it doesn't change the original sound much), much less interesting than the spread effect in the lower left of the figure, a granulation -like sonic event, but across the whole spectrum.
Finally it is possible to take an arbitrary image and apply it as a spectral envelope to additive synthesis. This opens up a vast space of new sonic possibilities. For example in the lower right image of figure 8.1, which depicts a long-time exposure of a nightly London street, you can hear every pillar of the building in the background as a pulse, while the white streaks of the lights of passing buses sound like a resonant frequency sweep. (In this image, amplitude maps to brightness, contrary to the other three images, but preserving the image.)
I agree that the serious applicability of this conversion is dubious, because the image manipulations have nothing to do with the properties of the data as a signal, so there is the risk of arbitraryness. Nevertheless, the results are surprising and stimulating. I don't claim that it is more than playing around, but isn't playing around--or exploring--at the very heart of creativity?