Video of the Gestural Music Sequencer

Here’s a video of me playing with my Gestural Music Sequencer. I’ll upload a better version at some point, but I think you can a least get a sense of how you might use this kind of tool (I’m talking about the sequencer, not the performer). As you can see the video has been mirrored so it’s easier to follow your own movements.

To reiterate how the sequencer works, the X axis of the brightest pixel is used to determine the pitch, while the Y axis is the dynamics. The application outputs MIDI data that I’m routing to Reason. I’ve programmed the up and down arrows on the keyboard to increase or decrease a multiplier that along with the 15 fps frame rate determines the time between each note on event. I’ve also enabled a group of keys to adjust the transposition.

 

Keyboard Controls to Adjust Transposition in GMS

I have recently added keyboard controls to dynamically set the transposition in my Gestural Music Sequencer. I set it up like a piano keyboard, so that “Q” is the key of C, “2” is C sharp and so on. In this example I have the application set to to play a minor pentatonic scale. This makes it quite clear when the key changes. In this case I’m modulating between C and F sharp. I also started with a slower note duration, then switched to a faster one. I would not classify this as a very musical performance. This was part of a test to make sure that all of the transposition key controls were operating properly.

Transposition Controls (C to F# Modulation)

Gestural Music Interface in Processing

A big thanks goes out to Jason Striegel and Nick Watts for inviting us to perform at Make: Day at the Science Museum of Minnesota. I performed with my group Keston and Westdal. Other performers included Savage Aural Hotbed and Tim Kaiser. Besides the performances there were some excellent presenters. Myself, Nils Westdal, our drummer Graham O’Brien, and our intern, Ben Siegel greeted visitors at our table. We presented bits and pieces that Graham used with his drums including sticks, pencils, and a chain. We also showed materials from Unearthed Music, Audio Cookbook, and I revealed a gestural music sequencer (GMS) I developed in Processing.

I was really excited to see the reaction to the sequencer. The application samples video and displays it inverted so it looks as though you’re looking into a mirror. Each frame is analyzed for brightness, then the X and Y data of the brightest pixel is converted into a note. The X axis is used to select a pitch, while the Y axis determines the dynamics. As visitors moved, danced, or gestured in front of the camera notes were generated based on a predetermined scale. Here’s a short sample of what the GMS can produce. I’ll post more about this soon.

Gestural Music Interface

Processed Noise Extracted from 1972 Dialogue

What you hear in this clip is all of the processed noise extracted from 1972 Social Commentary Degraded with a Halftone Pattern pasted together. I ran the lot through a short ping pong delay to give give it some stereo processing. Secondly, I decided to leave this file in its 48kHz 24bit state before rendering the MP3. This way you can tell whether or not you have the latest Flash plugin. I just played it on a computer with an older version of Flash Player 10 and got some horrible crackling in the playback, however, if I play it in QuickTime by clicking the link it sounds fine. If you get the same problem, try upgrading your Flash plugin in the browser to fix it.

Processed Noise from 1972 Dialogue

Audio Playback Bug Fixed in Flash Player 10.0.22.87

Get the Latest Flash PlayerOn December 1, 2008 I posted an article about a playback bug in Adobe Flash Player version 10.0.12.36. At that time I was encoding a majority of my audio clips at 48kHz and 24bit, then compressing the MP3 versions to 192kbps. This caused the audio to playback in the Flash player poorly with considerable amounts of warping and distortion. Since then I have been down-sampling my clips to 44.1kHz and 16bit before creating the MP3 to avoid the issue. Although the fix came out early in February, I think I will continue doing this for the time being, until most readers have had time to upgrade their Flash plugins. If you ever look at archived or random entries, I’d suggested upgrading so you don’t experience the playback problems.