The GMS is a Gestural Music Sequencer developed in Processing by John Keston. The application samples video and displays it either normally or inverted so it looks as though you’re looking into a mirror. Each frame is analyzed for brightness, then the X and Y data of the brightest pixel is converted into a MIDI note. The X axis is used to select a pitch, while the Y axis determines the dynamics. As users move, dance, gesture, or draw in front of the capture device, notes are generated based on a predetermined scale. Currently the available scales are pentatonic minor, whole tone, major, minor, and chromatic, all of which can be dynamically selected during a performance.
The scale can also be adjusted using probability distributions. Each note in the twelve tone chromatic scale can be given weighted randomness from zero to one hundred. Notes set to zero will not play, and notes set to one hundred are most likely to play when the probability distributions are enabled.
Other dynamic controls include MIDI out channel, BPM, low and high octave, transposition, sustain, duration selection (manual or randomized with probability distributions), BPM adjustment, and note randomization. A “free” mode allows the durations to be manipulated by the mean brightness of the video input. Finally, four, simple video filter presets were recently added that can be applied by pressing shift + [1-4]. The application works especially well in dark lighting while using a light source to control the sequencer. More information on the controls and features can be found in the readme.txt file distributed with the download.
Important notes:
1. Read the readme.txt!
2. There are known bugs in this software (see step 1).
3. Please follow the steps in the getting started portion of the readme.txt before emailing me.
4. [This step is not necessary with Mac OS X 10.6.x Snow Leopard] To use the inter-application drivers (IAC Drivers) with the GMS to send MIDI to and receive external sync from external applications like, Ableton Live, or Reason, you must install the Mac OS X universal binary java Midi subsystem (mmj.jar).
5. Have fun and if you make something cool post a link in a comment.
*UPDATE* I have suspended working on the GMS due to other projects and time commitments. The most current version does not run on the latest version of OS X (I don’t know about Windows). If you’re interested in revising the source code it is available on GitHub. If anyone is interested in taking this on I will do my best to help and answer questions.
Download:
Download the GMS Beta from Google Code
View a video short about the GMS
Gestural Music Sequencer Documentary Short
Audio examples and more video produced using the GMS
https://audiocookbook.org/category/gms/
Acknowledgements:
Special thanks goes to Grant Muller for his extensive work on improving the RWMidi library for Processing.org. Grant was also an essential resource on external MIDI synchronization and optimization of the application. Thanks, also to Ali Momeni for his suggestion of using probability distributions and inspiring me to develop the software as a potential tool for Minneapolis Art on Wheels.
I’m trying to figure out how to get IAC out on osx.. right now all I can see are my interface outs, the real time sequencer and a java sound synth. Any clues on how to get it to show me IAC?
Have you installed mmj – Midi for java on Mac OS X (http://www.humatic.de/htools/mmj.htm)? What you’re describing happens when it’s not installed.
Pingback: Audio Cookbook » Blog Archive » Sound Crawl 2010
Pingback: Gestural Music Sequencer Generative Track Competition « Unearthed Music
So here’s a random question: is it possible to invert the tracking, so that it’s looking for the darkest spot rather than the lightest?
I’m thinking of daylight/outdoor applications; I tried using the GMS in my house during the day but the sunlight kept stealing focus (so to speak).
I suppose I could just mess with the code (I speak Processing, a bit) for my own purposes. But this might be a useful feature, at some point.
Just a thought. Love the app — it’s SO much fun. I’m thinking of using it to generate ambient locational soundtracks — like a constant soundtrack for a specific room, for example.
@Joshua It is not possible with the current version, but that would be an easy mod to the existing feature set. There is a way to invert the video, but the brightness tracking happens prior to any video processing, so it would not effect the pitches selected.
i cant geht mmj to be displayed in ableton OR gms. i read the readme and copied the two files meantioned there to /lib/java/ext, rebooted and it had no effect. please could you give a more detailed instruction how to set this up? especially the routing to ableton – you mentioned in some youtube-video that you told someone how ;-)
great work btw!
@phabe With version 0.11 of the GMS and the latest java update on OS X you don’t need the mmj drivers anymore. Try removing them and then setting up external sync with the IAC drivers.
hey i can’t figure out how to import movies. i know they need to be in a .mov format but when i do a search to select it it will not show in the load screen. i know it is on my desktop but it still doesn’t show. and i noticed it only shows the ones with *.mov* but i still can’t get it to work. is there a way to make a folder to but it in or at least make it so the program reconises the .mov?
i figured out that it may not show the icon for it so you might have to type the name in and then it will show in your list to pull the video in.
Thanks Dusty. That’s good to know. What version and OS are you using?
Pingback: Audio Cookbook » Blog Archive » Reminder About the GMS Generative Track Competition
Hey!First of all, congrats to John and Grant (and everyone else) for this tremendous app!But, well, since I’m new in the MIDI world, I’m having some trouble making it work…I’m running Win XP, and I got GMS working with my webcam thanks to the VDIG libraries. The problem is that I can’t get the signal to any DAW (Ableton Live, Nuendo, Reason, etc).I tried via Loopbe, but nothing…Could it be that on ‘CHOOSE MIDI OUTPUT DEVICE’ I get every option with the legend ‘Unknown Vendor’??The only two options I got without it are ‘JAVA SOUND SYNTHESIZER’ and ‘REAL TIME SEQUENCER’ both of Sun Microsystems. But I don’t have those MIDI IN options in any DAW.
+1
Pingback: Audio Cookbook » Blog Archive » Experimental Music Mondays Part 4
Pingback: Unearthed Music » Blog Archive » Experimental Music Mondays the Fourth
Pingback: Gestural Music Sequencer [Processing, Sound] - Performance tool that converts video input into music /by @jkeston | CreativeApplications.Net
Pingback: Audio Cookbook » Blog Archive » GMS on CreativeApplications.net
Am I being stupid? I cant find the readme.txt in the downloaded file….Looks really awesome.
@robbie No, you’re right. It wasn’t in my latest upload for some reason. I have replaced it, so it should be there now. Thanks!
Pingback: Audio Cookbook » Blog Archive » Chromatic Textures Shown at 6X6 #5: Mystery
Pingback: John Keston » Blog Archive » Chromatic Textures Shown at 6X6 #5: Mystery
Pingback: Audio Cookbook » Blog Archive » Upcoming Ostraka Performance and New Album
Pingback: Gestural Music Sequencer [#Processing, Sound] – Performance tool that converts video input into music /by @jkeston « Fabian Astore
Pingback: Gestural Music Sequencer [#Processing, Sound] – Performance tool that converts video input into music /by @jkeston « Fabian Astore
Hi John – can you let me know what type of light you use and camera – I’m currently using my MBP iSight camera and I have a small flash light – however I’m finding it tricky for the software to follow my light well enough. Any tips?
Thanks. Dan
Hey, @Dan. I suggest being in a dark room without other light sources that might interfere. Also, it does have a tendency to jump around a little bit. This is because the code that generates the MIDI note information is faster than the frame rate of the video source in order to preserve the accuracy of the note durations.
Personally this doesn’t effect the way I use the GMS because I almost always have the note probability distributions enabled, which jogs the notes about based on how probable you make each step in the twelve tone scale.
Pingback: Audio Cookbook » Blog Archive » Ostracon Debut Limited Run on Cassette
Hi John, sorry to bother you. Running v0.9 on my mbp (10.5.8) and it doesn’t seem to produce sound via java sound, or send MIDI. It’s a beautifully realised idea, not being able to make it work is driving me nuts.
@Parkhill What version of the Java Runtime do you have installed? I haven’t tested it with the lastest version but past upgrades from Apple have caused the GMS to fail. Another thing to try is to remove the mme MIDI drivers if you installed them as instructed in the readme. They were only necessary with an older version of the JRE.
I’m using the most recent version of Java without the mme drivers. I also tried v0.11, which had the same result.
Ok, got it working, seems to have been a problem with my computer.
Pingback: Audio Cookbook » Blog Archive » Control the GMS Via Ableton Live
Very tasty, thanks!
I’ll send some samples after i get it wired into logic.
-Peace!
Pingback: Create Digital Music » In/Out Preview: Sounds, Sights, Thoughts, and Free Protofuse Download
@Bumpyscrew I’d love to hear it!
Hi John,
Wonderful piece of software, dying to try it but unfortunately neither of the 2 webcams I have installed/connected appear in the drop down menu, so I can’t choose any input. What am I doing wrong?
Apologies if this is a silly question
Thanks!
@Patrick Would you please provide a bit more information? Computer specs, OS version, GMS version, etc.? If you’re using Windows I won’t be much help, but I do know that some comments on this post describe how to configure the GMS properly for Windows.
I cannot get GMS to open. The application appears to open up, but no screen is coming up at all. I am using a Macbook Pro. Everything is up to date. I also have Processing installed, if that matters. I am not sure what I am doing wrong.
It’s hard to tell without more info, you don’t need the mmj.jar drivers anymore if that helps.
2.53 Ghz Intel Core 2 Duo Mac OS X V. 10.6.4 4 GB 1067 MHz DDR3′
Upon download I moved the GMS folder to my applications section. The menu bar up top claims that GMS is open, giving me the “Hide GMS” “Show All” options, etc. (window controls) but the GMS application itself does not appear to open.
I planned on using my mac’s built in camera to experiment with this program, but nothing is working right now.
@Tony That explains it. The GMS does not currently support Snow Leopard, and won’t until I upgrade my own Mac. I’m not sure what’s causing he error, but I haven’t had time to setup a separate partition for 10.6 and test, etc. Hopefully I’ll have a block of time this Winter to sort it out. I have made an older version available that some users are saying works for them on Snow Leopard here:
http://code.google.com/p/gestural-music-sequencer/downloads/detail?name=GMS_v0.9_Beta_MacOSX.zip
Dear John,
Sorry for delay – in addition to my post about five entries up there..
I am using Windows XP Professional 2002, Service Pack 3, with GMS V0.9
Basically, no drop-down menu appears to allow me to select a camera, everything else seems in order.
Many thanks for your help,
Patrick
Pingback: John Keston » Blog Archive » In Out Festival of Digital Performance, New York, September 2010
Pingback: John Keston Classes » Blog Archive » Week 3
Does the windows version work with FL studio? as rewire client?
Does the Win XP SP3 version work with FLStudio Rewire?
@delixae GMS is not a rewire device, but using inter-application communication drivers you can get it to work. PC wise, there are articles in the comments. On the Mac it’s the IAC drivers.
Please do your best to release a Snow Leopard compatible version of GMS. This program has so much potential.
@Tony I’m actually working on it now. I recently upgraded my Mac to Snow Leopard, so I absolutely have to get the GMS working. It shouldn’t be too long either since I’ll be doing some performances soon that will require that I use the GMS. Nothing like necessity to inspire productivity!
Awesome. I’ll check back soon. Good luck with your work.