The GMS is a Gestural Music Sequencer developed in Processing by John Keston. The application samples video and displays it either normally or inverted so it looks as though you’re looking into a mirror. Each frame is analyzed for brightness, then the X and Y data of the brightest pixel is converted into a MIDI note. The X axis is used to select a pitch, while the Y axis determines the dynamics. As users move, dance, gesture, or draw in front of the capture device, notes are generated based on a predetermined scale. Currently the available scales are pentatonic minor, whole tone, major, minor, and chromatic, all of which can be dynamically selected during a performance.
The scale can also be adjusted using probability distributions. Each note in the twelve tone chromatic scale can be given weighted randomness from zero to one hundred. Notes set to zero will not play, and notes set to one hundred are most likely to play when the probability distributions are enabled.
Other dynamic controls include MIDI out channel, BPM, low and high octave, transposition, sustain, duration selection (manual or randomized with probability distributions), BPM adjustment, and note randomization. A “free” mode allows the durations to be manipulated by the mean brightness of the video input. Finally, four, simple video filter presets were recently added that can be applied by pressing shift + [1-4]. The application works especially well in dark lighting while using a light source to control the sequencer. More information on the controls and features can be found in the readme.txt file distributed with the download.
Important notes:
1. Read the readme.txt!
2. There are known bugs in this software (see step 1).
3. Please follow the steps in the getting started portion of the readme.txt before emailing me.
4. [This step is not necessary with Mac OS X 10.6.x Snow Leopard] To use the inter-application drivers (IAC Drivers) with the GMS to send MIDI to and receive external sync from external applications like, Ableton Live, or Reason, you must install the Mac OS X universal binary java Midi subsystem (mmj.jar).
5. Have fun and if you make something cool post a link in a comment.
*UPDATE* I have suspended working on the GMS due to other projects and time commitments. The most current version does not run on the latest version of OS X (I don’t know about Windows). If you’re interested in revising the source code it is available on GitHub. If anyone is interested in taking this on I will do my best to help and answer questions.
Download:
Download the GMS Beta from Google Code
View a video short about the GMS
Gestural Music Sequencer Documentary Short
Audio examples and more video produced using the GMS
https://audiocookbook.org/category/gms/
Acknowledgements:
Special thanks goes to Grant Muller for his extensive work on improving the RWMidi library for Processing.org. Grant was also an essential resource on external MIDI synchronization and optimization of the application. Thanks, also to Ali Momeni for his suggestion of using probability distributions and inspiring me to develop the software as a potential tool for Minneapolis Art on Wheels.
hopefully this link works. anyway, this is just a simple little thing i made with the GMS. it was the first thing that i recorded, and i liked it enough to develop into something a little longer with a little effect automation and a pad that fades in a couple of times. anyway, it was fun, and i expect that i’ll be making more.
You sleep now by rockyrosga
Thanks, Rocky! Nice piece. Looking forward to hearing more.
Pingback: John Keston » Blog Archive » Chromatic Currents Part I
Pingback: Audio Cookbook » Blog Archive » GMS as Open Source on Google Code
Pingback: Free Gestural Music Sequencer For Mac, Windows | uptownboogiedown
i don’t know why, but i can’t get this to run.
The application starts, but the GUI never appears.
this is on a macbook pro unibody, 2.4GHz, 10.6.2
the application looks absolutely amazing – judging from the video
hmm. there could be snow leopard compatibility issue. i’m still on 10.5.8. anyone out there using it successfully on 10.6.x?
I thought it might be a compatibility issue, but you’re not the only one experiencing the problem. Please try this version and let me know if it there is any change.
http://code.google.com/p/gestural-music-sequencer/downloads/detail?name=GMS_v0.9_Beta_MacOSX.zip
On mac os x 10.6 the new version that you posted here at least launch, but once the video starts it freeze and nothing else happen!
looking forward to give it a try, I love processing project.
thanks, damaru. have you tried other all the options in the “choose a camera” menu? my macbook also freezes if i choose the wrong one, or select the camera more than once. the option that works for me with the isight is “usb video class video”.
I got Snow Leopard, downloaded mmj.jar and the version you linked to above, and things work great when I select “usb video class video” and the Java Sound Synthesizer.
I am having fun with it as week speak.
Thanks so much for making this available.
Pingback: WWW.CREATE-MUSIC.NET » Free Gestural Music Sequencer For Mac, Windows
Thanks, Alexander. It looks like v0.9 is the version to use if you’re on Snow Leopard, which is odd, because the only change in v0.10 is a bug fix for Windows. We might need to have separate forks for OS X and Windows, but I was hoping to avoid that.
One more quick update. According to http://www.humatic.de/htools/mmj.htm, it looks as though mmj.jar might be unnecessary after updating Java on Mac OS 10.6.2.
I’d love to hear confirmation of this, since I’m holding off on Snow Leopard until I get the all clear message from Ableton and Cycling ’74. Thanks!
i got the application to run after using the 0.9 version and deleting the java mmj lib.
although i still have no sound (when selecting the internal java synth) nor midi out via IAC. i’ll investigate further on this.
I’m running 10.5.8 on a Macbook and I’m not sure which combination of java libraries and software versions I should be using. Either the GUI doesn’t launch or I can’t select the right camera input (USB video class) to get iSight to initialize. Sometimes selecting the wrong camera mode initializes iSight but it freezes shortly afterward. In all scenarios I’m not getting any MIDI output no matter which device I choose. Thanks in advance for any help offered.
On 10.5.8 you should use GMS Beta v0.10 and also install the mmj.jar piece. That’s the same OS I’m using and it’s working fine. For the iSight I need to choose “usb video device video”. Let me know if that helps or doesn’t. Thanks! -John
John, is it possible that the 0.10 zip file actually includes the 0.9 version of the software?
No. I compared the versions on code.google.com just to make sure. But even if it was, v0.9 works the same on the Mac side. The only change in 0.10 is fixing a Windows bug.
Pingback: Audio Cookbook » Blog Archive » Almost 300 GMS Downloads, Responses Needed
Hey John,dled the GMS for Windows and tried running it in a VMware XP (as I am a linux user) and when I tried to open the executable I got an error saying the ‘lib” folder could not be found in c://blah/blah/exactly_where_the_folder_is_I-am-looking-right-at-it. so I gave up with that and booted up my “real” XP to see if there were any differences,but unfortunately the same result. Running on a 64 bit machine and the XP is sp3 I do believe.
Mitch, Grant here. I built the version you’re using on 32 Bit Windows 7, but tested it on both XP and 7. I haven’t tested it on a 64-bit machine but I think it should work properly. Can you send me exactly what the error is you’re getting (which lib is it complaining about?). In the meantime, I’ll try to test it on a 64-bit Windows OS.
The ONLY thing I can think of is that we might have a Quicktime discrepancy. In order to render video in Java via the processing lib, it makes a calls to the Quicktime wrapper that is built by Apple…if there is any 32/64 bit weirdness with that library we might have to make some modifications.
To be sure, using capture devices via Java/Processing in Windows is a bit trickier than it is on an Mac.
Here’s a screenshot of what happens.
http://i47.tinypic.com/732e1e.jpg
As you can see,it’s looking for something that is clearly there.
Just to make sure I just updated my jre as it was a bit old being I don’t boot up Windows often, so we can scratch that as a culprit.
I’m going to go crack open a 32 bit windows laptop I have to see how it works there.
Mitch, I’ve never seen an error like that before actually, and it doesn’t appear to be thrown by the JVM. I went digging around to see what 64-bit support for Processing looked like, and according to the 1.0.4 release notes (under known issues):
Currently no 64-bit support for any platforms. On some platforms, you’ll
simply need to replace the Java folder with the distribution with something
more suitable for your operating system.
That just means the that the JDK they supply is 32-bit, but it sounds like there hasn’t been a lot of 64-bit testing with Processing in general.
Let me know if it works on the 32 bit machine for you. If so, then at least we have the problem narrowed down and know where to start looking in order to fix it. Also…another comment coming right below this for Windows users…
Attention Windows Users, it looks like a little tidbit got left out of the release notes for the GMS
To use a video capture device with the GMS on windows, a Video Digitizer (VDIG) that is compatible with Quicktime must be installed. For details regarding why this is necessary check here:
http://processing.org/reference/libraries/video/index.html
There aren’t many free Windows VDIG libraries out there, but the one I used in testing (WinVDIG) is free and came from here:
http://www.eden.net.nz/7/20071008/
Make sure you install version 1.0.1 according to the Processing recommendation, later versions are quite buggy. Other than that the GMS should work exactly as it does on the Mac, keeping in mind that it doesn’t do much if you don’t have a capture device installed.
I’ll add these to the official notes ASAP.
No success at all here. Simply did nothing when launched. Tried the source which didn’t work either. Gave up after an hour.
MacBook with Snow Leopard; most recent Java, don’t know if that would matter.
i have a microsoft vx-1000 webcam,running xp sp3 and i dont see a camera selectable where it should be…no cameras shown available
Hey Grant. It didn’t work on the laptop either. Not sure what’s going on. If you can think of anything to check from my side let me know.
Hi !
This is my very first attempt : http://www.youtube.com/watch?v=VMDVkXAQ7zA
I had some troubles setting up the IAC driver, but after downloading mmj it worked.
Very nice. Thanks, Diego!
@nc, It sounds like you either don’t have WinVDIG installed (see the comment posted by me a few lines earlier) or you don’t have a capture device that is supported by Quicktime. Try installing (or reinstalling) WinVDIG and updating Quicktime to latest to be sure…
@Mitch, is it giving you the same error on the laptop, or just not coming up at all? If it’s not coming up at all you might want to check and make sure your Quicktime installation on that machine is ok. I had some buddies test this out on their machines and the only problem they ran into was a missing QTJava.zip file in the Quicktime installation. Reinstalling Quicktime 7 took care of it.
Let me know if that helps guys,
Yesterday I bought a webcam only to try this tool, and after installing WinVDIG under XP SP3 it works! And how it works! I love it and this night I will not go to sleep :) ‘Love you guys for this tool. Biggest suggestion for Nobel Price :)
Greets from Germany
Hi there,
Jens again.
As I read above that there are no tests on an 64Bit Win7 system.
Well I have the RC (Ultimate) here and a quicktest is successful :) In the next weeks I will install a production Win7 64Bit Home Premium and then I can tell about all functions.
My System is an AMD Quadcore Phenom 2.2, 4GB RAM, Microsoft VX-800 cam and Ableton Live 8.0.8 with “loopBe1”-Software for transmitting the Midi data from GMS to Live.
Greets again
Jens, I’d love to hear how it turns out with the 64-bit Windows 7. I have 32-bit here (Ultimate), but I’m sure the processing.org guys would be happy to hear that applications are working on 64-bit platforms without issue. Keep us updated!
Hi Grant,
sure, I will post my experiences with 64 Bit Win 7 as soon as possible :) CU, Jens
Using a pixelated video I made and the GMS running through Reasons Thor synthesizer I was able to create this bit of audio. Enjoy!
and I suppose it would help if I pasted the link into my comment as well, whoooops! How about you enjoy it now…
http://www
Hey!
First of all, congrats to John and Grant (and everyone else) for this tremendous app!
But, well, since I’m new in the MIDI world, I’m having some trouble making it work…
I’m running Win XP, and I got GMS working with my webcam thanks to the VDIG libraries. The problem is that I can’t get the signal to any DAW (Ableton Live, Nuendo, Reason, etc).
I tried via Loopbe, but nothing…
Could it be that on ‘CHOOSE MIDI OUTPUT DEVICE’ I get every option with the legend ‘Unknown Vendor’??
The only two options I got without it are ‘JAVA SOUND SYNTHESIZER’ and ‘REAL TIME SEQUENCER’ both of Sun Microsystems. But I don’t have those MIDI IN options in any DAW.
help, please?? :$
oh, and happy new year to everyone reading this!!
Matias, it sounds like the LoopBE drivers are showing up as ‘Unknown Vendor’ in your system, I’ve had that problem with loopback drivers in the past on XP (Hubi’s). The RealTime Sequencer and Java Sequencer can’t be used by your DAW. You SHOULD be able to use the ‘Unknown Vendor’ ports, as long as you match up the port numbers between the GMS and the DAW.
If you’re having trouble getting the LoopBE stuff working, you might try out Maple:
http://www.maplemidi.com/Maple_driver.html
Its a four port loopback that I’ve used time and time again without any problems.
http://www.youtube.com/watch?v=m3JVE4zG2M8
My first “fool-around” with GMS.
I used an old web-cam the light of my mouse, my cell’s flashlight and my cigarette.
I fed Ableton Live via LoopBe in windows XP. I installed WinVDIG fist and everything worked well.
I just recorded my desktop with Camstudio.
Cant wait to use it with a proper cam and a planned setup!
Pingback: Audio Cookbook » Blog Archive » Google Nexus One as an Electronic Music Device
Pingback: Free Music Applications For Everyone | ELEVENOONE
Hi there, i’ve donwloaded the windows version of GMS, and when i run the aplication, i got and alert saying:
Java virtual Machine Launcher: Could not find the main class. Program will exit.
I have already installed the latest version of java for vista.
How can i make it work? please!!
Hi Grant,
this is Jens again, you probably will remember my December posts here.
Now I had successfully migrated to Win 7 Home Premium 64Bit, Java 6 Update 17 (32Bit), and all the functions of GMS are working well with WinVDIG and Loopbe1 as the Midi transponder to Ableton Live 8.1. All on an AMD Phenom I Quad 2.2GHz, 4GB RAM, with Microsoft Lifecam VX-800 (cheap but cute).
Except one function like under XP: Choosing an External Midi Device will not exactly choose the clicked item but an item two rows up. But this isn’t something serious…
If you want me to test something special under 64Bit, please mail me.
Thanx again for this wonderful tool :)
Greets from Germany
This is brilliant. Well done. Managed to get it working on my MacBook Pro using the 0.9 build under 10.5.8 (the .10 build didn’t not load) – enabling the IAC drivers in Midi Setup and setting up Logic Pro to receive the Midi data via the IAC. Really cool. Thanks alot for your efforts.
Pingback: This week’s readings got me thinking of… « assistive tech buzz
I’m currently using the GMS in windows 32 bit, using maplemidi to sync with reason. I am currently unable to record in reason. I’ve tried to figure out how to get reason to record sound from the GMS but have been unsuccessful. Any guidance would be much appreciated
I’ve had a similar problem on the Mac. It’ll record fine win a controller, but only playback when routing a MIDI, not capturing the notes. I think I fixed it somehow by adjusting Reason settings. Has anyone solved this on Windows?
Pingback: Audio Cookbook » Blog Archive » Experimental Music Mondays Part 2
Pingback: Audio Cookbook » Blog Archive » Gestural Music Sequencer Generative Track Competition