GMS
The GMS is a Gestural Music Sequencer developed in Processing by John Keston. The application samples video and displays it either normally or inverted so it looks as though you’re looking into a mirror. Each frame is analyzed for brightness, then the X and Y data of the brightest pixel is converted into a MIDI note. The X axis is used to select a pitch, while the Y axis determines the dynamics. As users move, dance, gesture, or draw in front of the capture device, notes are generated based on a predetermined scale. Currently the available scales are pentatonic minor, whole tone, major, minor, and chromatic, all of which can be dynamically selected during a performance.
The scale can also be adjusted using probability distributions. Each note in the twelve tone chromatic scale can be given weighted randomness from zero to one hundred. Notes set to zero will not play, and notes set to one hundred are most likely to play when the probability distributions are enabled.
Other dynamic controls include MIDI out channel, BPM, low and high octave, transposition, sustain, duration selection (manual or randomized with probability distributions), BPM adjustment, and note randomization. A “free” mode allows the durations to be manipulated by the mean brightness of the video input. Finally, four, simple video filter presets were recently added that can be applied by pressing shift + [1-4]. The application works especially well in dark lighting while using a light source to control the sequencer. More information on the controls and features can be found in the readme.txt file distributed with the download.
Important notes:
1. Read the readme.txt!
2. There are known bugs in this software (see step 1).
3. Please follow the steps in the getting started portion of the readme.txt before emailing me.
4. To use the inter-application drivers (IAC Drivers) with the GMS to send MIDI to and receive external sync from external applications like, Ableton Live, or Reason, you must install the Mac OS X universal binary java Midi subsystem (mmj.jar).
5. Have fun and if you make something cool post a link in a comment.
Download the GMS Beta for Mac OS X:
GMS Version 0.11 Beta for Mac OS X
Download the GMS Beta for Windows:
GMS Version 0.10 Beta for Windows
View a video short about the GMS
Gestural Music Sequencer Documentary Short
Audio examples and more video produced using the GMS
http://audiocookbook.org/category/gms/
Acknowledgements:
Special thanks goes to Grant Muller for his extensive work on improving the RWMidi library for Processing.org. Grant was also an essential resource on external MIDI synchronization and optimization of the application. Thanks, also to Ali Momeni for his suggestion of using probability distributions and inspiring me to develop the software as a potential tool for Minneapolis Art on Wheels.
74 Responses to “GMS”
hopefully this link works. anyway, this is just a simple little thing i made with the GMS. it was the first thing that i recorded, and i liked it enough to develop into something a little longer with a little effect automation and a pad that fades in a couple of times. anyway, it was fun, and i expect that i’ll be making more.
You sleep now by rockyrosga
By rocky rosga on Nov 11, 2009
Thanks, Rocky! Nice piece. Looking forward to hearing more.
By John Keston on Nov 11, 2009
i don’t know why, but i can’t get this to run.
The application starts, but the GUI never appears.
this is on a macbook pro unibody, 2.4GHz, 10.6.2
the application looks absolutely amazing – judging from the video
By nq on Dec 6, 2009
hmm. there could be snow leopard compatibility issue. i’m still on 10.5.8. anyone out there using it successfully on 10.6.x?
By John Keston on Dec 6, 2009
I thought it might be a compatibility issue, but you’re not the only one experiencing the problem. Please try this version and let me know if it there is any change.
http://audiocookbook.org/wp-content/uploads/2009/09/GMS_v0.9_Beta_MacOSX.zip
By John Keston on Dec 6, 2009
On mac os x 10.6 the new version that you posted here at least launch, but once the video starts it freeze and nothing else happen!
looking forward to give it a try, I love processing project.
By damaru on Dec 6, 2009
thanks, damaru. have you tried other all the options in the “choose a camera” menu? my macbook also freezes if i choose the wrong one, or select the camera more than once. the option that works for me with the isight is “usb video class video”.
By John Keston on Dec 6, 2009
I got Snow Leopard, downloaded mmj.jar and the version you linked to above, and things work great when I select “usb video class video” and the Java Sound Synthesizer.
I am having fun with it as week speak.
Thanks so much for making this available.
By Alexander Maier on Dec 6, 2009
Thanks, Alexander. It looks like v0.9 is the version to use if you’re on Snow Leopard, which is odd, because the only change in v0.10 is a bug fix for Windows. We might need to have separate forks for OS X and Windows, but I was hoping to avoid that.
By John Keston on Dec 7, 2009
One more quick update. According to http://www.humatic.de/htools/mmj.htm, it looks as though mmj.jar might be unnecessary after updating Java on Mac OS 10.6.2.
I’d love to hear confirmation of this, since I’m holding off on Snow Leopard until I get the all clear message from Ableton and Cycling ’74. Thanks!
By John Keston on Dec 7, 2009
i got the application to run after using the 0.9 version and deleting the java mmj lib.
although i still have no sound (when selecting the internal java synth) nor midi out via IAC. i’ll investigate further on this.
By nq on Dec 7, 2009
I’m running 10.5.8 on a Macbook and I’m not sure which combination of java libraries and software versions I should be using. Either the GUI doesn’t launch or I can’t select the right camera input (USB video class) to get iSight to initialize. Sometimes selecting the wrong camera mode initializes iSight but it freezes shortly afterward. In all scenarios I’m not getting any MIDI output no matter which device I choose. Thanks in advance for any help offered.
By Daniel on Dec 10, 2009
On 10.5.8 you should use GMS Beta v0.10 and also install the mmj.jar piece. That’s the same OS I’m using and it’s working fine. For the iSight I need to choose “usb video device video”. Let me know if that helps or doesn’t. Thanks! -John
By John Keston on Dec 10, 2009
John, is it possible that the 0.10 zip file actually includes the 0.9 version of the software?
By Daniel on Dec 10, 2009
No. I compared the versions on code.google.com just to make sure. But even if it was, v0.9 works the same on the Mac side. The only change in 0.10 is fixing a Windows bug.
By John Keston on Dec 10, 2009
Hey John,dled the GMS for Windows and tried running it in a VMware XP (as I am a linux user) and when I tried to open the executable I got an error saying the ‘lib” folder could not be found in c://blah/blah/exactly_where_the_folder_is_I-am-looking-right-at-it. so I gave up with that and booted up my “real” XP to see if there were any differences,but unfortunately the same result. Running on a 64 bit machine and the XP is sp3 I do believe.
By Mitch on Dec 11, 2009
Mitch, Grant here. I built the version you’re using on 32 Bit Windows 7, but tested it on both XP and 7. I haven’t tested it on a 64-bit machine but I think it should work properly. Can you send me exactly what the error is you’re getting (which lib is it complaining about?). In the meantime, I’ll try to test it on a 64-bit Windows OS.
The ONLY thing I can think of is that we might have a Quicktime discrepancy. In order to render video in Java via the processing lib, it makes a calls to the Quicktime wrapper that is built by Apple…if there is any 32/64 bit weirdness with that library we might have to make some modifications.
To be sure, using capture devices via Java/Processing in Windows is a bit trickier than it is on an Mac.
By Grant Muller on Dec 12, 2009
Here’s a screenshot of what happens.
http://i47.tinypic.com/732e1e.jpg
As you can see,it’s looking for something that is clearly there.
Just to make sure I just updated my jre as it was a bit old being I don’t boot up Windows often, so we can scratch that as a culprit.
I’m going to go crack open a 32 bit windows laptop I have to see how it works there.
By Mitch on Dec 12, 2009
Mitch, I’ve never seen an error like that before actually, and it doesn’t appear to be thrown by the JVM. I went digging around to see what 64-bit support for Processing looked like, and according to the 1.0.4 release notes (under known issues):
Currently no 64-bit support for any platforms. On some platforms, you’ll
simply need to replace the Java folder with the distribution with something
more suitable for your operating system.
That just means the that the JDK they supply is 32-bit, but it sounds like there hasn’t been a lot of 64-bit testing with Processing in general.
Let me know if it works on the 32 bit machine for you. If so, then at least we have the problem narrowed down and know where to start looking in order to fix it. Also…another comment coming right below this for Windows users…
By Grant Muller on Dec 12, 2009
Attention Windows Users, it looks like a little tidbit got left out of the release notes for the GMS
To use a video capture device with the GMS on windows, a Video Digitizer (VDIG) that is compatible with Quicktime must be installed. For details regarding why this is necessary check here:
http://processing.org/reference/libraries/video/index.html
There aren’t many free Windows VDIG libraries out there, but the one I used in testing (WinVDIG) is free and came from here:
http://www.eden.net.nz/7/20071008/
Make sure you install version 1.0.1 according to the Processing recommendation, later versions are quite buggy. Other than that the GMS should work exactly as it does on the Mac, keeping in mind that it doesn’t do much if you don’t have a capture device installed.
I’ll add these to the official notes ASAP.
By Grant Muller on Dec 12, 2009
No success at all here. Simply did nothing when launched. Tried the source which didn’t work either. Gave up after an hour.
MacBook with Snow Leopard; most recent Java, don’t know if that would matter.
By Joe McMahon on Dec 13, 2009
i have a microsoft vx-1000 webcam,running xp sp3 and i dont see a camera selectable where it should be…no cameras shown available
By nc on Dec 13, 2009
Hey Grant. It didn’t work on the laptop either. Not sure what’s going on. If you can think of anything to check from my side let me know.
By Mitch on Dec 13, 2009
Hi !
This is my very first attempt : http://www.youtube.com/watch?v=VMDVkXAQ7zA
I had some troubles setting up the IAC driver, but after downloading mmj it worked.
By Diego Ferri on Dec 13, 2009
Very nice. Thanks, Diego!
By John Keston on Dec 13, 2009
@nc, It sounds like you either don’t have WinVDIG installed (see the comment posted by me a few lines earlier) or you don’t have a capture device that is supported by Quicktime. Try installing (or reinstalling) WinVDIG and updating Quicktime to latest to be sure…
@Mitch, is it giving you the same error on the laptop, or just not coming up at all? If it’s not coming up at all you might want to check and make sure your Quicktime installation on that machine is ok. I had some buddies test this out on their machines and the only problem they ran into was a missing QTJava.zip file in the Quicktime installation. Reinstalling Quicktime 7 took care of it.
Let me know if that helps guys,
By Grant Muller on Dec 14, 2009
Yesterday I bought a webcam only to try this tool, and after installing WinVDIG under XP SP3 it works! And how it works! I love it and this night I will not go to sleep :) ‘Love you guys for this tool. Biggest suggestion for Nobel Price :)
Greets from Germany
By Jens on Dec 15, 2009
Hi there,
Jens again.
As I read above that there are no tests on an 64Bit Win7 system.
Well I have the RC (Ultimate) here and a quicktest is successful :) In the next weeks I will install a production Win7 64Bit Home Premium and then I can tell about all functions.
My System is an AMD Quadcore Phenom 2.2, 4GB RAM, Microsoft VX-800 cam and Ableton Live 8.0.8 with “loopBe1″-Software for transmitting the Midi data from GMS to Live.
Greets again
By Jens on Dec 15, 2009
Jens, I’d love to hear how it turns out with the 64-bit Windows 7. I have 32-bit here (Ultimate), but I’m sure the processing.org guys would be happy to hear that applications are working on 64-bit platforms without issue. Keep us updated!
By Grant Muller on Dec 16, 2009
Hi Grant,
sure, I will post my experiences with 64 Bit Win 7 as soon as possible :) CU, Jens
By Jens on Dec 17, 2009
Using a pixelated video I made and the GMS running through Reasons Thor synthesizer I was able to create this bit of audio. Enjoy!
By Dane on Dec 22, 2009
and I suppose it would help if I pasted the link into my comment as well, whoooops! How about you enjoy it now…
http://www.vimeo.com/8326704
By Dane on Dec 22, 2009
Hey!
First of all, congrats to John and Grant (and everyone else) for this tremendous app!
But, well, since I’m new in the MIDI world, I’m having some trouble making it work…
I’m running Win XP, and I got GMS working with my webcam thanks to the VDIG libraries. The problem is that I can’t get the signal to any DAW (Ableton Live, Nuendo, Reason, etc).
I tried via Loopbe, but nothing…
Could it be that on ‘CHOOSE MIDI OUTPUT DEVICE’ I get every option with the legend ‘Unknown Vendor’??
The only two options I got without it are ‘JAVA SOUND SYNTHESIZER’ and ‘REAL TIME SEQUENCER’ both of Sun Microsystems. But I don’t have those MIDI IN options in any DAW.
help, please?? :$
oh, and happy new year to everyone reading this!!
By Matias on Dec 30, 2009
Matias, it sounds like the LoopBE drivers are showing up as ‘Unknown Vendor’ in your system, I’ve had that problem with loopback drivers in the past on XP (Hubi’s). The RealTime Sequencer and Java Sequencer can’t be used by your DAW. You SHOULD be able to use the ‘Unknown Vendor’ ports, as long as you match up the port numbers between the GMS and the DAW.
If you’re having trouble getting the LoopBE stuff working, you might try out Maple:
http://www.maplemidi.com/Maple_driver.html
Its a four port loopback that I’ve used time and time again without any problems.
By Grant Muller on Dec 31, 2009
http://www.youtube.com/watch?v=m3JVE4zG2M8
My first “fool-around” with GMS.
I used an old web-cam the light of my mouse, my cell’s flashlight and my cigarette.
I fed Ableton Live via LoopBe in windows XP. I installed WinVDIG fist and everything worked well.
I just recorded my desktop with Camstudio.
Cant wait to use it with a proper cam and a planned setup!
By George Ramantanis on Jan 2, 2010
Hi there, i’ve donwloaded the windows version of GMS, and when i run the aplication, i got and alert saying:
Java virtual Machine Launcher: Could not find the main class. Program will exit.
I have already installed the latest version of java for vista.
How can i make it work? please!!
By Gonz on Jan 28, 2010
Hi Grant,
this is Jens again, you probably will remember my December posts here.
Now I had successfully migrated to Win 7 Home Premium 64Bit, Java 6 Update 17 (32Bit), and all the functions of GMS are working well with WinVDIG and Loopbe1 as the Midi transponder to Ableton Live 8.1. All on an AMD Phenom I Quad 2.2GHz, 4GB RAM, with Microsoft Lifecam VX-800 (cheap but cute).
Except one function like under XP: Choosing an External Midi Device will not exactly choose the clicked item but an item two rows up. But this isn’t something serious…
If you want me to test something special under 64Bit, please mail me.
Thanx again for this wonderful tool :)
Greets from Germany
By Jens on Feb 2, 2010
This is brilliant. Well done. Managed to get it working on my MacBook Pro using the 0.9 build under 10.5.8 (the .10 build didn’t not load) – enabling the IAC drivers in Midi Setup and setting up Logic Pro to receive the Midi data via the IAC. Really cool. Thanks alot for your efforts.
By Dan on Feb 25, 2010
I’m currently using the GMS in windows 32 bit, using maplemidi to sync with reason. I am currently unable to record in reason. I’ve tried to figure out how to get reason to record sound from the GMS but have been unsuccessful. Any guidance would be much appreciated
By Johanna on Mar 8, 2010
I’ve had a similar problem on the Mac. It’ll record fine win a controller, but only playback when routing a MIDI, not capturing the notes. I think I fixed it somehow by adjusting Reason settings. Has anyone solved this on Windows?
By John Keston on Mar 8, 2010
I’m trying to figure out how to get IAC out on osx.. right now all I can see are my interface outs, the real time sequencer and a java sound synth. Any clues on how to get it to show me IAC?
By George on Apr 17, 2010
Have you installed mmj – Midi for java on Mac OS X (http://www.humatic.de/htools/mmj.htm)? What you’re describing happens when it’s not installed.
By John Keston on Apr 17, 2010
So here’s a random question: is it possible to invert the tracking, so that it’s looking for the darkest spot rather than the lightest?
I’m thinking of daylight/outdoor applications; I tried using the GMS in my house during the day but the sunlight kept stealing focus (so to speak).
I suppose I could just mess with the code (I speak Processing, a bit) for my own purposes. But this might be a useful feature, at some point.
Just a thought. Love the app — it’s SO much fun. I’m thinking of using it to generate ambient locational soundtracks — like a constant soundtrack for a specific room, for example.
By Joshua Ellis on Apr 25, 2010
@Joshua It is not possible with the current version, but that would be an easy mod to the existing feature set. There is a way to invert the video, but the brightness tracking happens prior to any video processing, so it would not effect the pitches selected.
By John Keston on Apr 26, 2010
i cant geht mmj to be displayed in ableton OR gms. i read the readme and copied the two files meantioned there to /lib/java/ext, rebooted and it had no effect. please could you give a more detailed instruction how to set this up? especially the routing to ableton – you mentioned in some youtube-video that you told someone how ;-)
great work btw!
By phabe on May 3, 2010
@phabe With version 0.11 of the GMS and the latest java update on OS X you don’t need the mmj drivers anymore. Try removing them and then setting up external sync with the IAC drivers.
By John Keston on May 3, 2010
hey i can’t figure out how to import movies. i know they need to be in a .mov format but when i do a search to select it it will not show in the load screen. i know it is on my desktop but it still doesn’t show. and i noticed it only shows the ones with *.mov* but i still can’t get it to work. is there a way to make a folder to but it in or at least make it so the program reconises the .mov?
By dusty on May 9, 2010
i figured out that it may not show the icon for it so you might have to type the name in and then it will show in your list to pull the video in.
By dusty on May 9, 2010
Thanks Dusty. That’s good to know. What version and OS are you using?
By John Keston on May 9, 2010
Hey!First of all, congrats to John and Grant (and everyone else) for this tremendous app!But, well, since I’m new in the MIDI world, I’m having some trouble making it work…I’m running Win XP, and I got GMS working with my webcam thanks to the VDIG libraries. The problem is that I can’t get the signal to any DAW (Ableton Live, Nuendo, Reason, etc).I tried via Loopbe, but nothing…Could it be that on ‘CHOOSE MIDI OUTPUT DEVICE’ I get every option with the legend ‘Unknown Vendor’??The only two options I got without it are ‘JAVA SOUND SYNTHESIZER’ and ‘REAL TIME SEQUENCER’ both of Sun Microsystems. But I don’t have those MIDI IN options in any DAW.
+1
By whatseek.com on May 20, 2010
Am I being stupid? I cant find the readme.txt in the downloaded file….Looks really awesome.
By robbie on Jun 19, 2010
@robbie No, you’re right. It wasn’t in my latest upload for some reason. I have replaced it, so it should be there now. Thanks!
By John Keston on Jun 19, 2010