GMS

GMS IconThe GMS is a Gestural Music Sequencer developed in Processing by John Keston. The application samples video and displays it either normally or inverted so it looks as though you’re looking into a mirror. Each frame is analyzed for brightness, then the X and Y data of the brightest pixel is converted into a MIDI note. The X axis is used to select a pitch, while the Y axis determines the dynamics. As users move, dance, gesture, or draw in front of the capture device, notes are generated based on a predetermined scale. Currently the available scales are pentatonic minor, whole tone, major, minor, and chromatic, all of which can be dynamically selected during a performance.

GMS ScreenshotThe scale can also be adjusted using probability distributions. Each note in the twelve tone chromatic scale can be given weighted randomness from zero to one hundred. Notes set to zero will not play, and notes set to one hundred are most likely to play when the probability distributions are enabled.

Other dynamic controls include MIDI out channel, BPM, low and high octave, transposition, sustain, duration selection (manual or randomized with probability distributions), BPM adjustment, and note randomization. A “free” mode allows the durations to be manipulated by the mean brightness of the video input. Finally, four, simple video filter presets were recently added that can be applied by pressing shift + [1-4]. The application works especially well in dark lighting while using a light source to control the sequencer. More information on the controls and features can be found in the readme.txt file distributed with the download.

Important notes:
1. Read the readme.txt!
2. There are known bugs in this software (see step 1).
3. Please follow the steps in the getting started portion of the readme.txt before emailing me.
4. [This step is not necessary with Mac OS X 10.6.x Snow Leopard] To use the inter-application drivers (IAC Drivers) with the GMS to send MIDI to and receive external sync from external applications like, Ableton Live, or Reason, you must install the Mac OS X universal binary java Midi subsystem (mmj.jar).
5. Have fun and if you make something cool post a link in a comment.

*UPDATE* I have suspended working on the GMS due to other projects and time commitments. The most current version does not run on the latest version of OS X (I don’t know about Windows). If you’re interested in revising the source code it is available on GitHub. If anyone is interested in taking this on I will do my best to help and answer questions.

Download:
Download the GMS Beta from Google Code

View a video short about the GMS
Gestural Music Sequencer Documentary Short

Audio examples and more video produced using the GMS
https://audiocookbook.org/category/gms/

Acknowledgements:
Special thanks goes to Grant Muller for his extensive work on improving the RWMidi library for Processing.org. Grant was also an essential resource on external MIDI synchronization and optimization of the application. Thanks, also to Ali Momeni for his suggestion of using probability distributions and inspiring me to develop the software as a potential tool for Minneapolis Art on Wheels.

171 thoughts on “GMS”

  1. Absolutely tons of fun. As a recent upgrader to Snow Leopard, I also encourage you on the update. Most of all though, a giant thank you for such a fun piece of software!

  2. Pingback: Audio Cookbook » Blog Archive » Generative Sequence Driving MDA JX10 Emulator

  4. @Tony Snow Leopard version is one the way. I’ll need it in June for the Eyeo festival, but I should have it available for download by March.

  5. Pingback: Audio Cookbook » Blog Archive » Eyeo Festival June 27-29, 2011

  6. Pingback: Audio Cookbook » Blog Archive » Wanted: Roland Super Jupiter MKS-80 with MPG-80 Programmer

  7. Hi, can someone please help me?
    I am trying to get GMS to send midi notes to ableton live and I can’t seem to figure it out.
    I can’t select ableton as an external device obviously, so how can I get GMS to send the midi?
    do I need to use OSCulator somehow?

    any help is appreciated.

    thank you

  8. Are you on Windows or Mac? On the Mac you need to use the IAC drivers (see the readme). I won’t be much help on Windows, but there are several comments from Grant Muller above that explains the setup details for Windows.

  9. Pingback: Audio Cookbook » Blog Archive » Ostracon Debut Excerpt: Photon Coercion

  10. Hello John!

    I’m sure you’ve explained this to a few people before but I can’t figure out how to get the GMS up and running. Ideally I want to use it in ableton on my mac. Ive downloaded the program and the drivers…Im not sure if the drivers are installed yet…I don’t understand what exactly I download. Let me know if there is something simple I’m missing. I’m just stuck on step one of the readme.

    Is this too hard to explain via comment board? i can give you my email.

    Can’t wait to use it!!!! its a beautiful program and I’m only bugging you so I can join in on the fun.

  12. Pingback: The Audio CookBook : une base essentielle pour le sound-design | Silent MOVE

  13. Pingback: Unearthed Music » Blog Archive » Ostracon Debut Excerpt: Photon Coercion

  14. Pingback: Unearthed Music » Blog Archive » Eyeo Festival June 27-29, 2011

  15. Pingback: Unearthed Music » Blog Archive

  17. Pingback: Audio Cookbook » Blog Archive » GMS Generated MKS-80 Bassline

  18. Pingback: Audio Cookbook » Blog Archive » Photon Coercion Video

  19. Pingback: Audio Cookbook » Blog Archive » New Version of GMS for Mac Available

  20. Pingback: Audio Cookbook » Blog Archive » GMS Microtrack with MKS-80

  21. Pingback: Audio Cookbook » Blog Archive » Cassette Tape and Evolving MKS-80 Pad Driven by GMS

  22. Pingback: Audio Cookbook » Blog Archive » Eyeo Preparations Microtrack

  23. Howdy John,
    I’m a huge fan of GMS and have been using it for some time in conjunction with Ableton on OSX Tiger. I finally upgraded to snow leopard (10.6.3) I am pleased to see the release of v0.12. However, After following all readme instructions and putting mmj etc in place; I am receiving no midi in Ableton. If you (or anyone) can assist me It would be greatly appreciated!
    Here is a screen grab if it helps.

    http://s3.amazonaws.com/data.tumblr.com/tumblr_lnjc295qri1qioh91o1_1280.jpg?AWSAccessKeyId=AKIAJ6IHWSU3BX3X7X3Q&Expires=1309409423&Signature=YH30YtLSzbajD0l26V0nZlN3bXw%3D

    Thankyou.

  24. Hey Luke. I’m pretty sure you will not need the mmj stuff anymore with Snow Leopard. Try removing it and let me know what happens.

  25. Success! Thanks again John. It seems I like to make things harder than they need to be. Love this App!

  26. Pingback: Audio Cookbook » Blog Archive » Ostracon Release Show Segment 2

  27. Pingback: Audio Cookbook » Blog Archive » GMS Practice Phrase #3

  28. Pingback: Audio Cookbook » Blog Archive » GMS Practice Phrase #4

  29. Pingback: Unearthed Music » Blog Archive » Photon Coercion Video

  30. Pingback: Unearthed Music » Blog Archive » Ostracon Release Show Segment 2

  31. Pingback: Unearthed Music » Blog Archive » Cassette Tape and Evolving MKS-80 Pad Driven by GMS

  32. Hey John- I’m finally getting around to installing GMS on my new laptop. running snow leopard with GMS v0.12 and i’m not able to see the midi data in Live. what settings do i need to adjust to send midi to live? i can get audio when i select ‘java sound synthesizer’ the only other option i see is ‘real time sequencer sum’ and that doesn’t work either.

    tips on syncing to live would be appreciated too.

  33. Hi …

    Thanks for making this available!

    I’d love to get it working on Windows XP but my webcam is not being recognised (Microsoft VX3000) – works fine with other apps.

    Is this a known issue, and has anyone found a fix?

    I’m hoping to try this out at an event on 23rd Sept – giving out LEDs and light pens maybe to see what we can create together.

    Cheers
    Richard

  34. Hi there,
    This looks great…unfortunately my built in iSight isn’t showing up and I’m unable to select USB VIDEO CLASS VIDEO.

    I’m able to send midi but it’s just a straight pulse due to lack of video input.

    Any ideas? I’d love to fool around with this.

  36. Hi John,
    Fantastic piece of software. I’m using it for a college project at the moment, but running into a little trouble. I filmed a modern dancer holding an white LED keyring, with the intention of these movements triggering sound in my DAW. I did some work on the image to try and make the light source be the only source of white on screen. But the GMS isn’t seeing it. Is there a certain number of pixels the white must inhabit before being read? Thanks!

  37. Hi Brendan. There is no maximum number of pixels. I imagine something else is wrong. Are the notes triggered when you hold the LED closer to the camera? You can hit tab to turn on a blue square that outlines the pixels that are being detected as the brightest (this is also on by default so you’re probably seen it). Either there is a brighter source than the LED in the frame (use tab to see it), or something else is not working. Let me know what you find out. Thanks! -John

  38. Hi John,
    Thanks for the response. I figured it out, but don’t know the exact issues involved. The original video I was using was in 1080. I had tried cropping it and the resolution was in 960, and the GMS was tracking something in the lower left, though it was only black space. Converting the resolution to 480 did the trick and the GMS was tracking the light source accurately. A further reduction to 240 resulted in the tracking of black space to the top right, opposite where it had been tracking when resolution was 960. So, odd, but at least my I’m getting the results I needed!

  39. Pingback: D.O.G. (DKO/Ostracon/Ghostband) Debut | Audio Cookbook

  40. Pingback: Roland MKS-80 Lead | Audio Cookbook

  41. Pingback: Gestures « Brendan Rehill

  42. Hi John,
    I completed the project I was doing with the GMS and thought you might be interested to hear. As I mentioned above, I collaborated with a contemporary dancer, recording audio of her performance, and also shooting footage of her with lights attached for the GMS to track. The piano parts are 4 separate parts from GMS MIDI information and are also triggering a sampler with the performance audio. Other than that, there’s just a Reaktor sound to create the drum pulse, and some Soundhack plug-ins.

    http://brendanrehillaudio.wordpress.com/2012/01/31/fruition/

    Thanks for such a great app!

  43. FYI: Using Lion, (OSX 10.7.3). No apparent MIDI output from IAC Bus to any program (Ableton, Garageband). Would love to know if/when you update this valuable tool. Thanks.

  44. @bill d. I have not personally installed Lion yet, but when I upgraded to Snow Leopard very little was necessary to get the GMS working. It may be as simple as recompiling the GMS for Lion. This is not something I will get to anytime soon (will require an OS update I’m not ready for), so if you want to take a crack at re-compiling yourself the source is available here:

    http://code.google.com/p/gestural-music-sequencer/

  45. John, thanks for the reply.

    Looked through the Google Code page but could not find the source code, except of course the svn repository, which I cannot access. So, I used the source code included with the app download. I am not a coder (obviously) but figured I’d run it through XCode and see what happens. Well, the build succeeded, but running the app I get a blank window. So much for that. Anyway, guess I’ll wait for your (hopefully eventual) update.

    Thanks again for this great tool!

    bill

  46. @bill d. Hmm… Well there might be a little more to it than I thought. I am assuming that you have manually installed Java on your Mac? Apparently Lion does not come with Java pre-installed like former versions of OSX.

  47. JOHN: Success! It wasn’t the Java (installed the recommended fix then removed it). I had a look at the errors in Console and noticed that there were some old files in the InputManager directory, so I deleted them and IAC data is flowing to LIVE. Will test and let you know how it goes….

    One noticeable issue: GMS is rockin’ the CPU for sure. Currently bouncing between 100 and 130 % in Activity Monitor.

    Thanks again!

  48. @bill d. Splendid! Good to hear that it works on Lion. Did you re-compile or are you using my Snow Leopard binaries? As far as the CPU usage goes, it might be time to try and optimize the brightness tracking algorithm, but I haven’t had any trouble on my aging Core 2 Duo.

  49. Hi John,
    I’m enjoying using GMS. The only problem I’m running into is trying to run it using Ableton. I am using Lion and have read the earlier posts about that. I don’t have much experience with programming but would like to know if you have any suggestions on how to make the IAC data recognizable to Ableton. I do have MAX for Live and wondering if there is a way to patch through that also. I would like to use GMS for a program I am doing in July! Thanks for your input.

  50. @Rob I’m pleased that you are interested in GMS. Although, without a Lion machine to test it on I am at a loss beyond suggesting that you install Java and look for the “old files in the InputManager directory”. In the meantime I will try and find someone with Lion who can help test.

Comments are closed.