I stopped into Best Buy yesterday and picked up something I had been considering toying with: a LEAP Motion. This is an interesting little $79 device which turns the airspace at your computer into a live space for interaction. It recognizes all ten fingers on your hands, differentiates between left and right hand, open and closed, and can read the pitch, roll, and yaw of your hand, pinch and swipe gestures and more, all in meatspace.

I haven't yet torn it down to see what type of imagine sensors it is using to map the space around it. Of course, the folks at SparkFun already have, however, so feel free to look at their article if you want to see what's in this tiny box.

It appears to be sensitive to IR light, similar to how the Kinect is sensitive to too much sunlight in a room. I turned on a compact fluorescent to take a photo and this popped up:

Installation

For some reason, the LEAP Motion uses a proprietary USB connector. It's normal USB on one end and then what looks like a mini and micro next to each other on the other end. I have no idea why they did this, as there's no electrical reason given the standard single USB connector on the USB side. They did think to provide two lengths of USB cable, so that's a help. However, if the LEAP Motion is something you intend to travel with, you'll need to make doubly sure you have the right cable, as you can't just borrow one from a friend or pick up a spare locally.

CORRECTION: This is not a proprietary connector. It's a USB3.0 Micro-B connector -- the first I've seen in the wild. Still, good luck finding a cable in a store. At least it's a standard cable, however. Also, this suggests that plugging into a USB 3.0 port would be a good idea.

As a matter of taste, I prefer my USB cables on my desk to be black so I don't see them. White cables are distracting. Also, this cable is a bit stiff and tends to pull the LEAP Motion off-center.

Anyway, once plugged in, the firmware updated. Unfortunately, that bricked it and afterwards, it wasn't recognized. It showed up as "Westbridge" under "other devices" in the device manager. I searched on this, and ended up on the LEAP Motion site where they had a very helpful troubleshooting page and firmware reset utility to fix this exact problem.

Once I saw "Leap Dev Kit" in the device manager, I knew it was working. It is interesting that it shows up as a "Dev Kit" and not a consumer device name. Perhaps the firmware fix download resets it to some sort of dev status, or was intended for developers.

As an aside, the LEAP Motion appears to be referred to both as "Leap Motion" and "LEAP Motion" on their website and by their employees. Not sure which is correct, but "Leap Motion" appears to be what shows up all over the PC after the installation. I'm surprised their branding folks didn't go crazy on them for mixing the two.

Testing with the Visualizer

Now that the device was plugged in and recognized, and showing the green power LED, I wanted to test it out. I right-clicked the LEAP Motion icon in the tray and selected "Visualizer…" so that I could test the basic operation.

This launched the "Leap Motion Visualizer" program. Here you can see a screen shot of the visualization of a single hand. It works with both hands (but I needed one to hit print-screen), and also identifies the position of each finger. This was with zero training or setup other than completing the basic install.

Cool, so it's working. For grins, let's see if I can control Windows 8.1 with it.

Controlling Windows

LEAP Motion includes an app store. I hate that they have a separate app store with their own payment model, DRM, and more when each supported platform has its own story anyway. I would have preferred they listed their app in the Windows Store as a desktop app instead.

One of the apps in the app store is "Touchless". This is a free app, but you need to create a LEAP Motion store account to download it. (Really. I thought I bought a device, not an ecosystem. It makes me nervous to have yet another company with my account info and, as you'll see later, my payment info.)

Once I created an account (with a fake birthday - why is birthday required?) and clicked the button to get the app, I was then able to see it in the Airspace client. Note that the apps don't download from the store, you need to use the Airspace client app. However, the airspace client includes no provision for interacting with the store. Have I mentioned my annoyance at having yet another store?

I launched the Airpspace client:

Which opened a full-screen window with the "Airspace Home", complete with all of the installed apps. It immediately started downloading the Touchless app I purchased on the web.

The touchless app is, by default, a single-point mouse replacement app. This is interesting, but not super useful. If you've ever tried to use your finger as a mouse using Kinect, you know there are more efficient ways to interact with your PC. I was more interested in the multi-touch side. To access multi-touch, right-click the Touchless tray app and select Interaction->Advanced.

Now the app sends each finger as discrete points. I test this out, I launched one of my favorite multi-touch Windows 8 apps: MorphWiz.

It takes some practice, but I was able to get the app to recognize from one to five touch points. I should be able to get it to recognize all ten, but I didn't have time to noodle with that much more. Note that MorphWiz was written without any knowledge of the LEAP Motion - it just saw it as touch input.

Not all apps worked like this, however. For example, I was unable to get the "Vintage Synth" app to recognize any of the finger inputs. In the future, I'll try the LEAP Motion with some of my own Windows 8.1 code to see if it's a question of how you use the pointer/mouse APIs.

LEAP Motion also have an interesting free app named "Kyoto" which lets you interact with a seed, tree, water, moon, meteors and other stuff while playing music reminiscent of Minecraft. It's free and you'll find it amusing for a few minutes.

Controlling a Synthesizer

When I bought this, I had no intention of using it to control Windows. I'm really happy with my Logitech Touchpad in that capacity. What I'm really interested in is how this could work for musicians and performers as another way to control dynamics.

This seems to be a scenario of interest to the Leap folks as well, as one of the bundled Airspace store apps, Lotus, is all about music performance control. Once you poke the eye you get offered a set of musical experiments that you select by pointing with one, two, three, or four fingers at once. All are worth a try, but I find myself coming back to the one with a head in it. Using two hands to control, you can get some really interesting performance capabilities. For example, by closing your fist, you close the filter. By spreading your hands out, you increase the stereo width. Lots of other effects as well. Very neat.

I searched their store for "MIDI" to find some MIDI controller apps.

The prices are high. That's one problem with a niche market and a small app store. $30 is almost unheard of in app stores these days, especially for controller software that requires $80 in investment in specialized hardware. I didn't both with the AeroMIDI app, but did decide to try out the Geco MIDI app. It's relatively expensive as well ($10), and had no trial, but I was able to find decent information on its operation on their web site. Besides, $10 is like a few coffees. Purchased.

This is installed like the other store apps. As this one requires payment, you have to have a credit card set up ahead of time. I was hoping to use PayPal, but credit card is the only method of payment accepted.

One installed, go ahead and run it from the Airspace shell. Be sure to turn off any other apps (such as Touchless) first, as they will interfere with each other.

I ran the app, and went into the "Document Settings" dialog to configure the MIDI connection. The configuration for which messages are assigned to which gestures, and which device to use, are all stored as a document.

(note the branding in the title bar. Another confused company: GECO in the app, Geco in the listing).

I wanted to control the filter cutoff of my Moog Slim Phatty. So, I pulled up the manual and looked up the MIDI CC (Continuous Controller) messages. The Sub Phatty supports 14 bit (high resolution) MIDI, but GECO does not. So, I had to go with the 0-127 value 7-bit classic MIDI.

The Filter Cutoff Frequency is CC #19. The value range shown in the table is for 14 bit. For 7 bit, it's 0-127. I then set this up in the GECO app, assigned to the vertical position of my open left hand.

One that is set, all I needed to do was play a note on my Sub Phatty with my right hand and then use my left hand to control the filter cutoff. Despite the lack of 14 bit MIDI support, it sounded great. The Sub Phatty does a decent job of interpolating the 7 bit MIDI messages to avoid obvious stepping with filter cutoff (I could hear stepping with resonance, however).

I also mapped resonance to roll inclination. That was a bit awkward as your hand needs to be turned almost upside down to get the full range, but not too bad once you got used to the behavior at the extreme values.

I pulled up an older sequence in Cubase with the intent of using both hands to control the dynamics while the sequence plays. However, as GECO is not yet a VST, and you can't share a MIDI output between two programs at once on Windows, I was unable to use both GECO and Cubase to control the Sub Phatty. A workaround here is to use one over USB and the other over regular DIN MIDI, but I did not get a chance to try that out. Instead, I did the gesture control with my left hand and playing with my right.

Further Thoughts

This is a pretty neat unit, especially for the price. Apps are (for a modern app store) overpriced, but there's a developer program you can sign up for to write your own apps. Given that it requires a custom driver, this won't work on Windows RT, but I'll check it out in Windows 8.1.

The LEAP Motion was incredibly smooth and well-tuned right out of the box. Like any airspace-gesture technology, it required a steady hand for things like touching points on the screen. In that way, it's a bit of a novelty. As an additional vector for performance control, however, it really shines. It's like the Roland D-Beam from their V-Synth and others, with many more types of recognition other than just distance.

Costing not much more than a decent mouse, the LEAP Motion is a great device to add to your desk, especially if you like to experiment with new ways of interacting with your computer and/or instruments. My advice: get one if you have other creative uses in addition to normal UI control (musicians, folks playing with NUI, etc.). If you're looking to buy one just to replace a touch screen on Windows or OSX, I don't think you'll be happy in daily use.

I didn't have a chance to record a video today, but once I learn more about how to incorporate the LEAP Motion into my work, I'll post some examples. For now, here's some very short audio: right hand is on the Sub Phatty, left hand is using the LEAP Motion to control filter cutoff with open hand Up & Down Position (1/19/1%/100%) and resonance with open hand Roll Inclination (1/21/-1%/-). On the keyboard, all I did was hold down a single key - all the dynamics are coming from MIDI control via the LEAP Motion.

http://soundcloud.com/psychlist1972/sub-phatty-leap-motion-filter

The above screen shot shows status of hand height (left bar) and roll (second bar).