hand tracking instrument — play music with your hands_

A hand tracking instrument you play in mid-air, in a browser tab

A violin needs years before it sounds like anything and a piano needs to be in the room. This instrument needs a webcam. The camera watches your hands, a landmark model returns 21 points per hand (four joints on each finger plus the wrist), and those points are read as musical intent: which fingers are out, where they are, how fast they move, whether the palm is tilted or pinched. A fifteen-voice synthesiser does the rest. There is nothing to install, no account and no extension.

It is closer to a theremin than a keyboard, because you play in space rather than against a surface. Unlike a theremin it can be locked to a scale, which removes the thing that makes gesture instruments frustrating: with C Minor Pentatonic selected, C Minor Pentatonic is all the air contains.

Six play modes for different music

Most webcam music demos implement one idea. This one gives you six, because different music wants different gestures. Jam Band puts a backing band behind you: you choose the chords and solo, it keeps time. Air Harp turns the frame into strings you sweep through, plucky and immediately satisfying, and the best mode for a beginner. Air Keys puts a keyboard strip under your hands for held notes and chords. Chord Conductor splits the work the way an accompanist does. Theremin is continuous pitch with optional scale snapping. Drum Pads gives you nine synthesised drums to punch in the air. Air Harp and Air Keys follow all ten fingertips, so you can hold a chord with one hand while the other picks out a melody; up to sixteen notes sound at once, and past that the oldest note is faded out to keep the audio clean.

How the Jam Band follows your hands

Pick a style and the band knows its part. Lo-fi (82 BPM) is a lazy swung beat with electric piano chords, House (122 BPM) is four-on-the-floor with off-beat bass and piano stabs, Synthwave (100 BPM) is a big backbeat, a pumping octave bass and a warm pad, and Funk (104 BPM) is a syncopated kick, ghost-note snare and short guitar chops. The tempo slider runs from 60 to 170 BPM and you can mute the drums, bass or chords separately.

Your left hand picks the chord with the same finger counts as Chord Conductor: one to five fingers, the thumb included, for chords I to V of your key, 🤘 for VI and 🤟 for VII. The band only plays chords that belong to the key, so C major gives C, Dm, Em, F, G, Am and Bdim, and a pentatonic or blues scale borrows the chords of its parent major or minor key. Height adds colour: hold the hand low for a plain triad, in the middle for a seventh chord and high for a ninth. Where a ninth would clash, as on III and VII of a major key, the chord stays a seventh. The ladder on the left of the stage always shows which chord each finger count will play.

With Snap to beat on, a new chord waits for the next beat and the drums, bass and keys change together, so a slightly late gesture still sounds deliberate. Every voicing is moved to the inversion closest to the previous chord, the way a keyboard player would, so changes glide instead of jumping around. Hold chord keeps the last chord going when you drop your left hand; switch it off and the band stops a second after your hand leaves.

Your right hand plays the melody. Pinch thumb and index together and a note sounds on the lane nearest your fingers, a little over two octaves of your scale with the notes of the current chord highlighted. Move up and down while pinching to play a line; release to stop. The melody uses whichever of the fifteen instruments you pick, and its notes snap to the next sixteenth so runs stay in time. A right fist is a breakdown: the backing band is pulled through a closing low-pass filter, and opening the hand drops it back in.

Loop your own layers

The looper records only what your hands play, not the backing band, for 1, 2 or 4 bars at the current tempo. Press L, the ● Loop button, or hover a fingertip over the ● LOOP air button at the top of the stage for about two thirds of a second. Recording starts exactly on the next bar; when nothing is playing yet you first get a one-bar count-in. The take is captured sample by sample in an AudioWorklet, so it lands on the beat grid, and it starts looping the moment it ends. Press again to add another layer, up to six; U removes the last one and Clear removes them all. The tempo is locked while loops exist, because a loop is audio and would drift at another tempo.

Loops keep playing when you switch mode, which is the fun part: loop eight bars of drums in Drum Pads, add a chord loop in Air Keys, then play Air Harp over both. The optional metronome click goes straight to your speakers and never into a loop or a recording. Loops live in memory only, so they are gone when you close the tab; press Record to keep the performance as a video and WAV.

How Chord Conductor works

Your left hand chooses the chord. One to five extended fingers, the thumb included, select scale degrees I to V; index and pinky together (🤘) give VI; index, pinky and thumb (🤟) give VII. Tilting the left hand inward makes the chord major, outward makes it minor, with a dead zone in the middle so a relaxed hand does not waver. Your right hand shapes it: one to four fingers step through the plain triad, its first inversion, a major or minor seventh and a dominant or diminished seventh. Palm height sets the volume, tilt opens or closes the filter, and sticking the right thumb out drops the chord an octave. A closed right fist silences it and a closed left fist stops the chord. With Left hand only ticked, voicing, volume and filter stay fixed.

Both hands are tracked at once. Roles follow where the hands appear on screen: the hand further left in the mirrored view is your left hand. With only one hand visible the app uses the model's own left or right label. The image is mirrored by default so that moving right moves the on-screen hand right; you can switch Mirror off in the panel.

Scales and keys: no wrong notes

Pick a key and a scale and every string, key and theremin position is locked to it. There are twelve scales in all twelve keys: Major Pentatonic, Minor Pentatonic, Major (Ionian), Natural Minor, Harmonic Minor, Dorian, Phrygian, Lydian, Mixolydian, Blues, Whole Tone and Chromatic. To jam along with a track, set the key of the song and choose Major or Natural Minor to match; Minor Pentatonic and Blues are the safest choices for soloing over almost anything. The octave slider (1 to 6) moves the range, and you can have 5 to 18 strings or keys in up to three rows.

Fifteen synthesised instruments, no samples

Every note is built live with the Web Audio API, using the synthesis method that suits the sound. Pads, strings, choir, brass and leads are subtractive: detuned oscillators through a resonant filter with its own envelope. The electric piano and vibraphone are two-operator FM, which is how those sounds were made in the 1980s. The struck piano and music box are additive with deliberately inharmonic partials. The nylon guitar and concert harp use Karplus-Strong: a burst of noise recirculating in a delay line one wavelength long, losing a little energy each pass, which is a real physical model of a vibrating string, rendered per note and cached. The organ is a drawbar stack and the flute is filtered noise. Switch sounds with [ and ].

• Sustained: Warm Pad, Bowed Strings, Church Organ, Choir Aah, Flute, Brass Section
• Plucked and struck: Concert Harp, Nylon Guitar, Struck Piano, Music Box, Vibraphone, Electric Piano
• Synth: Bright Lead, Retro Square, Sub Bass

The sound panel has volume, reverb, delay and glide. There is no MIDI input or MIDI export here; this instrument is about the hands, and it records audio and video. For MIDI, the online virtual piano records what you play as a .mid file.

Settings worth trying

• Best first take: Jam Band, Lo-fi, Vibraphone, C Major Pentatonic. Show 1, then 4, 5 and 4 fingers with your left hand, and pinch a few notes with your right.
• Easiest without a band: Air Harp, Concert Harp, C Major Pentatonic, 2 rows, reverb up. Sweep slowly with one hand.
• Build a track: Drum Pads for a 2-bar loop, then Air Keys with Warm Pad for a second layer, then Theremin on top.
• Cinematic: Air Keys, Warm Pad or Choir Aah, D Dorian, reverb 70, delay 40. Hold three fingers still.
• Jam over a track: Chord Conductor, Electric Piano, the song's key, right hand high for the chorus.
• Sci-fi soundtrack: Theremin, Bright Lead, scale lock off, glide 80. Slow horizontal drifts.
• Synthwave lead: Jam Band, Synthwave, Bright Lead, A Natural Minor, hand high for ninth chords, fist for the breakdown.

Recording video and WAV

Press Record and the page captures two things at once. A WebM video of the canvas (camera view, hand skeleton, glowing strings, ripples and the heads-up display, exactly as you see them) with the synth mixed into the audio track: the file you post. And an uncompressed 16-bit stereo WAV straight from the synth, with no lossy compression: the file you drop into a DAW. Switch the record button to AUDIO ONLY if you only want sound. After stopping you can preview the take and download both files.

WebM is what browsers can record natively. It plays in Chrome, Edge and Firefox and uploads fine to YouTube, Instagram and TikTok, which convert it on their side; some older iOS apps do not preview it, so convert it to MP4 if you need to. Recordings are held in memory until you download them, so a few minutes is comfortable and very long takes on a low-memory device may stall; download an important take before starting the next. Recording in fullscreen looks better because the canvas is larger. What you play is your own work: no watermark, no attribution required, commercial use fine.

Where the video goes: nowhere

The camera stream goes into a canvas and a model running inside your browser tab. It is not sent to a server. The band, the looper and every sound are generated in the same tab. The only recording that exists is one you start yourself, and it stays in memory until you download it or close the tab. In mouse / touch mode the camera is never opened at all. The page never asks for your microphone, which is also why the recorded audio has no room noise. Mode, instrument, key, scale, octave, string count, effect levels and display toggles are saved in your browser's local storage. The site counts page views with cookie-free analytics. After the first visit the model and code are cached and the sound is generated locally, so it largely works offline; the first visit needs a connection to fetch the model.

No webcam? Play with a mouse or touchscreen

Press Play with mouse / touch on the stage and the pointer takes over from your hands. It drives the same virtual hand the camera would, so every mode, the looper, the band and the recorder work the same way. A bar of chord buttons under the stage stands in for your left hand: I to VII pick the chord (each button shows its name in your key), triad / 7th / 9th replaces the hand-height zones of the Jam Band, and in Chord Conductor major / minor and voicing 1–4 replace palm tilt and right-hand fingers. Press a chord again to let go of it.

• Jam Band: pick a chord and the band starts; hold the mouse button or a finger on the stage and move up and down to play the melody lanes.
• Air Harp: click a string to pluck it, or drag across the strings to strum.
• Air Keys: hold a key; touching lower on the key plays louder.
• Chord Conductor: pick a chord, then hold on the stage to sound it; higher on the stage is louder.
• Theremin: hold and move; left and right is pitch, up is volume.
• Drum Pads: tap a pad, or drag across pads for a roll.

A mouse gives you one "finger", so ten-finger polyphony and two-handed gestures still need the camera. On a phone the chord buttons and the stage can be used with two thumbs at once. Press 📷 use camera to switch to hand tracking at any time.

Hand synth, gesture synth or theremin?

People search for this kind of instrument under many names: hand synth, hand gesture synth, hand synthesizer, hand sign music. They all mean the same idea: a synthesizer whose notes and controls come from the shape and position of your hands instead of keys and knobs. A classic theremin does it with two antennas and electric fields, and only for pitch and volume. A camera-based hand synth sees the whole hand, so finger counts can choose chords, a pinch can start a note and a fist can mute the band. That is why this page has six modes instead of one.

Getting good tracking and low latency

Tracking quality is mostly a lighting problem. Face a window or lamp rather than having one behind you, avoid a background the same colour as your skin, and keep whole hands inside the frame: once fingertips leave the edge, the model guesses. If jitter triggers notes you did not mean, raise Smoothing (steadier, slightly later) or lower Sensitivity; if gentle movements are ignored, raise Sensitivity. Some delay is unavoidable because the camera has to capture a frame and the model has to read it before a note can start. Good light, fewer open tabs competing for the GPU and wired headphones instead of Bluetooth keep it as short as possible. Fullscreen (F, leave with F or Esc) hides the page around the instrument and keeps the screen awake while you play.

Camera, sound and browser problems

If nothing happens on Start camera, the browser blocked the camera or another app is using it; while you sort that out, Play with mouse / touch works without it. Allow the site via the camera icon in the address bar, close any video-call app, and on macOS check System Settings → Privacy & Security → Camera, then reload. If the hands track but you hear nothing, click the page once (browsers only start audio after an interaction), check the volume and that the tab is not muted. Chrome and Edge run the model fastest because they can use the GPU; Firefox and Safari work too. The page needs https. On a phone or tablet use the front camera in landscape, propped up so both hands are free, and on older phones stick to Air Harp or Theremin with fewer strings.

More music tools

For chords with simple finger counts, try Simple Hand Tracking Chords or the 4 Chord Song Player. Build a beat to play over with the free online drum machine in Drum Loop Studio, which exports seamless loop WAVs, edit your recordings in the audio editor, or browse all the free music tools. It is free, with no account, no ads and no watermark.