Tokbox Ukulele: Play an imaginary instrument via webcam.
If you want to be in a band, but don't have an instrument and your friends are 500 miles away, use Tokbox Ukulele. We allow users to play an "air instrument" with friends via webcam. The application tracks hand motions using a webcam to determine which chord is being played. Once the chord is determined, it's sent over the network to all participants, allowing for collaboration with your 'virtual band'.
We use the Tokbox API for video capture and streaming, then draw the video to an html5 canvas for processing. Applying the Lucas-Kanade optical flow method and some color normalization, we track finger and marker positions. Normalizing these finger positions, we can classify to the nearest chord and play it to all participants whenever we detect a 'strum'.