Kiandra IT recently organised a hackaton titled Touch and Gestures. As you might guess it ended up being mostly about gesture enabling different things. Touch has been around for so long that most of us perceive it as something normal. While gestures are not new either there are some new exciting developments in that area going on, I am talking about https://www.leapmotion.com/. There is a lot of potential there. And of course there is the well known Microsoft Kinect.

There were four teams and the guys worked on a windows management system using Leap Motion, Kinect enabled story wall, a Leap Motion enabled web dashboard and of course the thing I am going to write about here.>

I teamed up with a colleague to Kinect enable my 3D Snooker game.  Here is a quick video of what we came up with (this is a slightly tweaked version, but the core remains the same).

So how does one go about using Kinect in a MonoGame?

Initializing Kinect is easy:

KinectSensor _sensor = KinectSensor.KinectSensors.First();
_sensor.SkeletonStream.Enable(); // Enable skeletal tracking
_sensor.Start();

First we get a reference to Kinect sensor, then enable tracking of data we are interested in and start the whole thing. In our case we opted to enable SkeletonStream as we wanted to track position of our hands.

Next we need to handle Kinect events. We opted to use RX extensions and observe the event stream that way. At this point my colleague took over as he is a bit of a gun when it comes to using RX extensions. Here is the code:

var skeletonData = new Skeleton[_sensor.SkeletonStream.FrameSkeletonArrayLength];
var events = Observable.FromEventPattern<SkeletonFrameReadyEventArgs>(_sensor, "SkeletonFrameReady").Publish();

var skeletons = events
    .Select(e =>
        {
            using (var skeletonFrame = e.EventArgs.OpenSkeletonFrame())
            // Open the Skeleton frame
            {
                if (skeletonFrame != null)
                // check that a frame is available
                {
                    skeletonFrame.CopySkeletonDataTo(skeletonData);
                    return skeletonData.LastOrDefault(x => x.TrackingState == SkeletonTrackingState.Tracked);
                }
                return null;
            }
        })
    .Where(x => x != null);

var handVectors = skeletons.Select(GetHandsVector);

// calculate angles and speed

events.Connect();

In a nutshell, we subscribed to SkeletonFrameReady event and whenever we had a skeleton frame in Tracked status used that data to calculate the vector between left and right hand.
The vector was calculated simply by substracting right hand’s position from left hand’s position, like so:

private static Vector3 GetHandsVector(Skeleton sk)
{
    var rightPosition = sk.Joints.First(x => x.JointType == JointType.HandRight).Position;
    var leftPosition = sk.Joints.First(x => x.JointType == JointType.HandLeft).Position;
    return ToVector3(leftPosition) - ToVector3(rightPosition);
}

private static Vector3 ToVector3(SkeletonPoint leftPosition)
{
    return new Vector3(leftPosition.X, leftPosition.Y, leftPosition.Z);
}

As you move your hands the vector changes and we used it’s angle relative to X and Y axis to rotate the cue.

We also wanted to use gesture to trigger a shot and the idea was to track the speed as you move hands together, then, when hands are close enough to each other we would take that speed and trigger the shot. This is something that still doesn’t work quite well. We were dealing with two problems here:

  • How to stop the cue from rotating when you want to make a shot and
  • Getting an accurate speed reading.

After the hackaton (actually today) I solved the first problem by cheating a little bit. If you looked at the video above you probably noticed that I was holding a wireless mouse. I decided to use left mouse click to lock rotation in. Right mouse button click is used to switch between moving the cue ball (when allowed) and rotating the cue. I’ve been told that Kinect for Windows allows you to track palm movements, so when you make a fist that is treated like a click. I am using Kinect for XBox 360 here and this one does not support such funky things. So the trick with a wireless mouse is here to stay until I get my hands on Kinect for Windows.

The problem with speed remains though. So how do we calculate the speed?

// calculate distance between hands
Distances = handVectors.Select(x => x.Length());

// determine if there was a shot played
const float NEAR_ZERO = 0.3f;
var distanceTime = Distances.TimeInterval();
var speeds = distanceTime.Zip(distanceTime.Skip(1), (a, b) => Math.Abs((a.Value - b.Value) / (a.Interval.TotalMilliseconds - b.Interval.TotalMilliseconds)));
Shoots = speeds.WhereOther(Distances, x => Math.Abs(x) < NEAR_ZERO);

First we track the distance between hands and the time intervals at which frames are coming in. The speed is the distance traveled divided by the time it took to travel this far. Just as your physics teacher told you 😉

We only report speeds to the game engine once hands are close enough to each other. Like I said, there are still some issues and this method is somewhat unreliable. But I am yet to have any new ideas.