Category Archives: Tickle my brain

Playing with Kinect

Kiandra IT recently organised a hackaton titled Touch and Gestures. As you might guess it ended up being mostly about gesture enabling different things. Touch has been around for so long that most of us perceive it as something normal. While gestures are not new either there are some new exciting developments in that area going on, I am talking about There is a lot of potential there. And of course there is the well known Microsoft Kinect.

There were four teams and the guys worked on a windows management system using Leap Motion, Kinect enabled story wall, a Leap Motion enabled web dashboard and of course the thing I am going to write about here :)

I teamed up with a colleague to Kinect enable my 3D Snooker game.  Here is a quick video of what we came up with (this is a slightly tweaked version, but the core remains the same).

So how does one go about using Kinect in a MonoGame?

Initializing Kinect is easy:

            KinectSensor _sensor = KinectSensor.KinectSensors.First();
            _sensor.SkeletonStream.Enable(); // Enable skeletal tracking

First we get a reference to Kinect sensor, then enable tracking of data we are interested in and start the whole thing. In our case we opted to enable SkeletonStream as we wanted to track position of our hands.

Next we need to handle Kinect events. We opted to use RX extensions and observe the event stream that way. At this point my colleague took over as he is a bit of a gun when it comes to using RX extensions. Here is the code:

            var skeletonData = new Skeleton[_sensor.SkeletonStream.FrameSkeletonArrayLength];
            var events = Observable.FromEventPattern<SkeletonFrameReadyEventArgs>(_sensor, "SkeletonFrameReady").Publish();
            var skeletons = events
                .Select(e =>
                        using (var skeletonFrame = e.EventArgs.OpenSkeletonFrame())
                        // Open the Skeleton frame
                            if (skeletonFrame != null)
                            // check that a frame is available
                                return skeletonData.LastOrDefault(x => x.TrackingState == SkeletonTrackingState.Tracked);
                            return null;
                .Where(x => x != null);

            var handVectors = skeletons.Select(GetHandsVector);

            // calculate angles and speed


In a nutshell, we subscribed to SkeletonFrameReady event and whenever we had a skeleton frame in Tracked status used that data to calculate the vector between left and right hand.
The vector was calculates simply by substracting right hand’s position from left hand’s position, like so:

        private static Vector3 GetHandsVector(Skeleton sk)
            var rightPosition = sk.Joints.First(x => x.JointType == JointType.HandRight).Position;
            var leftPosition = sk.Joints.First(x => x.JointType == JointType.HandLeft).Position;
            return ToVector3(leftPosition) - ToVector3(rightPosition);

        private static Vector3 ToVector3(SkeletonPoint leftPosition)
            return new Vector3(leftPosition.X, leftPosition.Y, leftPosition.Z);

As you move your hands the vector changes and we used it’s angle relative to X and Y axis to rotate the cue. Simple! :)

We also wanted to use gesture to trigger a shot and the idea was to track the speed as you move hands together, then, when hands are close enough to each other we would take that speed and trigger the shot. This is something that still doesn’t work quite well. We were dealing with two problems here:

  • How to stop the cue from rotating when you want to make a shot and
  • Getting an accurate speed reading.

After the hackaton (actually today) I solved the first problem by cheating a little bit. If you looked at the video above you probably noticed that I was holding a wireless mouse. I decided to use left mouse click to lock rotation in. Right mouse button click is used to switch between moving the cue ball (when allowed) and rotating the cue. I’ve been told that Kinect for Windows allows you to track palm movements, so when you make a fist that is treated like a click. I am using Kinect for XBox 360 here and this one does not support such funky things. So the trick with a wireless mouse is here to stay until I get my hands on Kinect for Windows.

The problem with speed remains though. So how do we calculate the speed?

            // calculate distance between hands
            Distances = handVectors.Select(x => x.Length());

            // determine if there was a shot played
            const float NEAR_ZERO = 0.3f;
            var distanceTime = Distances.TimeInterval();
            var speeds = distanceTime.Zip(distanceTime.Skip(1), (a, b) => Math.Abs((a.Value - b.Value) / (a.Interval.TotalMilliseconds - b.Interval.TotalMilliseconds)));
            Shoots = speeds.WhereOther(Distances, x => Math.Abs(x) < NEAR_ZERO);

First we track the distance between hands and the time intervals at which frames are coming in. The speed is the distance traveled divided by the time it took to travel this far. Just as your physics teacher told you 😉

We only report speeds to the game engine once hands are close enough to each other. Like I said, there are still some issues and this method is somewhat unreliable. But I am yet to have any new ideas.

This is it. Hope you found it interesting :)

How to figure out if a point is within triangle

While playing around with XNA and my simple Snooker game I bumped into a geometrical problem, basically I wanted to know if the ball coordinate is in the corner of the table, which in my simple game equates to determining if a coordinate is inside a triangle near table corner.

I was looking at the algorithm I wrote in the original DirectX version of this game and quite frankly I had no idea what I was doing there. Speaking of which, my code from 10 years ago is quite shocking, I came a long way since :)

Anyway, after scratching my bald head for a while I came up with this.

I am dealing with a triangle that has two sides of the same length, mathematicians would call it an Isosceles triangle. What I did first was calculate how much into the pocket the ball was along X and Y axises and if the sum of these two is bigger than the size of the triangle then I assume the coordinate is inside triangle.

In the image below Px + Py is greater than Sx (which is the same as Sy since the sides are of equal length) and P is inside triangle.


Now consider the image below, here the sum of Px and Py is less than Sx (Sy) and my coordinate is outside of the triangle.