Kinect sideways skeleton tracking

c# kinect
Velth picture Velth · Apr 17, 2012 · Viewed 10.5k times · Source

Currently I am using the Microsoft Kinect for measuring angles between joints. Most measurements are working correctly. Whenever a person is sitting sideways (on a chair) the Kinect won't track the skeleton accurate. To illustrate my problem I've added 3 pictures of the Kinect depthview.

Seated sideways measurement with skeleton tracking

Seated sideways measurement without skeleton tracking

Sideways measurement with skeleton tracking

As you can see 2 out of 3 measurements work "correctly". Whenever I lift my leg, the Kinect stops skeleton tracking correctly. Does anyone have a solution to this problem, or is this just a limitation of the Kinect?

Thanks.

Update 1: The JointTrackingState-Enumeration on these tracked joints shown at screenshot 2 are marked as Inferred, however the depth view is tracking my full body.

Update 2: At screenshot 2 I'm trying to track my front leg, highlighted as green. I know the other leg isn't tracked but this does not matter, I guess.

Update 3: The following code selects a skeleton:

private Skeleton StickySkeleton(Skeleton[] skeletons)
{
    if (skeletons.Count<Skeleton>(skeleton => skeleton.TrackingId == _trackedSkeletonId) <= 0)
    {
        _trackedSkeletonId = -1;
        _skeleton = null;
    }

    if (_trackedSkeletonId == -1)
    {
        Skeleton foundSkeleton = skeletons.FirstOrDefault<Skeleton>(skeleton => skeleton.TrackingState == SkeletonTrackingState.Tracked);

        if (foundSkeleton != null)
        {
            _trackedSkeletonId = foundSkeleton.TrackingId;
            return foundSkeleton;
        }
    }

    return _skeleton;
}

Whenever a skeleton is tracked that data will be used for drawing the joint-points and calculating the angle between joints.

Update 4: I've tested sitting on a 'block', far more simplified than a chair. Unfortunately the Kinect still acts the same.

Below 2 screenshots:

Sitting on a block 1

Sitting on a block 2

Answer

Liam McInroy picture Liam McInroy · Apr 21, 2012

As Renaud Dumont stated, I would do something with JointTrackingState. Since your using knees, I used the variables leftknee and rightknee which are Joints to do it. Here's code, You might use JointType.FootRight and JointType.FootLeft and the Hip types, but I'll leave that up to you.

static Skeleton first = new Skeleton();

Joint leftknee = first.Joints[JointType.KneeLeft];
Joint rightknee = first.Joints[JointType.KneeRight];

if ((leftknee.TrackingState == JointTrackingState.Inferred ||
                leftknee.TrackingState == JointTrackingState.Tracked) && 
                (rightknee.TrackingState == JointTrackingState.Tracked ||
                rightknee.TrackingState == JointTrackingState.Inferred))
            {

            }

Or alternately, if you just wanted one knee to be tracked at a time, or both, you could do this:

if ((leftknee.TrackingState == JointTrackingState.Inferred ||
                leftknee.TrackingState == JointTrackingState.Tracked) && 
                (rightknee.TrackingState == JointTrackingState.Tracked ||
                rightknee.TrackingState == JointTrackingState.Inferred))
            {

            }

            else if (leftknee.TrackingState == JointTrackingState.Inferred ||
                    leftknee.TrackingState == JointTrackingState.Tracked)
            {

            }

            else if (rightknee.TrackingState == JointTrackingState.Inferred ||
                    rightknee.TrackingState == JointTrackingState.Tracked)
            {

            }

FYI the reason the Skeleton is first is static is because then it can be used in making the joints

 static Skeleton first;

Opposed to 

 Skeleton first;

Edit 1


I've come to the conclusion this is extremely difficult to do, I think the above method will work, but I just wanted to include what I'm working on in case you might be able to find some way to make it work. Anyways Here's the code I was working on which is another class which is just another SkeletalTrackingState I was trying to make an Inferred enum in it. But unfortunately enum are impossible to inherit. If you find something to this effect that works, I will respect you as a superior programmer to me forever ;). Without further ado: The .dll I was trying to make:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Kinect;

namespace IsInferred
{
public abstract class SkeletonInferred : Skeleton
{
    public bool inferred;
    static Skeleton first1 = new Skeleton();
    Joint handright;
    Joint handleft;
    Joint footright;
    Joint footleft;
    Joint ankleleft;
    Joint ankleright;
    Joint elbowleft;
    Joint elbowright;
    Joint head;
    Joint hipcenter;
    Joint hipleft;
    Joint hipright;
    Joint shouldercenter;
    Joint shoulderleft;
    Joint shoulderright;
    Joint kneeleft;
    Joint kneeright;
    Joint spine;
    Joint wristleft;
    Joint wristright;

    public SkeletonInferred(bool inferred)
    {

    }

    public enum Inferred
    {
        NotTracked = SkeletonTrackingState.NotTracked,
        PositionOnly = SkeletonTrackingState.PositionOnly,
        Tracked = SkeletonTrackingState.Tracked,
        Inferred = 3,
    }

    private void IsInferred(object sender, AllFramesReadyEventArgs e)
    {
        handright = first1.Joints[JointType.HandRight];
        handleft = first1.Joints[JointType.HandLeft];
        footright = first1.Joints[JointType.FootRight];
        footleft = first1.Joints[JointType.FootLeft];
        ankleleft = first1.Joints[JointType.AnkleLeft];
        ankleright = first1.Joints[JointType.AnkleRight];
        elbowleft = first1.Joints[JointType.ElbowLeft];
        elbowright = first1.Joints[JointType.ElbowRight];
        head = first1.Joints[JointType.Head];
        hipcenter = first1.Joints[JointType.HipCenter];
        hipleft = first1.Joints[JointType.HipLeft];
        hipright = first1.Joints[JointType.HipRight];
        shouldercenter = first1.Joints[JointType.ShoulderCenter];
        shoulderleft = first1.Joints[JointType.ShoulderLeft];
        shoulderright = first1.Joints[JointType.ShoulderRight];
        kneeleft = first1.Joints[JointType.KneeLeft];
        kneeright = first1.Joints[JointType.KneeRight];
        spine = first1.Joints[JointType.Spine];
        wristleft = first1.Joints[JointType.WristLeft];
        wristright = first1.Joints[JointType.WristRight];

        if (handleft.TrackingState == JointTrackingState.Inferred &
            handright.TrackingState == JointTrackingState.Inferred &
            head.TrackingState == JointTrackingState.Inferred &
            footleft.TrackingState == JointTrackingState.Inferred &
            footright.TrackingState == JointTrackingState.Inferred &
            ankleleft.TrackingState == JointTrackingState.Inferred &
            ankleright.TrackingState == JointTrackingState.Inferred &
            elbowleft.TrackingState == JointTrackingState.Inferred &
            elbowright.TrackingState == JointTrackingState.Inferred &
            hipcenter.TrackingState == JointTrackingState.Inferred &
            hipleft.TrackingState == JointTrackingState.Inferred &
            hipright.TrackingState == JointTrackingState.Inferred &
            shouldercenter.TrackingState == JointTrackingState.Inferred &
            shoulderleft.TrackingState == JointTrackingState.Inferred &
            shoulderright.TrackingState == JointTrackingState.Inferred &
            kneeleft.TrackingState == JointTrackingState.Inferred &
            kneeright.TrackingState == JointTrackingState.Inferred &
            spine.TrackingState == JointTrackingState.Inferred &
            wristleft.TrackingState == JointTrackingState.Inferred &
            wristright.TrackingState == JointTrackingState.Inferred)
        {
            inferred = true;
        }
      }
    }
  }

The code in your project (compiler error)

    using IsInferred;

    static bool Inferred = false;
    SkeletonInferred inferred = new SkeletonInferred(Inferred);
    static Skeleton first1 = new Skeleton();

    Skeleton foundSkeleton = skeletons.FirstOrDefault<Skeleton>(skeleton =>  skeleton.TrackingState == SkeletonTrackingState.Inferred);

Good luck, I hope this helps you get going in the right direction or helps you at all!

My Code


Here's my code that you asked for. Yes it is from the Skeletal Tracking Fundamentals, but this code was here and I didn't want to start a new project with most of the same stuff. Enjoy!

Code

     // (c) Copyright Microsoft Corporation.
     // This source is subject to the Microsoft Public License (Ms-PL).
    // Please see http://go.microsoft.com/fwlink/?LinkID=131993 for details.
    // All other rights reserved.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Kinect;
using Coding4Fun.Kinect.Wpf; 

namespace SkeletalTracking
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
    public MainWindow()
    {
        InitializeComponent();
    }

    bool closing = false;
    const int skeletonCount = 6; 
    Skeleton[] allSkeletons = new Skeleton[skeletonCount];

    private void Window_Loaded(object sender, RoutedEventArgs e)
    {
        kinectSensorChooser1.KinectSensorChanged += new  DependencyPropertyChangedEventHandler(kinectSensorChooser1_KinectSensorChanged);

    }

    void kinectSensorChooser1_KinectSensorChanged(object sender, DependencyPropertyChangedEventArgs e)
    {
        KinectSensor old = (KinectSensor)e.OldValue;

        StopKinect(old);

        KinectSensor sensor = (KinectSensor)e.NewValue;

        if (sensor == null)
        {
            return;
        }




        var parameters = new TransformSmoothParameters
        {
            Smoothing = 0.3f,
            Correction = 0.0f,
            Prediction = 0.0f,
            JitterRadius = 1.0f,
            MaxDeviationRadius = 0.5f
        };
        sensor.SkeletonStream.Enable(parameters);

        //sensor.SkeletonStream.Enable();

        sensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(sensor_AllFramesReady);
        sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30); 
        sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

        try
        {
            sensor.Start();
        }
        catch (System.IO.IOException)
        {
            kinectSensorChooser1.AppConflictOccurred();
        }
    }

    void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
    {
        if (closing)
        {
            return;
        }

        //Get a skeleton
        Skeleton first =  GetFirstSkeleton(e);

        if (first == null)
        {
            return; 
        }



        //set scaled position
        //ScalePosition(headImage, first.Joints[JointType.Head]);
        ScalePosition(leftEllipse, first.Joints[JointType.HandLeft]);
        ScalePosition(rightEllipse, first.Joints[JointType.HandRight]);
        ScalePosition(leftknee, first.Joints[JointType.KneeLeft]);
        ScalePosition(rightknee, first.Joints[JointType.KneeRight]);

        GetCameraPoint(first, e); 

    }

    void GetCameraPoint(Skeleton first, AllFramesReadyEventArgs e)
    {

        using (DepthImageFrame depth = e.OpenDepthImageFrame())
        {
            if (depth == null ||
                kinectSensorChooser1.Kinect == null)
            {
                return;
            }


            //Map a joint location to a point on the depth map
            //head
            DepthImagePoint headDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.Head].Position);
            //left hand
            DepthImagePoint leftDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.HandLeft].Position);
            //right hand
            DepthImagePoint rightDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.HandRight].Position);

            DepthImagePoint rightKnee =
                depth.MapFromSkeletonPoint(first.Joints[JointType.KneeRight].Position);

            DepthImagePoint leftKnee =
                depth.MapFromSkeletonPoint(first.Joints[JointType.KneeLeft].Position);


            //Map a depth point to a point on the color image
            //head
            ColorImagePoint headColorPoint =
                depth.MapToColorImagePoint(headDepthPoint.X, headDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);
            //left hand
            ColorImagePoint leftColorPoint =
                depth.MapToColorImagePoint(leftDepthPoint.X, leftDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);
            //right hand
            ColorImagePoint rightColorPoint =
                depth.MapToColorImagePoint(rightDepthPoint.X, rightDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);

            ColorImagePoint leftKneeColorPoint =
                depth.MapToColorImagePoint(leftKnee.X, leftKnee.Y,
                ColorImageFormat.RgbResolution640x480Fps30);

            ColorImagePoint rightKneeColorPoint =
                depth.MapToColorImagePoint(rightKnee.X, rightKnee.Y,
                ColorImageFormat.RgbResolution640x480Fps30);



            //Set location
            CameraPosition(headImage, headColorPoint);
            CameraPosition(leftEllipse, leftColorPoint);
            CameraPosition(rightEllipse, rightColorPoint);


            Joint LEFTKNEE = first.Joints[JointType.KneeLeft];
            Joint RIGHTKNEE = first.Joints[JointType.KneeRight];

            if ((LEFTKNEE.TrackingState == JointTrackingState.Inferred ||
            LEFTKNEE.TrackingState == JointTrackingState.Tracked) &&
            (RIGHTKNEE.TrackingState == JointTrackingState.Tracked ||
            RIGHTKNEE.TrackingState == JointTrackingState.Inferred))
            {
                CameraPosition(rightknee, rightKneeColorPoint);
                CameraPosition(leftknee, leftKneeColorPoint);
            }

            else if (LEFTKNEE.TrackingState == JointTrackingState.Inferred ||
                    LEFTKNEE.TrackingState == JointTrackingState.Tracked)
            {
                CameraPosition(leftknee, leftKneeColorPoint);
            }

            else if (RIGHTKNEE.TrackingState == JointTrackingState.Inferred ||
                    RIGHTKNEE.TrackingState == JointTrackingState.Tracked)
            {
                CameraPosition(rightknee, rightKneeColorPoint);
            }
        }        
    }


    Skeleton GetFirstSkeleton(AllFramesReadyEventArgs e)
    {
        using (SkeletonFrame skeletonFrameData = e.OpenSkeletonFrame())
        {
            if (skeletonFrameData == null)
            {
                return null; 
            }


            skeletonFrameData.CopySkeletonDataTo(allSkeletons);

            //get the first tracked skeleton
            Skeleton first = (from s in allSkeletons
                                     where s.TrackingState == SkeletonTrackingState.Tracked
                                     select s).FirstOrDefault();

            return first;

        }
    }

    private void StopKinect(KinectSensor sensor)
    {
        if (sensor != null)
        {
            if (sensor.IsRunning)
            {
                //stop sensor 
                sensor.Stop();

                //stop audio if not null
                if (sensor.AudioSource != null)
                {
                    sensor.AudioSource.Stop();
                }


            }
        }
    }

    private void CameraPosition(FrameworkElement element, ColorImagePoint point)
    {
        //Divide by 2 for width and height so point is right in the middle 
        // instead of in top/left corner
        Canvas.SetLeft(element, point.X - element.Width / 2);
        Canvas.SetTop(element, point.Y - element.Height / 2);

    }

    private void ScalePosition(FrameworkElement element, Joint joint)
    {
        //convert the value to X/Y
        //Joint scaledJoint = joint.ScaleTo(1280, 720); 

        //convert & scale (.3 = means 1/3 of joint distance)
        Joint scaledJoint = joint.ScaleTo(1280, 720, .3f, .3f);

        Canvas.SetLeft(element, scaledJoint.Position.X);
        Canvas.SetTop(element, scaledJoint.Position.Y); 

    }


    private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
    {
        closing = true; 
        StopKinect(kinectSensorChooser1.Kinect); 
    }

    private void kinectDepthViewer1_Loaded(object sender, RoutedEventArgs e)
    {

    }

   }
}

XAML

<Window x:Class="SkeletalTracking.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="MainWindow" Height="600" Width="800" Loaded="Window_Loaded" 
    xmlns:my="clr-namespace:Microsoft.Samples.Kinect.WpfViewers;assembly=Microsoft.Samples.Kinect.WpfViewers" 
    Closing="Window_Closing" WindowState="Maximized">       
<Canvas Name="MainCanvas">
    <my:KinectColorViewer Canvas.Left="0" Canvas.Top="0" Width="640" Height="480" Name="kinectColorViewer1" 
                          Kinect="{Binding ElementName=kinectSensorChooser1, Path=Kinect}" />
    <Ellipse Canvas.Left="0" Canvas.Top="0" Height="50" Name="leftEllipse"  Width="50" Fill="#FF4D298D" Opacity="1" Stroke="White" />
    <Ellipse Canvas.Left="100" Canvas.Top="0" Fill="#FF2CACE3" Height="50" Name="rightEllipse" Width="50" Opacity="1" Stroke="White" />
    <my:KinectSensorChooser Canvas.Left="250" Canvas.Top="380" Name="kinectSensorChooser1" Width="328" />
    <Image Canvas.Left="66" Canvas.Top="90" Height="87" Name="headImage" Stretch="Fill" Width="84" Source="/SkeletalTracking;component/c4f-color.png" />
    <Ellipse Canvas.Left="283" Canvas.Top="233" Height="23" Name="leftknee" Stroke="Black" Width="29" />
    <Ellipse Canvas.Left="232" Canvas.Top="233" Height="23" Name="rightknee" Stroke="Black" Width="30" />
</Canvas>

Here's a picture just to show how off Kinect can be sometimes:

Hint: Notice how only my arm is detected and part of the background

Related questions

Implicitly convertible to 'System.IDisposable' error

This is what I'm trying to do: private KinectAudioSource CreateAudioSource() { var source = KinectSensor.KinectSensors[0].AudioSource; source.NoiseSuppression = _isNoiseSuppressionOn; source.AutomaticGainControlEnabled = _isAutomaticGainOn; return source; } private object lockObj = new object(); private void RecordKinectAudio() { lock (lockObj) { using (var source = CreateAudioSource()) { } } } The 'using' statement …

Read More
Using XBox 360 Kinect with Kinect for Windows SDK

I'm working on a class project which is using a Kinect. According to the [Microsoft Kinect for Windows Information Page][1]: If you’re receiving either of these error messages, you’re probably using a Kinect for Xbox 360 and not the …

Read More
How to make a control in XAML public in order to be seen in other classes

I'm working in wpf application i made a checkbox in the XAML, then my code calls a function in a class and in this function there is an if condition where its checking on whether the checkbox is checked or …

Read More