Kinect Biomechanics: Part 1

The Hardware

Dubbed ‘the fastest selling electronic product in history’, Microsoft’s Kinect has clearly captured the attention of the gamer. The market was fundamentally changed by Nintendo and the Wii, the Kinect is Microsoft’s attempt at a user-friendly, demographic spanning input device. While the Wii takes signals from a hand-held ‘Wii-Mote’, the Kinect does away with controllers completely.

The Kinect can track a person and register their movements within a gaming environment, as demonstrated by the video below (although I don’t believe perpetual grinning is necessary).

This interaction is made possible by the Kinect’s hardware. In addition to a standard RGB camera it contains a ‘depth sensing’ camera which feeds very clever algorithms. The depth camera utilises the principle of ‘structured light’; an infra-red projector sprays an invisible pattern of dots onto the surroundings which is captured and measured by a corresponding infra-red camera (See FuturePicture for a more in-depth description). Depth information allows separate objects to be detected within a field of view much more reliably than traditional image processing techniques. Processing algorithms take this depth information and identify individual users as well as the position and orientation of their limbs, amazing stuff and potentially very powerful.

Kinect Hacked

The device has made a massive impact with programmers who hacked it only days after its release. As attention around Kinect ‘hacks’ grew, a large number of novel applications were developed and inevitably, the research community has begun to utilise the device. A month after the release of the Kinect (and the first hacks) the makers of the hardware – PrimeSense – released drivers to give access to the powerful motion detection algorithms within. Microsoft has also announced that they are releasing a ‘non-commercial’ software developement kit (SDK) with future releases and more powerful features promised for the future.

The primesense drivers give access to object (user) detection, gesture recognition and the ability to model full 3D skeletons (with access to segment position and angle). Functionally at least, the Kinect seems ideal for use in a sports science or biomechanical environment. There are however two questions which need to be answered first;

  1. What is the scope for analysis?
  2. What are the applications?

This is the first of a series of articles which will look at different possible applications of the Kinect in the field of Biomechanics.

Body Segment Tracking

In sport biomechanics the position and orientation of an athlete is vital knowledge when assessing performance, injury risk and joint loading. Since the inception of the discipline scientists have searched for a method of obtaining this information quickly and accurately. Complicated calibration routines take time and can introduce error if not performed correctly, intrusive body markers can be a distraction to an athlete when asked to perform a complicated action.

As an accessory to the Xbox 360, the Kinect is not only used as a gesture recognition tool. It also tracks a user’s body position as a means of controlling an on -screen avatar (such as in dancing games). In other words, the Kinect is capable of tracking a user’s body segment positions and orientations in 3D, in real time. No markers are needed on the body and the minimal calibration consists of standing in a specific position for just a few seconds.

If we compare this with a cutting edge motion analysis suite, calibration is a complex process taking several minutes, a user must be accuractely palpated and markered and they cost over 100’s times more than a Kinect (£80 at the time of writing). Given a cursory glance, the Kinect seems more like a revolutionary analysis tool rather than living room trinket, however, there must be a caveat to all these wondorous properties?

Body Segments
A calibrated user in the Kinect’s field of view, the lines represent the fitted body segments

Of course, the Kinect is a tool aimed at the gamer, not a researcher where errors of a fraction of a millimetre can be unacceptable. As impressive as Kinect body tracking is, it’s accuracy is unlikely to compare to current standard methodologies in which anatomical joints are palpated and marked individually. However, there are thousands of students interested in tracking body motion. Sports enthusiasts and students would probably find an inexpensive and accessible tool for measuring joint angles and body timings invaluable.

Could the Kinect be used as a tool for simple motion analysis?

We recorded two simple body motions using a standard DV video camera and the Microsoft Kinect. The motions were a depth squat from standing and a walk and run on a Kistler Gaitway treadmill. These simple motions involved different speeds and movements. The aim was to compare the angle between the upper and lower leg (knee angle) as measured by the standard video analysis and the Microsoft Kinect.

The Kinect’s normal mode of use is with with the user face-on so that leg movement would be directly perpendicular to the image plane. While the depth camera should be able to cope with this, we were aiming to test the quality of the body tracking algorithms, not the robustness of the depth camera. For this reason the Kinect was placed at 45 degrees to the plane of motion, while the video camera viewed the motion directly side-on.

In order to obtain the Kinect’s value of the knee angle throughout the motion, we wrote a basic computer program to access skeleton position and orientation data. Dartfish was used to manually analyse the recorded video motion, a tool used widely in schools and universities (the animated gifs above show the calculated angles). The ankle, knee and hip joint were palpated and marked to enable accurate analysis.

It can be seen in both plots that the Kinect under-predicts knee angle at all points. It is unlikely you’d want to use the Kinect values for any kind of quantative study, but they could be used to show general trends in the data. I’m sure there are many more applications for this kind of information (illustrations of body angles) which don’t require a high level of accuracy but would benefit from instant results and minimal setup.

Sources of error

The Kinect hasn’t performed as accurately as we might have hoped, which leads to the following quesion;

What are the sources of inaccuracy and could it be accounted for and improved?

The angle obtained from the tracked skeleton data is a resultant angle in three dimensions. The computer program doesn’t account for the fact that the knee bends in a single plane.  Any error in the Kinect’s body tracking (resulting in an impossibly crooked knee bend) will be carried into the angle measurement. It should be relatively straightforward to introduce filters and logic gates to ensure this is minimised in future.

The video below shows how the Kinect performed at tracking body segments throughout the movement, difficulties in torso identification clearly amplify the error in thigh position and hence knee angle. Developing the existing Kinect algorithms was obviously a massive project. Therefore, it’s unlikely these could ‘re-developed’ to a great extent. However, it’s likely that the camera position could be optimised to mimise errors. In addition, it should be noted that this is only the first generation Kinect, who knows what will be released in the next few years.

Another possible source of error comes from the type of movement analysed; a single, lower body joint angle at an odd camera angle. It is possible that the Kinect is much better at calculating upper body segments in plane view, after all, it was designed with arm-based gesture recognition in mind. Hopefully more specific skeleton tracking software will be developed in time, its likely it could be greatly improved with the addition of easily tracked visual markers placed on anatomical landmarks or perhaps using multiple Kinect’s to analyse the same movement.

Conclusion

Overall, it may not be up to scratch for serious analysis, but to get (very) approximate values almost instantly the Kinect is hard to beat. It would be great to see some specific software which addresses some of the issues highlighted in this post, and maybe even some more work investigating the accuracy of joint angles for an upper-body motion. With the release of Microsoft’s SDK I wouldn’t be suprised to see PC specific programs which utilise the Kinect as an analysis tool with some nice interfaces.

Keep checking back for the next part of the article which will look at a different potential application for this amazing device.

Simon Choppin

About wiredchop

Simon Choppin Simon’s sports engineering career began at the age of six when he loosened the wheels of his skateboard in order to make it go faster. While the experiment was chalked up as his first failure, his resulting dimpled skull has provided an aerodynamic advantage in more recent sporting pursuits. Academically, Simon completed a degree in Mechanical Engineering with Mathematics at Nottingham University before joining the Sports Engineering Research Group at Sheffield to start his PhD. His main interests include work with high speed video, mathematical modelling of various sorts and experimental work involving machines with big buttons. As a sportsman, Simon has an unfortunate lack of talent for anything requiring skill, tactical awareness or the ability to learn from mistakes. He does however seem to posess the ability to move his legs around for a long time until other people get tired, for this reason you’re most likely to see him on a bike of some sort or running up a hill in offensively small shorts. Simon was fortunate enough to have a stint at the Guardian newspaper as part of the BSA’s media fellowship, which gave him the idea for this blog. Other than this, his writing experience includes his PhD thesis and various postcards to his Mum.

28 Responses

  1. David James

    Really fantastic blog Simon and very interesting to see a critical appraisal of the Kinect’s capabilities. I was starting to get a little tired of hearing how the Kinect was so amazing that it could solve all the world’s problems… I’m glad that you’ve shown that there’s still some development work out there for us sports engineers and biomechanists!

  2. theclownshoes

    I agree David. I am particularly interested in what will come out over the next few years. In my opinion this is only the beginning.

  3. Super interesting article !

    Indeed, it is not always practical to make sure that what we measure is exactly on the image plane (and it’s impossible to know by how much the measurement plane is tilted).

    If improved, this would be a great way to decrease the perspective error.

  4. Chrissy King

    Great article Simon, have shared on my Facebook page! Things are moving in the right direction though, Nintendo may have the next ‘game’ sorted out.

  5. stevehaake

    Simon, What is a typical angle measurement error using Dartfish? Also, it looks like there is a sytematic 20-40 degree error rather than random errors?

  6. Mark Sena

    Great stuff Simon! I’m in a biomechanics lab in the US and I recently got my PI to buy me a Kinect. I did a quick shoulder joint angle analysis for an occupational biomechanics class and it worked pretty well.
    Are you going to ISSSMC in Newcastle this August by chance?

  7. Tonia Lim

    Is it possible that the software you used with the kinect measures the angles at the actual axis of rotation instead of one fixed point on the knee? That may contribute to the discrepencies between the two sets of measurements.

  8. jaap jansen

    Simon, thanks for this great information. Being a physical therapist and movement scientist I was hoping that kinect would make motion analysis affordable for the large mass. Despite large errors the potential seems enormous with improving algorithms over time! I am not a computer software specialist, but I am very interested in obtaining the raw data to calculate the segment angles, to use the kinect for eductional purposes at the department of physical therapy. Could you please help me with that?
    Thanks for your blog!

  9. alex

    Could you please provide the methodology of your analysis or a reference to the paper that provided you this information.
    thks

  10. john

    Simon,
    Can you comment on the software that you used to obtain the sitckfigure and joint angles from the Kinect?
    John

  11. Hello,
    you have inspired me!

    I’m developing a software using Microsoft Kinect. My wish is to capture and analyze movements like squats, deadlifts and kettlebells swing.

    Just a video:

    http://www.facebook.com/video/video.php?v=121110901319217

    and the video list

    http://www.facebook.com/video/?id=181867965210688

    My software is still a prototype, the kinect is fantastic but the raw data flows are instable and there is a poor coerency, but I hope to solve the problem adding information about the movement to correct the data.

    Thanks you for the idea!

  12. Linda

    Is there information about the accuracy of length measurements taken from a Kinect? I want to measure upper extremity work space volume.

  13. Bob K

    Have you guys done any analysis on maximum distance from event and field of view? I’m particularly interested in whether or not the Kinect can pick up the movement of an athlete with a ball, and make any estimates of
    release of the object… for instance, see noahbasketball

  14. Job

    Hi everyone, my name is Job. Its really a gr8 blog Simon. In fact i am doing some stuffs related to the accuracy of kinect(particularly through microsoft SDK) in terms of the X- factor analysis. In my case am trying to find the angle between the left-rght shoulder vector and left-right hip vector. I think am gonna borrow a lot of concepts from your blog, if you don’t mind. Anyway, if you have some thing to share with regard to the X-factor stuff, you are really welcome. here is my e-mail: eyolla2010@yahoo.com

  15. Great stuff. Thanks for this blog. I am currently doing some research about developing a realistic gaming simulator using a systems of kinect and treadmill to enable player (or patient) to explore its enviroment freely (in just 1D forward direction for now). Hope we can discuss sometimes soon.

    Kin F. Kam

  16. Julia

    Hello! I am a biomechanics researcher just getting interested in how I might be able to use this technology to study subjects who can’t be brought into a lab. I’m wondering if you are aware of any new advances since you wrote this article that would help to increase the accuracy of the position and angle data. I am interested in angular velocities of the arm, which of course requires good accuracy of the position data. I’m wondering, for example, if there might be a way to calibrate in post processing to reduce some of the error. Thanks in advance for any help!

Comments are closed.