Building an Interactive Mirror with Unreal Engine and Microsoft Kinect

by Noah Fernandes

The Diabetes Foundation launched an exhibition in Central Amsterdam to raise awareness about sugar consumption in the Netherlands. But how do we make this a fun experience, one that's engaging with their audience? Their creative agency (Lemz) proposed an educational interactive mirror that could 'x-ray' your body, test your knowledge about sugar levels in foods and highlight the internal organs most affected by sugar and diabetes.
And we at Magic Bullet, we turned this idea to reality.

Kid playing with our interactive mirror

The technology behind this magic is Microsoft Kinect + Unreal Engine + one giant screen! Given that I had little experience in Unreal and had to launch this project in 4 weeks, I dove straight into the deep end and learnt on job.

Learning on the job

This intensive crash course I plunged into was a great opportunity. Unreal Engine is something I always wanted to play with but never found the time to, and this project required me to cover a lot of ground including:

  • Lighting
  • Audio cueing and synching
  • Camera and Perspective
  • Materials, Textures and Shading
  • World and Relative Transformations
  • Skeletal Meshes, Rigging and Sockets
  • Trigger Collisions
  • UI Widgets and Animations
  • Events, Functions and Macros
  • Blueprints and a bit of C++

It took me a while to get comfortable programming in Blueprints. I'm more of a 'code needs to be written' kinda guy, but surprisingly found dragging and connecting nodes to be quite satisfying too. Blueprints made it easy to start playing around in Unreal without having to know C++.

Blueprints are pretty cool too

I used the Kinect4Unreal plugin by Opaque Media, which made integrating the Kinect with Unreal a breeze. Their team was also very helpful whenever I had questions, a great service specially when considering they give away this plugin for FREE.
I plan to write a tutorial showing how easy it is to animate a skeletal mesh using their plugin, but for now, it's important to know three things:

  • The plugin works with the latest version of Unreal (4.13 at the time of writing this)
  • Use the Kinect Interface component (rather than the Kinect Player Controller thats deprecated)
  • Unfortunately, the 'mirror' option for the meshes doesn't work out of the box and requires a lot of manual joint rotation tweaking, but giving the skeletal mesh a -1 on the y-scale is an easy solution to this issue
Clients doing a final test before launch
Setting up the exhibition
It's got to have laser perfection
Panoramic view of the exhibition

The exhibit was covered by the media on several websites and TV channels like RTLXL, RTL Nieuws, Parool, Binnenland, and radio stations Radio 1, 538 Nieuws, ANP, Q-Music & BNR, with the Interactive Mirror being a regular highlight of the show. The famous and animated chef and food TV presenter, Pierre Wind, almost got run over by a tram in his excitement while filming his piece on our exhibition.

Careful Pierre!
Media coverage by RTL Live

This was a really awesome project to work on, mainly because I finally had the chance to fool around with Unreal. Now I'm keen to learn and do more things with it! If you're in Amsterdam from the 7th to the 13th of November'16, check out the exhibition at Leidsestraat 46, 1017 PC Amsterdam.

A quick time lapse of the project development: Learn basics in Unreal > Integrate the Kinect > Prototype with some meshes > Implement a rigged skeleton > Polish the materials and textures > et voila!