This is a simple, but nonetheless very effective example of 2D digital puppetry created using Unity and a Kinect.

Faceshift Markless Motion Capture

Faceshift is software that promises "markless motion capture at every desk". It works with consumer-level cameras like the Kinect to track and analyze the facial expressions of a performer and uses them to animate a virtual character in real-time. It also offers the option of recording a performance so that it can be edited and polished in post-production.

There are lots of potential applications for this kind of software in game and film production and, of course, digital puppetry applications!

You can learn more at www.faceshift.com.

A digital dragon puppet

A nice example of a digital shadow puppet, made by Luis Leite using Kinect and Unity 3D. To animate the puppet, a human body is tracked in real-time using the Kinect sensor, with one hand controlling the head and the other controlling the tail. The physical movement of the performer's body is remapped on to the virtual shadow puppet using Inverse Kinematics via Unity's Mecanim animation system.

Luis was also responsible for a Kinect-based digital puppet that was mentioned in a post about Kinect-based digital puppetry on Machin-X two years ago.

Digital Shadow Puppet Installation

A nice look at an interactive, Kinect-based digital shadow puppet installation created by New York based motion designer Yang Yang.

Digital Wayang Kulit Music Video

We don't see a lot of music videos that feature digital puppetry, so when one comes along it's worth noting!

This video for Boshra Al Saadi's Snowyman was created using a modified version of Antonius Wiriadjaja's Java-based Wayang Kinect project. Hit the link to learn more about his work and download the digital Wayang Kulit applet so you can try it out yourself!

Flaming Skull Face Tracking Demo

Some clever face tracking from a Spanish studio called Paradox D&D. They've written a custom application that positions a 3D skull over top of the user's head, capturing the user's body movements and using them to control a 3D model in real-time. This is an example of a "virtual dresser" that allows a user to "wear" 3D elements in a manner very similar to augmented reality.

Via KinectHacks.

Researchers Demo New Kinect-based 3D Puppetry System

This video highlights a new Kinect-based 3D puppetry system developed by researchers at the University of Washington and the University of California, Berkeley that allows users to create 3D animation with minimal experience.

What's somewhat unique about this system is that unlike most Kinect-based systems, instead of the using body movement to control an onscreen character or object, users manipulate physical objects to control corresponding 3D models on a virtual set. I like this approach to digital puppetry a lot, because it retains the basic concept of puppetry (real-time manipulation of a physical object). To demonstrate how effective they believe this approach is, the researchers used novices in puppetry and animation to test the system.

Although the system these researchers have developed appears to be somewhat limited - note that the objects in the demo are solid objects without any kind of skeleton or use of Inverse Kinematics or Forward Kinematics - systems like this are very positive developments for digital puppetry. It would be great to see this get out of the lab and be developed in to some kind of open source platform.

A technical paper on this project can be found here.

Kinect Digital Puppetry Experiment

Here's a recent experiment in 2D digital puppetry, created using the Kinect and KinectSDK block for Cinder / C++ (compiled using Microsoft Visual Studio 2010). If you're able to program, you can find the code that was used to create this digital puppet here and try it yourself!

Another Blender / Kinect Demo

This is an early test of Kinect and Blender working together using OSCeleton. This is a custom coded solution that's different than the NI Mate software that I mentioned in my previous post. As you can see, it's not quite production ready yet, but it's still very, very exciting to see Blender and Kinect working together. Full details can be found here.

Great work Machinima Studios!

Real-Time Motion Capture in Blender at last!

NI Mate is software that imports Kinect motion capture data in real-time into 3D apps and they've announced that Blender is the first application that they're supporting. When I was working on the aborted Panda Puppet project several years ago I realized that one of the things Blender really needed in order to become a practical and effective tool for digital puppetry was a straight forward method of accepting motion capture data in real-time, so I think this is very exciting news and long overdue!

It should be really interesting to see what people start doing with this new capability in Blender. 

NI Mate is now available in open beta for both Windows and Mac OS X (Linux support is coming soon). You can register to participate in the beta and download all of the necessary software at www.ni-mate.com.

Via BlenderNation.

SoftKinetic Digital Puppetry Demo

SoftKinetic used a digital puppetry demo to show off some new firmware for their DepthSense cameras at last week's CES 2012 trade show. It highlights their new support for finger recognition, something that's difficult to accomplish with the X-Box Kinect without a lot of hacking. The firmware is still in the alpha stage, but it looks like SoftKinetic is putting together a gesture recognition system that could be really useful for digital puppetry applications.

Via The Verge.

Digital Trekkie Monster

Yoke created this very cool interactive display as a publicity stunt to help promote performances of Avenue Q at the Fredericia Theater in Denmark. Passersby are able to directly control a digital version of Trekkie Monster (one of the puppet characters who star in Avenue Q) via an X-Box Kinect. A great example of how to use interactive technologies to market a product and, if you ask me, an all-around brilliant idea!

Via Kinect Hacks.

Flash Animation Using Kinect

The Dead Man and the Lawyer is a short film and experiment by Nick Fox-Gieg, a Canadian animator and all-round interesting guy I've had the pleasure of getting to know over the past few months. Although this might look like relatively straight forward 2D animation, he actually made it using the Kinect and I think this opens up some exciting possibilities for creating Kinect animation.

To make the film Nick used OSCeleton to record Kinect data to an .xml file via a utility he's written called Processing Sketch. The beauty of this approach is that the motion capture information was written to an .xml file in real-time without any need for video capture. That way the animation data can be read back from .xml in Flash, tweaked and then rendered in full HD. A short tutorial and download links for all the necessary utilities and libraries you need to try this yourself can be found on Nick's website.

Nick's also a semifinalist in YouTube's NextUp competition right now. If you like his work, show him some love and vote for him.

First Kinect Motion Capture Series - Under the HUD

Under the HUD is what looks to be the first motion capture web series created using the Kinect. It was made using the OpenNI and Brekel Kinect hacks, as well as Autodesk's MotionBuilder and Maya software. Although the animation isn't completely fluid or elegant, it's not bad either. The important point is that, as Fast Company points out, until recently you needed to spend thousands (not the millions claimed by FC) to do this.

Episodes are coming soon; you can follow @thwackers on Twitter and visit the Thwackers website for updates.

Via Kinect Hacks.

New Sesame Street Game For Kinect

Here are some digital puppets in a very literal sense...Sesame Street: Once Upon a Monster is a new game for Xbox Kinect by acclaimed games studio Double Fine. The family-friendly game promises to take players into a living storybook populated with famous Muppet characters like Elmo, Cookie Monster and Oscar the Grouch. Gamasutra has an interview with Double Fine's Nathan Martz and Tim Schafer discussing the development of the game in depth.



The team at Double Fine is obviously having fun with this project (and who wouldn't?). Have a look at this video of Tim Schafer showing off some "fur rendering" technology.

Once Upon a Monster looks like it will be a lot of fun. It's due for release this fall.

Via Tough Pigs.

We Be Monsters: Multi-Person Digital Puppets

Carnegie Mellon University has been a leader in interactive studies (including lots of digital puppetry work) for years now so it's not surprising that CMU students have been busy dreaming up some pretty cool projects using the Kinect. A good example of this is We Be Monsters, a Kinect hack by students Caitlin Rose Boyle and Asa Foster that turns participants into a two-person puppet, “inspired by multi-person Chinese dragon costumes and (Mr.) Snuffleupagus.” It's a cool project that's somewhat similar to the previously-blogged Animata software.

You can see more examples of the great work that Carnegie Mellon students are cooking up over at Create Digital Motion.

Chris O’Shea's Little Magic Stories

The more artists explore new frontiers in interactive art, the more I often mind myself wondering where exactly to draw the line between what is animation and what is puppetry. A good example of this is the augmented shadow puppetry that I recently wrote about over at PuppetVision, which is sort of shadow puppetry, but also sort of not.

Another good example is this performance system created by Chris O’Shea called Little Magic Stories. It was made using the X-Box Kinect (see previous post) and the old Pepper's Ghost Vaudeville trick.

So is this animation? Digital puppetry? Or something else entirely?

Via CrunchGear.

Kinect Digital Puppetry: It's Here and it's Real

Kinect hacking has come a long way in a few short months and as most people following it already know, the potential implications for digital puppetry are huge. I was at a small meet-up for Kinect hackers in Toronto last night and tried a very rudimentary version of "Kinect puppetry" that an animator I know has been working on and it was very cool. Affordable, accessible digital puppetry is here, it's real and if you already have a computer and a little technical know-how you can do it yourself for under $200.

Here's an impressive demo of Kinect puppetry using the Japanese "Vocaloid" Hatsune Miku using the Kinect and free software from the VPVP Project:



Aside from the Kinect itself (which retails for around $150), most of the software needed to try this yourself can be downloaded for free. Of course you need a 3D program like Blender along with some sort of 3D game engine. Fortunately "lite" or "learning" versions of commercial engines like Motionbuilder and Unity can be downloaded for free and some people are using Ogre 3D, which is open source. The exact methods to make your own digital puppets will vary depending on which software you use, but lots of tutorials can be found by searching online.

Not all digital puppetry with the Kinect has to be three dimensional though. Here's a neat 2D demo done in the style of shadow puppetry. It's part of a project called Virtual Marionette:



Although I do think this is huge step forward for open, accessible digital puppetry there are still some problems to solve before Kinect can be used to create truly emotive, expressive characters. For one thing, the skeletal tracking in these demos is very simple without individual finger movements or any kind of facial expression. Those problems aren't unsolvable though and I have some ideas on how to tackle them, but that's another post.

Cross-posted from PuppetVision.

Kinect-Powered Hologram

Another amazing Kinect hack...researchers at MIT built a rudimentary holographic video system using an X-Box Kinect. They even re-create the famous hologram of Princess Leia from Star Wars in this demo video.

Real-time, 3D holographic digital characters anyone?

Via Fast Company.

More Kinect Digital Puppetry Developments

New X-Box Kinect developments seem to keep coming fast and furious. At CES 2011, Microsoft has been demoing it's forthcoming Avatar Kinect for X-Box Live; essentially it's a platform that allows you to control a 3D avatar of yourself in real-time via body and facial movements. I suspect that the technology isn't quite ready to go yet (Microsoft CEO Steve Ballmer's demo above was canned and not done live), but Engadget has a video of a live demo of Avatar Kinect here.

Meanwhile, in other Kinect news, someone has hacked together some impressive-looking real-time motion capture using the Kinect:



(Motion capture video via Blendernation)