Virtual Marionette Project

Luis Miguel Barbosa is doing some interesting experiments with digital puppetry. His Virtual Marionette project uses an iPad to control a digital puppet character inside Animata (see previous posts). This takes advantage of the iPad's multi-touch interface to control multiple control points simultaneously in real-time.

He's also experimenting with something he calls WIIMATA, which uses two WIImotes to control a character inside Animata:



Neat stuff.

Puppet Time

Puppet Time is a new project from students at Georgia Tech that aims to "bring the art of digital puppetry to a wide range of users by taking advantage of the motion tracking capabilities of Android smart phones".

There have been a lot of projects very similar to this in the last few years so this isn't terribly new, although the use of Android smart phones as controllers is an interesting twist. Puppet Time was built using Unity 3D; it works by having Android phones send sensor data that controls the onscreen digital puppets to a computer running the main Unity application over a wifi network.

If you want to try it out for yourself, both the Unity and Android apps are available to download for free.

Motion Capture, Performance and Avatar

Some excerpts from a "making of" documentary about Avatar and its pioneering use of motion capture.

Jim Henson Google Doodle and Digital Puppet Karoke

If you visit the Google homepage tonight or tomorrow the "Google Doodle" features six digital puppets that you can perform with your mouse. It's the result of a collaboration between The Jim Henson Company and Google to commemorate what would have been Jim Henson's 75th birthday tomorrow.

What I love most about this is that such a clever, but simple thing is unleashing people's creativity. Within hours of the Google Doodle appearing "digital puppet karoke" videos of the puppets lip syncing to songs started popping up on YouTube.

Here's one of The Black Eyed Peas' Boom Boom Pow:



...and another of "Earth Angel":



Neat, huh?

Facial Puppetry: Face tracking and replacement

I've been casually doing some research in to face tracking and face substitution techniques for a film I'm currently directing and producing. I'm exploring the idea of blending conventional "Bunraku-style" puppetry techniques with real-time animation. What I'm hoping to be able to do is replace the face of a physical puppet with an animated one (preferably in real-time, although I might have to settle for some kind of post-production process).

The basic concept is to take a person’s expressions and map them in real-time to either a digital model of a face or match the expression with a photo of another face in an image database. There are a number of people doing some interesting work in this area, especially Jason Saragih who has created the FaceTracker library; you can see an example of what it can do in the video embedded above (I mentioned this briefly in my previous post).

This isn't really a new idea. Companies like ILM and Rhythm and Hues pioneered similar post production techniques back in the 1990s and they've been used for years to create talking animals in commercials, movies and TV shows. There has also been similar commercial software like CrazyTalk around for several years. What is relatively new is that the technology now works in real-time and is accessible to anyone with a decent computer and some basic programming knowledge.

Technology like this further blurs the line between puppetry and animation (which has been happening for awhile now) and offers artists the chance to have the best of both worlds. I really love the idea of being able to endow conventional puppets with a level of expression that just isn't possible in a physical puppet. Exciting stuff.

Digital Puppetry Round-up

It's been far too long since I've updated Machin-X, so to try to catch up here's a look at bunch of interesting things that have been happening in the past few months:

Carnegie Mellon University and Disney Research are working on a system for "motion capture on the go" that utilizes body mounted cameras, which opens up some intriguing technological possibilities (via Tech Crunch).

In Australia, Jason Saragih is working on an interesting system for doing facial puppetry. His work on facial expression recognition looks particularly impressive so far.

Check out the visual insanity that is the Unreal 3 graphics engine.

Using the Kinect for digital puppetry keeps getting easier.

Digital puppetry with Wiimotes, using the Unity 3D engine.

Blender continues to be expanded and refined. The latest stable version (and last of the 2.5 series) Blender 2.59 was recently released and a number of exciting new features are being planned for future releases, including better support for motion capture and an improved real-time animation system.

Finally, old puppetry meets new technology with the "Naked Puppet" digital sock puppet.

Another Real-time Digital Puppetry Demo

This is a fun little demo of a digital puppet that Seth Hunter is working on with puppeteer Dan Butterworth at the MIT Media Lab. Like a lot of digital puppetry projects such as Animata, it draws a lot of inspiration from shadow puppetry techniques. The software works by tracking points on a puppet that correlate in real-time to a digital version of the character. Users have the ability to change costumes and record animations and there are future plans to enable character customization as well as remote play.