Experimental Digital Shadow Puppets in Blender

Lyle at LJ Puppetry recently shared this digital puppetry experiment. Lyle was inspired by the Henson Digital Puppetry Studio and its use of a Waldo, but wanted to see if he could create digital shadow puppets that were controlled in real-time via motion capture inside Blender. Lyle himself admits that his results were mixed, but I think his project may indirectly suggest some really interesting technological possibilities.

Of course, experimenting with digital shadow puppets is a well-trodden path. I began my own journey in to real-time animation experimenting with digital shadow puppets when I first started writing Machin-X, although back then we had to rely on colour blob tracking and custom scripting in Adobe Director because there were no off the shelf tools. 

Today, it's much, much easier to do this with off-shelf open source software like Blender, which is what Lyle used to perform this multi-scene shadow story - a magi, a tiger that transforms into a woman, a figure that coalesces from stars - entirely through hand and finger movements captured by a webcam:

The shadow characters in the video are flat black figures built using Blender's grease pencil tool. Lyle controlled them in real time by mapping specific finger positions to specific puppet movements — thumb open stands the magi up, a closed fist brings him to his knees. What we're seeing in the video isn't playback; it's a live performance into Blender, with the puppeteer off-camera and the output on screen.

Although his experiment wasn't entirely successful, Lyle's documentation is an excellent starting point for someone else who wants to try experimenting with this on their own. You can read an in-depth write-up of his workflow here. 

This might be the right approach with the wrong tool

I've been following motion capture developments in Blender for a long time. I was very excited by the BlendArMocap plugin when I first heard about it, but I've since discovered that it has some real limitations. Dropped frames (the "jerky mocap" that Lyle mentions) and the ability to only perform one puppet at a time are two of the known issues.

Personally, I suspect that BlendArMocap might be the wrong implementation of the right technology. It's notoriously difficult to manage motion tracking inside Blender itself (which is what BlendArMocap does), but the underlying technology that makes that markerless motion tracking on a standard webcam possible, Google's open source Mediapipe framework, is brilliant.

TouchDesigner's node-based interface can look intimidating, but it's a popular tool used by artists all over the world.

Mediapipe offers real possibilities for digital puppetry. The ability to use markerless motion capture - especially in a tool like Blender's innovative Grease Pencil - is tantalizing, but a better way of utilizing it might be with a program like TouchDesigner. TouchDesigner is used by artists and performers to process, and react to live data — from motion capture to audio to video — in real time. If you've seen a live concert where the visuals pulse and shift in sync with the music, that was likely done with TouchDesigner, and it can also send clean motion data directly into Blender in real time, via OSC. 

Frankly, it's much easier to fix problems like "jerky mocap" in TouchDesigner before the data ever reaches Blender, which in turn creates a much more streamlined workflow.

We need more experiments like this 

I love what Lyle has shared here, and that he is encouraging others to learn from his project. He's given me some things to think about, and I'm very excited to see what others do with Blender and tools like TouchDesigner and Mediapipe.

More than anything though, this experiment serves to remind us that creating great puppetry is hard, creating digital puppetry is especially hard...and recreating shadow puppetry in particular is deceptively hard. 

Digital Wayang Kulit

Here's a nine minute demo of a Digital Wayang Kulit (Indonesian shadow puppetry) program developed at the MSc in Digital Education program at University of Edinburgh. The 2D figure is controlled by the digital Dalang (puppeteer) using a Gametrak controller and a Wiimote.

There have been a lot of shadow puppet inspired digital puppetry demos created over the years (this one is from 2013), but I love how fluid the movement of this one is.

Special thanks to Jane for submitting this!

Character Animator - A 2D Real-Time Animation tool from Adobe

Adobe has just unveiled a new 2D digital puppetry - or have we all agreed to call this field real-time animation now? - application called Character Animator. A demo of the software is provided in the video above, but essentially it's a tool for animating 2D bitmap (Photoshop) and vector (Illustrator) still characters in real-time using a camera, microphone, head tracking and facial mo-cap.

In addition to basic mo-cap and lip sync capabilities, it also allows users to create programmable behaviors and will support the creation of 3rd party plug ins. I haven't tried it myself yet, but it looks like it could be a very easy-to-use and potentially powerful tool for creating basic 2D animated characters in real-time.

Adobe Character Animator is currently in beta and available for testing by After Effects CC users. You can learn more here.

Via The Labyrinth.

Unbelievably impressive demo of the Unreal Engine 4

It's been far, far too long since I posted an update here on Machin-X, but I may be (finally) returning to some digital puppetry work in the near future and when I saw this demo for the Unreal Engine 4 I had to share it.

It is, well, pretty unreal:

The Kite open world demo created in Unreal Engine 4 features a diverse and beautifully realized 100 square mile landscape. Everything is generated completely in real-time at 30fps and includes fully dynamic direct and indirect illumination, cinematic depth of field and motion blur, and procedurally placed trees and foliage.

Real-time 3D sure has come a long way since I first worked on a TV pilot for a proposed kids' series using cardboard cut-outs and blob tracking with a webcam to create some 2D flash animation almost a decade ago.

I wonder where this technology will go in the ten years or so?

Henson Digital Puppetry Studio

This is a brand new promotional reel for the Henson Digital Puppetry Studio, the patented real-time animation/digital puppetry system developed by Jim Henson's Creature Shop.

Animatic Digital Puppetry System

A look at "Animatic", a digital puppetry system that was developed by Luis Leite (see previous post) using 3D Studio Max and Macromedia Director in 2006. The system was developed as part of his research thesis Marionetas Virtuais.

Faceshift Markless Motion Capture

Faceshift is software that promises "markless motion capture at every desk". It works with consumer-level cameras like the Kinect to track and analyze the facial expressions of a performer and uses them to animate a virtual character in real-time. It also offers the option of recording a performance so that it can be edited and polished in post-production.

There are lots of potential applications for this kind of software in game and film production and, of course, digital puppetry applications!

You can learn more at www.faceshift.com.

Hakanaï: Dancing with Digital Puppetry

Hakanaï is one of the more unconventional examples of a digital puppetry performance I've discovered (although, is there anything truly "conventional" about any form of digital puppetry?). Its creators describe it as a "haiku dance performance taking place in a cube of moving images projected live by a digital performer".

The performance involves a dancer performing live, whose movements are tracked in real-time and used as the basis for an interactive, digitally animated environment that is projected around them:

 

It was created by the French Company Adrien M / Claire B using their proprietary software eMotion. Here's more from their description of the project:

 ...Performed by an artist as a “digital score”, it is generated and interpreted live. The dancer’s body enters into a dialogue with the moving images in motion. These simple and abstract black and white shapes behave according to physical rules that the senses recognise and to mathematical models created from the observation of nature.

The audience experiences the performance in several stages. They first discover the exterior of the installation. As the dancer arrives, they gather around to watch the performance. When the choreography has ended, the audience can then take some time to wander amongst the moving images.

Through a minimalist transposition, this piece is based on images drawn from the imaginary realm of dreams, their structure and their substance. The box in turns represents: the bedroom where, once the barrier of sleep is passed, walls dissolve and a whole new inner space unfolds; the cage, of which one must relentlessly test the limits; the radical otherness, as a place of combat with an intangible enemy; the space where impossible has become possible, where all the physical points of reference and certitudes have been shaken.

Through the encounter of gesture and image, two worlds intertwine. The synchronicity between the real and the virtual dissolves and the boundary that was keeping them separate disappears, forming a unique space filled with a high oneiric charge. 

Very cool, no? You can learn more from the video's description on Vimeo.

Digital Wayang Kulit Music Video

We don't see a lot of music videos that feature digital puppetry, so when one comes along it's worth noting!

This video for Boshra Al Saadi's Snowyman was created using a modified version of Antonius Wiriadjaja's Java-based Wayang Kinect project. Hit the link to learn more about his work and download the digital Wayang Kulit applet so you can try it out yourself!

Researchers Demo New Kinect-based 3D Puppetry System

This video highlights a new Kinect-based 3D puppetry system developed by researchers at the University of Washington and the University of California, Berkeley that allows users to create 3D animation with minimal experience.

What's somewhat unique about this system is that unlike most Kinect-based systems, instead of the using body movement to control an onscreen character or object, users manipulate physical objects to control corresponding 3D models on a virtual set. I like this approach to digital puppetry a lot, because it retains the basic concept of puppetry (real-time manipulation of a physical object). To demonstrate how effective they believe this approach is, the researchers used novices in puppetry and animation to test the system.

Although the system these researchers have developed appears to be somewhat limited - note that the objects in the demo are solid objects without any kind of skeleton or use of Inverse Kinematics or Forward Kinematics - systems like this are very positive developments for digital puppetry. It would be great to see this get out of the lab and be developed in to some kind of open source platform.

A technical paper on this project can be found here.

How to make your own Waldo

Ever wanted to create your own Waldo-like control and perform real-time digital characters? Friedrich Kirchner demonstrates how it's done using an arduino microcontroller, sensors, buttons and his free Moviesandbox software.

Another Blender / Kinect Demo

This is an early test of Kinect and Blender working together using OSCeleton. This is a custom coded solution that's different than the NI Mate software that I mentioned in my previous post. As you can see, it's not quite production ready yet, but it's still very, very exciting to see Blender and Kinect working together. Full details can be found here.

Great work Machinima Studios!

Power Puppets

Power Puppets are a series of digital puppet characters produced by Little Mountain Productions, a firm that provides a variety of design and production services to Christian churches.

There are a variety of Power Puppet 2D and 3D characters available. The characters are controlled using an X-Box controller. The software is available for both Windows and Mac and is sold as a standalone product, although Little Mountain also builds complete hardware systems as well.

Virtual Marionette Project

Luis Miguel Barbosa is doing some interesting experiments with digital puppetry. His Virtual Marionette project uses an iPad to control a digital puppet character inside Animata (see previous posts). This takes advantage of the iPad's multi-touch interface to control multiple control points simultaneously in real-time.

He's also experimenting with something he calls WIIMATA, which uses two WIImotes to control a character inside Animata:



Neat stuff.

Puppet Time

Puppet Time is a new project from students at Georgia Tech that aims to "bring the art of digital puppetry to a wide range of users by taking advantage of the motion tracking capabilities of Android smart phones".

There have been a lot of projects very similar to this in the last few years so this isn't terribly new, although the use of Android smart phones as controllers is an interesting twist. Puppet Time was built using Unity 3D; it works by having Android phones send sensor data that controls the onscreen digital puppets to a computer running the main Unity application over a wifi network.

If you want to try it out for yourself, both the Unity and Android apps are available to download for free.

Motion Capture, Performance and Avatar

Some excerpts from a "making of" documentary about Avatar and its pioneering use of motion capture.

Facial Puppetry: Face tracking and replacement

I've been casually doing some research in to face tracking and face substitution techniques for a film I'm currently directing and producing. I'm exploring the idea of blending conventional "Bunraku-style" puppetry techniques with real-time animation. What I'm hoping to be able to do is replace the face of a physical puppet with an animated one (preferably in real-time, although I might have to settle for some kind of post-production process).

The basic concept is to take a person’s expressions and map them in real-time to either a digital model of a face or match the expression with a photo of another face in an image database. There are a number of people doing some interesting work in this area, especially Jason Saragih who has created the FaceTracker library; you can see an example of what it can do in the video embedded above (I mentioned this briefly in my previous post).

This isn't really a new idea. Companies like ILM and Rhythm and Hues pioneered similar post production techniques back in the 1990s and they've been used for years to create talking animals in commercials, movies and TV shows. There has also been similar commercial software like CrazyTalk around for several years. What is relatively new is that the technology now works in real-time and is accessible to anyone with a decent computer and some basic programming knowledge.

Technology like this further blurs the line between puppetry and animation (which has been happening for awhile now) and offers artists the chance to have the best of both worlds. I really love the idea of being able to endow conventional puppets with a level of expression that just isn't possible in a physical puppet. Exciting stuff.

Digital Puppetry Round-up

It's been far too long since I've updated Machin-X, so to try to catch up here's a look at bunch of interesting things that have been happening in the past few months:

Carnegie Mellon University and Disney Research are working on a system for "motion capture on the go" that utilizes body mounted cameras, which opens up some intriguing technological possibilities (via Tech Crunch).

In Australia, Jason Saragih is working on an interesting system for doing facial puppetry. His work on facial expression recognition looks particularly impressive so far.

Check out the visual insanity that is the Unreal 3 graphics engine.

Using the Kinect for digital puppetry keeps getting easier.

Digital puppetry with Wiimotes, using the Unity 3D engine.

Blender continues to be expanded and refined. The latest stable version (and last of the 2.5 series) Blender 2.59 was recently released and a number of exciting new features are being planned for future releases, including better support for motion capture and an improved real-time animation system.

Finally, old puppetry meets new technology with the "Naked Puppet" digital sock puppet.

Another Real-time Digital Puppetry Demo

This is a fun little demo of a digital puppet that Seth Hunter is working on with puppeteer Dan Butterworth at the MIT Media Lab. Like a lot of digital puppetry projects such as Animata, it draws a lot of inspiration from shadow puppetry techniques. The software works by tracking points on a puppet that correlate in real-time to a digital version of the character. Users have the ability to change costumes and record animations and there are future plans to enable character customization as well as remote play.

Flash Animation Using Kinect

The Dead Man and the Lawyer is a short film and experiment by Nick Fox-Gieg, a Canadian animator and all-round interesting guy I've had the pleasure of getting to know over the past few months. Although this might look like relatively straight forward 2D animation, he actually made it using the Kinect and I think this opens up some exciting possibilities for creating Kinect animation.

To make the film Nick used OSCeleton to record Kinect data to an .xml file via a utility he's written called Processing Sketch. The beauty of this approach is that the motion capture information was written to an .xml file in real-time without any need for video capture. That way the animation data can be read back from .xml in Flash, tweaked and then rendered in full HD. A short tutorial and download links for all the necessary utilities and libraries you need to try this yourself can be found on Nick's website.

Nick's also a semifinalist in YouTube's NextUp competition right now. If you like his work, show him some love and vote for him.