Lyle at LJ Puppetry recently shared this digital puppetry experiment. Lyle was inspired by the Henson Digital Puppetry Studio and its use of a Waldo, but wanted to see if he could create digital shadow puppets that were controlled in real-time via motion capture inside Blender. Lyle himself admits that his results were mixed, but I think his project may indirectly suggest some really interesting technological possibilities.
Of course, experimenting with digital shadow puppets is a well-trodden path. I began my own journey in to real-time animation experimenting with digital shadow puppets when I first started writing Machin-X, although back then we had to rely on colour blob tracking and custom scripting in Adobe Director because there were no off the shelf tools.
Today, it's much, much easier to do this with off-shelf open source software like Blender, which is what Lyle used to perform this multi-scene shadow story - a magi, a tiger that transforms into a woman, a figure that coalesces from stars - entirely through hand and finger movements captured by a webcam:
The shadow characters in the video are flat black figures built using Blender's grease pencil tool. Lyle controlled them in real time by mapping specific finger positions to specific puppet movements — thumb open stands the magi up, a closed fist brings him to his knees. What we're seeing in the video isn't playback; it's a live performance into Blender, with the puppeteer off-camera and the output on screen.
Although his experiment wasn't entirely successful, Lyle's documentation is an excellent starting point for someone else who wants to try experimenting with this on their own. You can read an in-depth write-up of his workflow here.
This might be the right approach with the wrong tool
I've been following motion capture developments in Blender for a long time. I was very excited by the BlendArMocap plugin when I first heard about it, but I've since discovered that it has some real limitations. Dropped frames (the "jerky mocap" that Lyle mentions) and the ability to only perform one puppet at a time are two of the known issues.
Personally, I suspect that BlendArMocap might be the wrong implementation of the right technology. It's notoriously difficult to manage motion tracking inside Blender itself (which is what BlendArMocap does), but the underlying technology that makes that markerless motion tracking on a standard webcam possible, Google's open source Mediapipe framework, is brilliant.
 |
| TouchDesigner's node-based interface can look intimidating, but it's a popular tool used by artists all over the world. |
Mediapipe offers real possibilities for digital puppetry. The ability to use markerless motion capture - especially in a tool like Blender's innovative Grease Pencil - is tantalizing, but a better way of utilizing it might be with a program like TouchDesigner. TouchDesigner is used by artists and performers to process, and react to live data — from motion capture to audio to video — in real time. If you've seen a live concert where the visuals pulse and shift in sync with the music, that was likely done with TouchDesigner, and it can also send clean motion data directly into Blender in real time, via OSC.
Frankly, it's much easier to fix problems like "jerky mocap" in TouchDesigner before the data ever reaches Blender, which in turn creates a much more streamlined workflow.
We need more experiments like this
I love what Lyle has shared here, and that he is encouraging others to learn from his project. He's given me some things to think about, and I'm very excited to see what others do with Blender and tools like TouchDesigner and Mediapipe.
More than anything though, this experiment serves to remind us that creating great puppetry is hard, creating digital puppetry is especially hard...and recreating shadow puppetry in particular is deceptively hard.
