Gadzooks Studio, a Manchester-based stop motion animation studio founded by animation directors Steve Boot and Haydn Secker, has been sharing some really delightful digital puppetry experiments on Instagram. They’re building a Blender-based pipeline that utilizes TouchDesigner, which lets artists and technologists build interactive animation systems without writing traditional code.
What Gadzooks is doing here is using TouchDesigner to feed their live performance in to Blender, which acts as the character/animation environment. This allows them to create digital puppets that respond to a performer in real time: facial expressions,
head turns, timing, reactions, and little bits of character business.
TouchDesigner, created by Toronto-based Derivative, is a big part of Gadzooks’ not-so-secret sauce (although, to their credit, they’ve been sharing much of their workflow on Instagram). TouchDesigner is
able to take a variety of live inputs - everything from camera footage to audio, MIDI, sensors, controllers, or just about anything else that outputs a signal -
and use that input to control a character in real time without traditional hand-keyed animation. It’s a more sophisticated version of what I was trying to build with Panda Puppet many years ago - a way to make it possible to use almost anything to perform a digital character.
It’s a really great example of the advantage digital performance has over AI-based prompting: the immediacy of puppetry, with the flexibility and control of a digital animation pipeline. The best of both worlds.
I'm excited to see what others might be able to do with TouchDesigner, as well as what else Gadzooks is working on. To keep up-to-date with them, follow Gadzooks.Studio on Instagram!
Lyle at LJ Puppetry recently shared this digital puppetry experiment. Lyle was inspired by the Henson Digital Puppetry Studio and its use of a Waldo, but wanted to see if he could create digital shadow puppets that were controlled in real-time via motion capture inside Blender. Lyle himself admits that his results were mixed, but I think his project may indirectly suggest some really interesting technological possibilities.
Of course, experimenting with digital shadow puppets is a well-trodden path. I began my own journey in to real-time animation experimenting with digital shadow puppets when I first started writing Machin-X, although back then we had to rely on colour blob tracking and
custom scripting in Adobe Director because there were no off the shelf
tools.
Today, it's much, much easier to do this with off-shelf open source software like Blender, which is what Lyle used to perform this multi-scene shadow story - a magi, a tiger that transforms into a woman, a figure that coalesces from stars - entirely through hand and finger movements captured by a webcam:
The shadow characters in the video are flat black figures built using Blender's
grease pencil tool. Lyle controlled them in real time by mapping specific finger
positions to specific puppet movements — thumb open stands the magi up, a
closed fist brings him to his knees. What we're seeing in the video isn't
playback; it's a live performance into Blender, with the puppeteer
off-camera and the output on screen.
Although his experiment wasn't entirely successful, Lyle's documentation is an
excellent starting point for someone else who wants
to try experimenting with this on their own. You can read an in-depth write-up of his workflow here.
This might be the right approach with the wrong tool
Personally, I suspect that BlendArMocap might be the wrong implementation of the right technology. It's notoriously difficult to manage motion tracking inside Blender itself (which is what BlendArMocap does), but the underlying technology that makes that markerless motion tracking on a standard webcam possible, Google's open source Mediapipe framework, is brilliant.
TouchDesigner's node-based interface can look intimidating, but it's a popular tool used by artists all over the world.
Mediapipe offers real possibilities for digital puppetry. The ability to use markerless motion capture - especially in a tool like Blender's innovative Grease Pencil - is tantalizing, but a better way of utilizing it might be with a program like TouchDesigner. TouchDesigner is used by artists and performers to process, and react to live data — from motion capture to audio to video — in real time. If you've seen a live concert where the visuals pulse and shift in sync with the music, that was likely done with TouchDesigner, and it can also send clean motion data directly into Blender in real time, via OSC.
Frankly, it's much easier to fix problems like "jerky mocap" in TouchDesigner before the data ever reaches Blender, which in turn creates a much more streamlined workflow.
We need more experiments like this
I love what Lyle has shared here, and that he is encouraging others to learn from his project. He's given me some things to think about, and I'm very excited to see what others do with Blender and tools like TouchDesigner and Mediapipe.
More than anything though, this experiment serves to
remind us that creating great puppetry is hard, creating digital puppetry is especially hard...and recreating shadow puppetry in particular is deceptively hard.
