How OK Go Used Blender As a Puppet Stage

One of my favourite music videos of the past year is OK Go's Impulse Purchase, but it's not just an animated music video. It's a small, but important demonstration of how Blender can be used as a live puppet stage.

OK Go frontman Damian Kulash described the origin of his collaboration on this with directors Lucas Zanotto and Will Anderson at Blender Studio to Live Music Blog:

“It began with my love of Lucas Zanotto’s short animated loops — they’re so inventive, so full of joy, always delivering little doses of the kind of wonder we’re always searching for in our own videos. So I thought the universe flowing from his brain might be the perfect setting for a lyric video, but when I reached out to him, he had a more ambitious project to propose: a ‘live performance’ with me AS one of his characters. Suddenly it was a much weirder, more wonderful project than what I’d envisioned,”  

To make the video, Damian performed the character via a live facial motion-capture app on an iPhone. The app used OSC (the Open Sound Control protocol) to send his real-time eye, lip, and head movements into Blender, where they drove the character’s erratic facial expressions on the fly.

What makes this true digital puppetry, not just digital animation, is that Damian isn't simply supplying reference material for an animator to interpret; he is performing the character through a real-time control system. His face is the input device. Blender is the puppet stage.

Best of all, in keeping with Blender Studio's open source filmmaking ethos, a demo version of this set-up has been released for anyone to try:

To run the free demo you will need to download the following:

• Blender 4.5 LTS (the latest long-term support release, not the newest version of Blender)
• The .blend demo file
• Blender Studio's bespoke Foscap add-on
• The Live Link Face app for iOS or Android

The full .blend production file used to make Impulse Purchase is also available for Blender Studio subscribers. Complete step-by-step instructions can be found on the Blender Studio Blog. 

Character Animator - A 2D Real-Time Animation tool from Adobe

Adobe has just unveiled a new 2D digital puppetry - or have we all agreed to call this field real-time animation now? - application called Character Animator. A demo of the software is provided in the video above, but essentially it's a tool for animating 2D bitmap (Photoshop) and vector (Illustrator) still characters in real-time using a camera, microphone, head tracking and facial mo-cap.

In addition to basic mo-cap and lip sync capabilities, it also allows users to create programmable behaviors and will support the creation of 3rd party plug ins. I haven't tried it myself yet, but it looks like it could be a very easy-to-use and potentially powerful tool for creating basic 2D animated characters in real-time.

Adobe Character Animator is currently in beta and available for testing by After Effects CC users. You can learn more here.

Via The Labyrinth.

Faceshift Markless Motion Capture

Faceshift is software that promises "markless motion capture at every desk". It works with consumer-level cameras like the Kinect to track and analyze the facial expressions of a performer and uses them to animate a virtual character in real-time. It also offers the option of recording a performance so that it can be edited and polished in post-production.

There are lots of potential applications for this kind of software in game and film production and, of course, digital puppetry applications!

You can learn more at www.faceshift.com.

Flaming Skull Face Tracking Demo

Some clever face tracking from a Spanish studio called Paradox D&D. They've written a custom application that positions a 3D skull over top of the user's head, capturing the user's body movements and using them to control a 3D model in real-time. This is an example of a "virtual dresser" that allows a user to "wear" 3D elements in a manner very similar to augmented reality.

Via KinectHacks.

Facial Capture Using Blender 3D

The latest version of Blender 3D was released recently and has a bunch of great new features. One of the most impressive is motion tracking, which enables you to reconstruct camera animation from video and film footage, and composite 3d rendered object into movie clips. The Blender community has been busy exploring the possibilities of this new feature and several people have been attempting to use it for rudimentary motion capture.

As you can see above and below, the results so far look pretty good!





If you're feeling ambitious and want to try this for yourself you can find a tutorial here.

I've always believed that Blender had the potential to become an ideal (and totally free) professional tool for creating digital puppetry. Although I don't believe that this works in real-time - I haven't had a chance to try it myself yet - this is a huge step in that direction and suggests that we can look forward to a lot more amazing things in the future.

Via BlenderNation.

Facial Puppetry: Face tracking and replacement

I've been casually doing some research in to face tracking and face substitution techniques for a film I'm currently directing and producing. I'm exploring the idea of blending conventional "Bunraku-style" puppetry techniques with real-time animation. What I'm hoping to be able to do is replace the face of a physical puppet with an animated one (preferably in real-time, although I might have to settle for some kind of post-production process).

The basic concept is to take a person’s expressions and map them in real-time to either a digital model of a face or match the expression with a photo of another face in an image database. There are a number of people doing some interesting work in this area, especially Jason Saragih who has created the FaceTracker library; you can see an example of what it can do in the video embedded above (I mentioned this briefly in my previous post).

This isn't really a new idea. Companies like ILM and Rhythm and Hues pioneered similar post production techniques back in the 1990s and they've been used for years to create talking animals in commercials, movies and TV shows. There has also been similar commercial software like CrazyTalk around for several years. What is relatively new is that the technology now works in real-time and is accessible to anyone with a decent computer and some basic programming knowledge.

Technology like this further blurs the line between puppetry and animation (which has been happening for awhile now) and offers artists the chance to have the best of both worlds. I really love the idea of being able to endow conventional puppets with a level of expression that just isn't possible in a physical puppet. Exciting stuff.