Hakanaï: Dancing with Digital Puppetry

Hakanaï is one of the more unconventional examples of a digital puppetry performance I've discovered (although, is there anything truly "conventional" about any form of digital puppetry?). Its creators describe it as a "haiku dance performance taking place in a cube of moving images projected live by a digital performer".

The performance involves a dancer performing live, whose movements are tracked in real-time and used as the basis for an interactive, digitally animated environment that is projected around them:

 

It was created by the French Company Adrien M / Claire B using their proprietary software eMotion. Here's more from their description of the project:

 ...Performed by an artist as a “digital score”, it is generated and interpreted live. The dancer’s body enters into a dialogue with the moving images in motion. These simple and abstract black and white shapes behave according to physical rules that the senses recognise and to mathematical models created from the observation of nature.

The audience experiences the performance in several stages. They first discover the exterior of the installation. As the dancer arrives, they gather around to watch the performance. When the choreography has ended, the audience can then take some time to wander amongst the moving images.

Through a minimalist transposition, this piece is based on images drawn from the imaginary realm of dreams, their structure and their substance. The box in turns represents: the bedroom where, once the barrier of sleep is passed, walls dissolve and a whole new inner space unfolds; the cage, of which one must relentlessly test the limits; the radical otherness, as a place of combat with an intangible enemy; the space where impossible has become possible, where all the physical points of reference and certitudes have been shaken.

Through the encounter of gesture and image, two worlds intertwine. The synchronicity between the real and the virtual dissolves and the boundary that was keeping them separate disappears, forming a unique space filled with a high oneiric charge. 

Very cool, no? You can learn more from the video's description on Vimeo.

Multitouch Puppetry

There has been an explosion of experiments with various forms of digital puppetry in the past couple of years, but we're still lacking a true "killer app" that makes real-time manipulation of digital characters simple an intuitive. That doesn't mean that progress isn't being made though; little breakthroughs are being made all the time. I've seen a number of interesting experimental interfaces in the past year, including this one created by Quan Nguyen and Michael Kipp, who have attempted design a simple, user-friendly multitouch system that can be used to control the complex movement of the human arm.

From their research:

"Controlling a high-dimensional structure like a 3D humanoid skeleton is a challenging task. Intuitive interfaces that allow non-experts to perform character animation with standard input devices would open up many possibilities. Therefore, we propose a novel multitouch interface for simultaneously controlling the many degrees of freedom of a human arm. We combine standard multitouch techniques and a morph map into a bimanual interface, and evaluate this interface in a three-layered user study with repeated interactions. The multitouch interface was found to be as easy to learn as the mouse interface while outperforming it in terms of coordination...Our results show that even complex multitouch interfaces can be easy to learn and that our interface allows non-experts to produce highly coordinated arm-hand animations with subtle timing." 

Quan and Michael are members of EMBOTS (Embodied Agents Research Group), a research group based in Germany. They've worked on a number of interesting projects with digital puppetry applications; you can find an overview of their work here.

Georgia Tech unveils new way to control CG characters without skeletons

A team of researchers at Georgia Tech have developed an innovative new way to control CG characters and objects without a skeletal structure like worms, Jello-like blobs of goo and even the human tongue. The system eschews the typical bones-based skeleton control approach that is commonly used in computer animation in favour of a new approach that involves simulating soft bodies.

The researchers claim that the result is non-skeletal characters that can be easily animated using simple point-and-click mouse movements and/or touchscreen gestures, which opens up all kinds of exciting possibilities for accessible animation including possible applications for real-time animation. You can find more information and a detailed technical explanation here.

Via Cartoon Brew Biz.

How to use anything as a digital interface

Here's an interesting project from Disney Research and Carnegie Mellon University...a system called Touché that can be used to create gesture controls on everyday, non-computer objects. Cool stuff.

Hmmm...can anyone think of possible digital puppetry applications for this?

Via Ars Technica.

SoftKinetic Digital Puppetry Demo

SoftKinetic used a digital puppetry demo to show off some new firmware for their DepthSense cameras at last week's CES 2012 trade show. It highlights their new support for finger recognition, something that's difficult to accomplish with the X-Box Kinect without a lot of hacking. The firmware is still in the alpha stage, but it looks like SoftKinetic is putting together a gesture recognition system that could be really useful for digital puppetry applications.

Via The Verge.

Virtual Marionette Project

Luis Miguel Barbosa is doing some interesting experiments with digital puppetry. His Virtual Marionette project uses an iPad to control a digital puppet character inside Animata (see previous posts). This takes advantage of the iPad's multi-touch interface to control multiple control points simultaneously in real-time.

He's also experimenting with something he calls WIIMATA, which uses two WIImotes to control a character inside Animata:



Neat stuff.

Digital Puppetry Round-up

It's been far too long since I've updated Machin-X, so to try to catch up here's a look at bunch of interesting things that have been happening in the past few months:

Carnegie Mellon University and Disney Research are working on a system for "motion capture on the go" that utilizes body mounted cameras, which opens up some intriguing technological possibilities (via Tech Crunch).

In Australia, Jason Saragih is working on an interesting system for doing facial puppetry. His work on facial expression recognition looks particularly impressive so far.

Check out the visual insanity that is the Unreal 3 graphics engine.

Using the Kinect for digital puppetry keeps getting easier.

Digital puppetry with Wiimotes, using the Unity 3D engine.

Blender continues to be expanded and refined. The latest stable version (and last of the 2.5 series) Blender 2.59 was recently released and a number of exciting new features are being planned for future releases, including better support for motion capture and an improved real-time animation system.

Finally, old puppetry meets new technology with the "Naked Puppet" digital sock puppet.

Chris O’Shea's Little Magic Stories

The more artists explore new frontiers in interactive art, the more I often mind myself wondering where exactly to draw the line between what is animation and what is puppetry. A good example of this is the augmented shadow puppetry that I recently wrote about over at PuppetVision, which is sort of shadow puppetry, but also sort of not.

Another good example is this performance system created by Chris O’Shea called Little Magic Stories. It was made using the X-Box Kinect (see previous post) and the old Pepper's Ghost Vaudeville trick.

So is this animation? Digital puppetry? Or something else entirely?

Via CrunchGear.

Kinect-Powered Hologram

Another amazing Kinect hack...researchers at MIT built a rudimentary holographic video system using an X-Box Kinect. They even re-create the famous hologram of Princess Leia from Star Wars in this demo video.

Real-time, 3D holographic digital characters anyone?

Via Fast Company.

More Kinect Digital Puppetry Developments

New X-Box Kinect developments seem to keep coming fast and furious. At CES 2011, Microsoft has been demoing it's forthcoming Avatar Kinect for X-Box Live; essentially it's a platform that allows you to control a 3D avatar of yourself in real-time via body and facial movements. I suspect that the technology isn't quite ready to go yet (Microsoft CEO Steve Ballmer's demo above was canned and not done live), but Engadget has a video of a live demo of Avatar Kinect here.

Meanwhile, in other Kinect news, someone has hacked together some impressive-looking real-time motion capture using the Kinect:



(Motion capture video via Blendernation)

Puppetry on the iPad

I'm really intrigued by the possibilities that Multi-touch devices like the iPad might offer puppeteers in the near future. Several developers are exploring exactly that and looking for ways to combine puppetry and interactive media. One of these is iPuppeteer, which allows you to control a simple, marionette-like puppet on an iPad, iPhone or iPod Touch. Another is Pollock's Toy Theatre, a new iPad app that rides the wave of resurgent interest in toy theatre that's been happening during the past few years and tries to take it in to the digital age.

While both of these apps seem to offer only basic functionality (for now), it will be interesting to watch the app market and see what develops over the next year or so.

Cross-posted from PuppetVision.

Digital Puppetry with Microsoft's Touchless

Touchless is the name of new multitouch software from Microsoft Office Labs. Much like the recently-blogged CamTrax software (see previous post), it allows you to use almost anything to create input for a computer via an ordinary webcam. MS Labs has released a software developer kit under an open-source Microsoft Public License that can be used to create custom apps, like controllers for digital puppetry purposes.

I really think that the concept of optically tracking everyday objects (I believe the technical term is "colour blob tracking") has enormous potential for (affordable) digital puppetry; much more so than motion capture. I've added this to my already-lengthy list of technologies to check out as soon as I have some time to spare.

Via TechCrunch.

A Wii For Everyone

Okay, this is very exciting...a company called Cam-Trax Technologies has developed software that allows almost any PC webcam (they say 95% of webcams are supported) to track up to four objects in real-time and with very high accuracy and reliability. Essentially, this allows you to "build your own Wii" (minus the $250 console and controllers) and opens up lots of very exciting possibilities.

The way it works is that you download an emulator to your PC (it only works with Windows unfortunately) and build a controller for your favourite video game. While the four objects is a little limiting, I don't see why this couldn't be used for conrolling digital puppets.

I have to play around with this sometime in the next week or two.

Via TechCrunch.

Video and Blender's Game Engine

Work continues with Panda Puppet slowly, but surely. I haven't been able to share much of what I have been doing lately because my work is focused on two client projects that I can't really blog about at this point, but I am starting up my Bear Town web series project again and plan to be using Panda Puppet heavily on that in the coming months.

What has me most excited right now though is that "Ashsid" - who wrote a lot of the original Blender scripts that much of Panda Puppet is based on - has been working on a cool new video texture plugin for Blender's game engine that allows you to combine real-time graphics with various other sources like video files, live video, rendered 3d scenes, etc. inside Blender.

This has all sorts of interesting applications, the coolest of which is tracking the movement of a camera and applying that movement to a Blender object. Here's a demo of it in action with Suzanne (the famous Blender monkey head) combined with live webcam video:



What excites me most about this is that it enables puppeteers working with monitors to have physical puppets and digital ones interact in real-time. That isn't a new idea of course, but what's great about this is that now anyone can download the software and try it themselves.

Nice work Ashsid!

Head tracking with the Wiimote

Johnny Chung Lee (see previous post) has done it again! In his latest video he explains how to use the Wiimote and Wii sensor bar to make a head tracking system. As always, you can download the software to do this yourself from Johnny's site (link goes directly to a .zip file).

The most obvious application that I can see for this in digital puppetry is a quick and easy way to track a camera in 3D space. It could also be used to allow a puppeteer to see through a character's "eyes" in a virtual environment/set.

So many things to do with the Wiimote, so little time!

Multi-touch Interface using the Wii Remote

This is another incredibly cool Wiimote hack from Johnny Chung Lee (see previous post) that suggests all kinds of interesting possibilities for controlling digital puppet characters in real-time. Since the Wii tracks IR lights, Johnny has discovered that almost any surface - a projector screen, a tabletop, or an LCD - can be turned into a multi-touch screen that you can use like the iPod touch or iPhone. All you need is the Wiimote, a ball point pen equipped with an LED and some free software that can be downloaded here. Watch the video for details.

And as cool as it is, I'm even more excited to see the results of what Johnny is working on next - 3D tracking and head tracking with the Wiimote!

Special thanks to Brian for sending this to me!

Track Fingers With The Wiimote

Here's another potentially cool user interface for creating digital puppetry and Machinima...Johnny Lee at Carnegie Mellon University has worked out a method for tracking his fingers using the Wiimote's infrared camera. The basic idea is very similar to the iPhone and the Gesturetek technology I wrote about back in March.

Via MAKE.

Do-It-Yourself Cintiq Display

I have to look in to building a dedicated digital puppetry workstation for work which has me thinking about all sorts of possibilities for controllers and input devices. One thing I would absolutely love to incorporate is a Cintiq. If you aren't familiar with these, Cintiqs are hybrid monitors/graphics tablets that allow you to "draw" on a screen and interact with a computer using a stylus. I think this is a much more elegant interface for a production environment than having puppeteers fiddle with a keyboard and mouse while they are putting together a performance. There's only one problem - money. The Cintiq costs about $2,500 US which is a difficult cost to justify for a somewhat unnecessary (albeit very nice) peripheral.

Luckily, the underlying technology in the Cintiq isn't new or terribly expensive, you just have to know how to put it together and of course some enterprising artist has already built their own Cintiq for roughly $150 US. The preceding link goes to their build log and you can watch a video of it in action above. It looks pretty nice!

This is slightly beyond my current level of electronics experience, but I just may try building one as an experiment if I can round up the necessary parts.

Controlling Video With Hand Gestures

A company called Gesturetek has some very interesting looking technology that allows you to control video using hand gestures. It's essentially a more sophisticated version of Puppet Show, which I've mentioned here before. As you can imagine from this brief video, there are lots of potential puppetry applications for something like this.

How to Puppeteer a Head in Machinima

Lip sync and effective control of a characters' head on screen has been a difficult challenge for Machinma creators. Unless you count the high-end real-time animation being done by Disney and Henson, I haven't seen any Machinima with effective animation of a character's head. There have been some noble attempts, but all of them look pretty stiff and boring to me.

An interesting solution to this problem that I have been toying with for awhile is to have the computer visually track the movement of a puppeteer's hand, much like the way that the Nintendo Wii tracks the movements of players holding a Wii remote. Michael Nitsche, an Assistant Professor at Georgia Tech recently emailed me about PuppetShow, a very cool looking system some of his students have put together to control the heads of digital puppets with the Unreal Game Engine.



As you can see in the video, PuppetShow tracks the coloured object and the digital puppet moves on screen accordingly. This isn't new technology of course, but what I think is really cool about it is how accessible it is...all you need is a coloured piece of paper, a USB webcam and some free software. It's a limited system right now, but I like the way it puts head movement back (literally) in the hand of the puppeteer.

Basic lip sync can be achieved in a system like this through the movement of the puppeteer's thumb, but the puppet's mouth can't enunciate and the results are still pretty stiff. Many Machinima creators have suggested using some kind of voice analysis or lip sync software like Magpie as a way to create lip sync and enunciation, but I like the idea of a puppeteer having direct control over all aspects of the puppet's head.

In conventional puppetry puppeteers do not actually articulate each word that a character says. The puppet's mouth usually only opens once for each syllable that the character speaks; the mouth is open on vowel sounds (a-e-i-o-u) and closed on consonants. Therefore to achieve realistic lip sync a puppet doesn't have to be able to enunciate perfectly, it just has to be rigged to be able to open and close it's mouth and make these four different shapes:



Each of these mouth shapes could be pre-established as a keypose on a digital puppet and triggered by the puppeteer pressing a button (possibly located inside some kind of data glove or control mitt) or making a specific gesture.

By combining this approach to lip sync with the visual tracking of PuppetShow, Machinima creators could really, finally have a great system for puppeteering their characters' heads in real time.