Tuesday, March 13, 2007
Controlling Video With Hand Gestures
A company called Gesturetek has some very interesting looking technology that allows you to control video using hand gestures. It's essentially a more sophisticated version of Puppet Show, which I've mentioned here before. As you can imagine from this brief video, there are lots of potential puppetry applications for something like this.
Monday, March 12, 2007
What you could do in Machinima
Brian Stokes, whose blog I always enjoy reading, linked to the fantastic French student animation film Burning Safari a little while ago alluded to the fact that it's difficult to do something like it in Machinima. While I agree completely that it would be nearly impossible to make Burning Safari with most existing Machinima and digital puppetry tools, after spending sometime looking at the films in terms of what I am working on with Panda Puppet I think this type of animated film could definitely be done using real-time techniques.
As a theoretical exercise I broke down the main sections of the film and I'll describe here how would approach them using digital puppetry techniques:
The Spaceship Landing - Easy enough to do, lots of video games already have vehicles like this in them. `Nuff said.
The Little Robots - These guys sure are cute. There are two possibilities for performing them that I can imagine; the simplest way would be to create walk cycles for them and then having a puppeteer simply control their X-Y movement within a scene. Another possibility would be to use a data glove on a flat surface like a table with each leg controlled by a different finger. This would allow more spontaneous and specific control, but running around might be difficult if you didn't have a large enough space to perform in.
"Robo-Vision" - To be perfectly honest, I would cheat this shot in After Effects.
The Monkey - Boy, I love monkeys. There's just something funny about them. In fact, he bares a striking resemblance to "Suzanne" the Blender monkey-head primitive. Controlling a character like this will be relatively straight forward with a control system like Panda Puppet.
See Monkey Get Mad - The only thing funnier than a monkey is an angry monkey. This type of moment in a scene seemed like a real challenge at first, but I have been playing with an idea I call "Emotion State Control" which would influence the movements and poses of a character according to the character's emotion state. If the emotion state "happy" is assigned to a joystick's trigger as long as the trigger is pressed all of the "happy" versions of different movements and poses would be used.
See Monkey Run - Once again, running is done using simple walk cycles. Leaping, jumping, etc. is fairly easily using a combination of walk-cycles and physics. Even the squash and stretch is fairly easy to achieve.
See Monkey On Fire - Ragdoll physics would work nicely in this situation. I would do the energy squares or whatever the robots are chucking at the monkey and the fire on the monkey's face in post using After Effects or something similar.
The other big problem that needs to be addressed is image quality. The graphics in real-time rendering are tied to a computer's ability to draw polygons onscreen. The more polygons a computer can draw and the faster it can draw them, the better the graphics. So having high resolution real-time graphics is really just a matter of having enough computational power. With enough of it, theoretically, you can render graphics of this quality in real-time. In fact I've seen real-time 3D demonstrations that exceed the quality of Burning Safari. Another work around is to simply do performances in real-time at low resolution and then render out higher quality resolution frame by frame afterwards.
There you have it, easy-peasy. Well, almost.
I don't mean to suggest that something being technically doable means it's easy artistically. Making great films is hard no matter how you do it. And even when I get Panda Puppet or someone else gets another digital puppetry system to the point where it can do all this all it will be is just a great tool. Having great tools is awfully nice, but what ultimately makes great art is a great artist.
As a theoretical exercise I broke down the main sections of the film and I'll describe here how would approach them using digital puppetry techniques:
The Spaceship Landing - Easy enough to do, lots of video games already have vehicles like this in them. `Nuff said.
The Little Robots - These guys sure are cute. There are two possibilities for performing them that I can imagine; the simplest way would be to create walk cycles for them and then having a puppeteer simply control their X-Y movement within a scene. Another possibility would be to use a data glove on a flat surface like a table with each leg controlled by a different finger. This would allow more spontaneous and specific control, but running around might be difficult if you didn't have a large enough space to perform in.
"Robo-Vision" - To be perfectly honest, I would cheat this shot in After Effects.
The Monkey - Boy, I love monkeys. There's just something funny about them. In fact, he bares a striking resemblance to "Suzanne" the Blender monkey-head primitive. Controlling a character like this will be relatively straight forward with a control system like Panda Puppet.
See Monkey Get Mad - The only thing funnier than a monkey is an angry monkey. This type of moment in a scene seemed like a real challenge at first, but I have been playing with an idea I call "Emotion State Control" which would influence the movements and poses of a character according to the character's emotion state. If the emotion state "happy" is assigned to a joystick's trigger as long as the trigger is pressed all of the "happy" versions of different movements and poses would be used.
See Monkey Run - Once again, running is done using simple walk cycles. Leaping, jumping, etc. is fairly easily using a combination of walk-cycles and physics. Even the squash and stretch is fairly easy to achieve.
See Monkey On Fire - Ragdoll physics would work nicely in this situation. I would do the energy squares or whatever the robots are chucking at the monkey and the fire on the monkey's face in post using After Effects or something similar.
The other big problem that needs to be addressed is image quality. The graphics in real-time rendering are tied to a computer's ability to draw polygons onscreen. The more polygons a computer can draw and the faster it can draw them, the better the graphics. So having high resolution real-time graphics is really just a matter of having enough computational power. With enough of it, theoretically, you can render graphics of this quality in real-time. In fact I've seen real-time 3D demonstrations that exceed the quality of Burning Safari. Another work around is to simply do performances in real-time at low resolution and then render out higher quality resolution frame by frame afterwards.
There you have it, easy-peasy. Well, almost.
I don't mean to suggest that something being technically doable means it's easy artistically. Making great films is hard no matter how you do it. And even when I get Panda Puppet or someone else gets another digital puppetry system to the point where it can do all this all it will be is just a great tool. Having great tools is awfully nice, but what ultimately makes great art is a great artist.
Sunday, March 11, 2007
Blender Gets Even Better
Plans are afoot to add motion tracking capability to Blender. This is still in the earliest stages of development, but it looks like the idea is to add the ability to both match-move camera shots and track motion on a object that can be assigned to Blender controls. These were two huge features I really wanted for Panda Puppet (and dreaded the prospect of having to code on my own) and now it looks like they'll end up built right in to Blender.
How cool is that?
How cool is that?
Saturday, March 10, 2007
Panda Puppet: Head Control Test #2
A second head control test, just some basic left/right movement, but now with an articulated jaw for extremely basic lip sync is possible. The second clip utilizes deformation to create a squash/stretch effect as the mouth opens. The third clip has two different keyposes triggered by pressing different joystick buttons. I also smoothed out the surface of the head for the second and third clips. It's all very crude still, but coming along nicely I think.
Work on this will continue through the weekend...
Friday, March 09, 2007
Nintendo - A Digital Puppetry Pioneer
Most people don't know this, but Nintendo is something of a pioneer in the field of digital puppetry. Since 1994 they have been using a system they call MIRT ("Mario in Real-Time") to have many of their iconic characters like Mario appear at trade shows and in-store promotional events. The clip of MIRT in action above shows everyone's favourite Italian plumber bashing Sony, doing a Jack Nicholson impersonation and re-enacting a scene from Star Wars.
The earliest version of MIRT was a facial tracking system that ran on a Silicon Graphics super computer, but over the years the technology has been refined to the point where puppeteer Charles Martinet (the voice of Mario, Luigi and many of Nintendo's other characters) is able to work off a laptop at home and virtually perform characters all over the word. Charles briefly discusses his career as Mario & the gang in this short interview with GT.tv from last summer.
Thursday, March 08, 2007
Panda Puppet: Head Control Test #1
Success!
After spending the day working out several bugs, here is a quick test of the head control system I am programming for Panda Puppet. It's nothing fancy, just some basic head movement (up/down, right/left, tilt), but it works quite nicely. The head is controlled using a joystick, but almost any kind of device imaginable could be used as a controller. If you've read my last post on the control system, this is an example of direct control.
In the morning I'm going to tinker with this a bit to get some smoother animation and add some eyes and basic facial expressions so I can start playing with poses.
Wednesday, March 07, 2007
Controlling Digital Puppets
Since a few people have emailed me asking about this, here is some more info on the control system I am developing for Panda Puppet and how it works, or at least how I am currently trying to make it work.
Panda Puppet is shaping up as a Python plug-in for Blender. At this stage I am not trying to make it do anything that isn't already possible in Blender's Game Engine, I just want to streamline the way characters can be set-up and controlled in real-time. The core of Blender's GE is Logic Bricks (sometimes called Logic Blocks) which are used to set up and control interactions between different game elements like characters, props, etc.
This is what Blender's Game Logic Panel looks like:
Once again I won't go in to all the technical nitty-gritty of Logic Bricks here, but if you want to learn more just click on the link above.
Panda Puppet's control system is a simplified interface for quickly setting up Logic Bricks in way that's ideal for controlling digital characters. I am still in the early stages of programming this so all of it may change, but this is how I am designing the control system right now:
Sensor Type
The sensor type is the type of device used by the puppeteer to control on an on screen character or object. This can be almost any kind of device imaginable, including a joystick, keyboard, gamepad, dataglove or even a Wiimote. I want Panda Puppet to be as flexible and "controller agnostic" as possible. Rather than forcing a puppeteer to adapt to a specific type of input device, I want a control system can be customized to the needs and preferences of a puppeteer and I want different puppeteers performing in a scene together to use different types of controls if they want.
Sensor Input
Sensor inputs are specific types of inputs from an input device. For example, the press of a button on a keyboard or the x-y axis movement of a joystick.
Sensor Action
Sensor Actions are control types that govern a character's action and determine what happens when a puppeteer uses a specific sensor input. There are three different types of Sensor Actions - direct controls, pose controls and emotion controls.
Direct Control
Using direct control a puppeteer has direct control over the movement of a specific part of a digital character. For example, The x-y "walking" movement of a character can be assigned to the x-y axis movement of a joystick so that when the joystick moves in a specific direction the character walks in a corresponding direction.
Pose Control
Pose Control is used to assign specific poses or movements to a specific sensor input. For example, an elaborate shriek or a comedic double-take could be triggered by a joystick button.
Emotion State Control
Emotion State Control influences the movements and poses of a character according to the character's emotion state. An emotion state can be assigned to a specific sensor input and triggered when that control is activated and will influence the character as long as the input control remains active. If the emotion state "happy" is assigned to a joystick's trigger as long as the trigger is pressed the character will remain "happy".
Typically, there are six primary emotional states - anger, disgust, fear, joy, sadness and surprise. Just as primary colours can be modified and mixed to create every other imaginable colour, primary emotional states can also be mixed to create every emotional state imaginable. For example, Anger + Disgust = Rage. Emotion state combinations can be triggered by activating a combination of sensor inputs (when primary emotion states are assigned to individual sensor inputs) or mixed emotion states can be grouped and triggered by a single sensor input.
If anyone out there has thoughts, ideas and/or suggestions about all of this please feel to drop me a line at puppetvision {at} gmail dot com.
Panda Puppet is shaping up as a Python plug-in for Blender. At this stage I am not trying to make it do anything that isn't already possible in Blender's Game Engine, I just want to streamline the way characters can be set-up and controlled in real-time. The core of Blender's GE is Logic Bricks (sometimes called Logic Blocks) which are used to set up and control interactions between different game elements like characters, props, etc.
This is what Blender's Game Logic Panel looks like:
Once again I won't go in to all the technical nitty-gritty of Logic Bricks here, but if you want to learn more just click on the link above.
Panda Puppet's control system is a simplified interface for quickly setting up Logic Bricks in way that's ideal for controlling digital characters. I am still in the early stages of programming this so all of it may change, but this is how I am designing the control system right now:
Click on the diagram to enlarge it
Sensor Type
The sensor type is the type of device used by the puppeteer to control on an on screen character or object. This can be almost any kind of device imaginable, including a joystick, keyboard, gamepad, dataglove or even a Wiimote. I want Panda Puppet to be as flexible and "controller agnostic" as possible. Rather than forcing a puppeteer to adapt to a specific type of input device, I want a control system can be customized to the needs and preferences of a puppeteer and I want different puppeteers performing in a scene together to use different types of controls if they want.
Sensor Input
Sensor inputs are specific types of inputs from an input device. For example, the press of a button on a keyboard or the x-y axis movement of a joystick.
Sensor Action
Sensor Actions are control types that govern a character's action and determine what happens when a puppeteer uses a specific sensor input. There are three different types of Sensor Actions - direct controls, pose controls and emotion controls.
Direct Control
Using direct control a puppeteer has direct control over the movement of a specific part of a digital character. For example, The x-y "walking" movement of a character can be assigned to the x-y axis movement of a joystick so that when the joystick moves in a specific direction the character walks in a corresponding direction.
Pose Control
Pose Control is used to assign specific poses or movements to a specific sensor input. For example, an elaborate shriek or a comedic double-take could be triggered by a joystick button.
Emotion State Control
Emotion State Control influences the movements and poses of a character according to the character's emotion state. An emotion state can be assigned to a specific sensor input and triggered when that control is activated and will influence the character as long as the input control remains active. If the emotion state "happy" is assigned to a joystick's trigger as long as the trigger is pressed the character will remain "happy".
Typically, there are six primary emotional states - anger, disgust, fear, joy, sadness and surprise. Just as primary colours can be modified and mixed to create every other imaginable colour, primary emotional states can also be mixed to create every emotional state imaginable. For example, Anger + Disgust = Rage. Emotion state combinations can be triggered by activating a combination of sensor inputs (when primary emotion states are assigned to individual sensor inputs) or mixed emotion states can be grouped and triggered by a single sensor input.
If anyone out there has thoughts, ideas and/or suggestions about all of this please feel to drop me a line at puppetvision {at} gmail dot com.
Subscribe to:
Posts (Atom)