Close Mobile Menu

Next Up: You, in 3D

September 17, 2009
by Katherine Luce

A Berkeley undergraduate named Charles stands in the middle of a three-meter cube. He moves slowly, watching a ghostly projection of himself move on a screen in front of him. The colors are murky, and his dark pants have completely disappeared, leaving empty space between his shirt and his shoes. Still, what’s up on the screen looks three-dimensional, like a hologram; it moves precisely as he does and is definitely him.

The cube, high up in the Hearst Mining Building on the Berkeley campus, looks like a boxing ring surrounded by equipment that includes 48 cameras and at least 13 computers. Its purpose is to enable long-distance collaboration by gathering full, three-dimensional information about anyone moving inside it, instantaneously creating the image, and sending it over the Internet. The team working on so-called tele-immersion (also known as three-dimensional video conferencing) comprises computer scientists, as one might expect, but also dancers and dance scholars. Charles is in the cube because he is taking CS 84, probably the only sophomore seminar in computer science to require choreography. The unlikely tele-immersion crew is led at Berkeley by electrical engineering professor Ruzena Bajcsy, is choreographed by Lisa Wymore from the theater and dance department, and includes counterparts at the University of Illinois. Though tele-immersion may seem like just another cool computer-generated effect, these researchers have grand ambitions for their project.

Some of those ambitions are practical, such as the ability to diagnose a disease from a long distance or document performances or enable simple video conferencing in three dimensions. But Wymore and Bajcsy also project that tele-immersion will help us reinvent our relationship with technology itself. What if we could use our whole human bodies, idiosyncratic and imperfect as they are, to communicate technologically, rather than just our keyboarded fingers, cell-phoned ears, and computer-screened eyes?

Bajcsy asserts that dance is a language, and she imagines using tele-immersion to analyze movement across cultures, just as written and spoken languages are analyzed. With the richness of information that tele-immersion captures, she postulates, every movement could be examined in exquisite three-dimensional detail and compared to similar movements from elsewhere. Imagine tracing the African diaspora through dance or seeing how the gypsies’ Indian roots affect their dance in different parts of the world.

But first there are some thorny technical problems to solve. The computers must separate the moving figure from its background and reassemble an entire three-dimensional human body from images of its surface, without losing more than a fraction of a second. (If they could delay by a few minutes, or even seconds, the task would be much easier.) In order for the system to reconstruct people well, they have to be wearing the right clothing. If they move too quickly, the tele-immersive image may disappear or show eight or ten arms at once. Right now, everything has to be done in super-slow motion. The system can also get confused between the front and back of the body and switch them, leaving the head facing forward over its owner’s rear end.

Tele-immersion makes most people think of motion-capture technology, but it’s quite different. In motion capture, the person wears a special suit, and it’s actually the movements of the suit that are recorded digitally. Then three-dimensional movement is reconstructed in a digital file. Motion capture is used in sports, say to correct a problematic golf swing, and in animation. It’s how the penguin Mumble in Happy Feet managed to tap like Savion Glover. Motion capture allows some of the indefinable qualities of human movement to be used in animations, but it doesn’t really convey the individuality of the person moving. Initially, all you have is a video of some dots moving in space, however, the digital files are relatively small and easy to work with. By contrast, the tele-immersion researchers are trying to capture the full human uniqueness instantly, so their digital files are very large.

Another difference between tele-immersion and motion capture or 3D animation is that so many of its applications involve active users. You might participate in long-distance training or rehabilitation or meet virtually with far-away family members. All without sitting at a keyboard. Once the camera system is portable (expected to happen in a couple of years), the possibilities will expand.

Meanwhile, the dancers make dances. They have performed together in a virtual meeting place while physically dancing in their respective boxing rings in California and Illinois. (See it online.) They have to be as intensely conscious of how they’re both moving as if they were in the same room, but they’re coordinating with someone who’s 2,000 miles away.

The dancers challenge the capabilities of the tele-immersive technology; they also inspire the creativity of its developers, expanding their ideas about it and offering new approaches. If we agree with Bajcsy that dance is a language, it’s one of three involved in this project. The other two are the language of technology, which is used to solve problems, and the language of scholarship, used to explain the meaning of it all. In order to bridge some of the gaps between these approaches, Wymore plans to train the engineers working on the project in Laban Movement Analysis, which describes a motion’s intention and emotion as well as what happens physically. Laban is part of every dance major’s world, and it offers the hope of being able to describe the subjective qualities of movement—love, say, or sadness—in numbers. The numbers are necessary for computer programming, but the intent is to make individual quirks part of the equation.

Wymore says that tele-immersion makes her think of death. Perhaps she’s responding to the way tele-immersive images make people seem present and absent at the same time. The people on the screen flicker in and out of virtual life very easily, and as they come into camera range they seem to step out from behind an invisible wall in an unnerving way.

Yet the experience of being in front of those 48 cameras is anything but deadening. Wymore says her tele-immersion work has made her deeply aware of how all of her is moving, not just what an audience can see. Given how attuned dancers are to their every motion, that’s saying something. And in the tele-immersion lab I noted that the students using it felt more, not less, alive. As they walked into camera range and started moving, looking at their three-dimensional, projected selves, they looked both amazed and happy. There was a tangible sense of awe. These students, most of them neither computer scientists nor dancers, had access to a whole new appreciation of their bodies through, weirdly enough, technology.

Share this article