Although the slick visual effects in DreamWorks’ figure-skating flick Blades of Glory easily slid past audiences, the filmmakers for this box-office hit would have been skating on thin ice without the skill of Rainmaker Visual Effects. Rainmaker’s Visual Effects Supervisor Mark Breakspear led the 150-person team that filled 3D stadiums with digital crowds and gave the four main actors Olympic-class skating moves — or rather, convinced the audience these skaters, not their stunt doubles, could perform an Olympian’s triple salchows, double axels, spins, lifts, and spirals.

Directed by Will Speck and Josh Gordon, the film stars Will Ferrell and Jon Heder as an unlikely male figure-skating pair. Writes Kevin Crust of the Los Angeles Times, "Whatever combination of choreography, camera trickery and special effects were required to render the over-the-top, hyper-real skate numbers, they're executed with wit and ingenuity.”

That wit and ingenuity was due, in part, to a lucky break. “Jon Heder cracked an ankle bone during practice, so we had a three-month hiatus while he healed,” says Breakspear. “That gave us an opportunity to previs every shot in the skating sequences, and we came up with new moves.”

To make the moves work, Rainmaker used new, state of the art techniques to replace the four stunt skaters’ faces with the actors’ faces. The techniques rely on a blend of commercial software and proprietary tools. “People have done fantastic face replacement in CG on mega-movies, but we didn’t have four years or a blockbuster budget,” Breakspear says. At first, the studio pushed for a 2D solution, but Breakspear convinced the directors that switching faces in 2D would limit their ability to shoot the skaters and restrict the skaters’ movements.

Fortunately, Rainmaker had a gold-medal face-replacement expert on its team, CG supervisor Kody Sabourin, who had worked on the legendary “Super Punch” shot in The Matrix: Revolutions. In that shot, accomplished with proprietary technology, Agent Smith’s face deforms in an extreme close-up when Neo lands a punch.

“Given my experience on the Matrix films and from working at ESC with people like George Borshukov and Kim Libreri, I knew the best way to approach this project was with animated textures that we captured at the same time we captured the performance,” he says.

Re-inventing Face Replacement
At first, Rainmaker tried to hire a studio or company to do the work. “We couldn’t find anyone we could outsource from,” Sabourin says. “People could capture the performance, but not the textures, or if they captured textures, the textures were low-res and not usable. Or, they’d texture-map a still frame on an animated mesh. I knew, based on my experience on Reloaded and Revolutions, that wouldn’t work.” As a result, Sabourin led a team at Rainmaker that developed the process used for Blades of Glory.

They began by taking digital photographs and making plaster casts of the actors’ faces. Then, they had the casts scanned in high resolution at XYZ RGB in Ottawa. Using the photographs and a 3D model created from the scan data in Autodesk’s Maya, visual effects artists determined where to place 102 tracking markers on each face. Before the performance capture session, makeup artists copied each marker’s position onto each actor’s face.

To capture the actors’ facial performance and animated textures simultaneously, Rainmaker devised a clever set up using three HD cameras, one film camera, and mirrors. “Imagine that the actor is sitting in a chair and around him are four cameras about two meters away,” says Breakspear. “Above and behind him are mirrors.”

The team captured the tracking markers with the three HD cameras, and recorded textures during the actors’ performances with the film camera, which they positioned in the center. Breakspear put lights behind silks to capture the faces with completely flat lighting. “Using the film camera was something that has never been done before,” Sabourin says. “Not even on the Matrix films. We could scan the film and have textures at 4K resolution if we needed, whereas with HD, we were capped at HD resolution.”

The actors performed to footage of his or her stunt double. “We had split screens on set so the actors could watch the stunt doubles in the plate and then watch themselves reacting,” says Sabourin. “The director had free reign.”

The mirrors doubled the data collection and provided textures from multiple angles. “With mirrors, rather than using more cameras, we could split one image into three,” says Sabourin. “We got textures from the front and both sides of the face, and an extra three cameras tracking the performance.” To match the frames once the cameras rolled, a time code generator linked all the cameras.

The effects team tracked the markers using RealViz’s MatchMover Pro and triangulated the actors’ facial motion into 3D space. “We ended up with 102 dots that looked somewhat like the actors’ faces moving in 3D space,” Sabourin says. “That is, you could make out many of the moving shapes, like their ears, throat, lips, eyes, and nose.”

Next, they applied the animation from the dots to animatable meshes created from the high resolution XYZ RGB scans of the actors’ facial casts. Proprietary code helped them transpose as much detail from the actor’s performance as possible. “That gave us an animated face mesh,” Sabourin says.