Camera Tracking and Matchmoving?
Say what?

Although SynthEyes is being used on major Hollywood features, we expect it to interest many local and regional edit/effects studios and independent feature producers that might be new to the technology, but wish to "move up" to offer more sophisticated effects to their clients or audience — who increasingly expect Hollywood excitement even from local productions.  

Here, we've tried to look at what camera tracking is about, how you can do it, and how to use it effectively with clients. For technical details such as import and export formats, see the feature list.

What is Camera Tracking?

"Camera tracking" or "Matchmoving" or "3-D Tracking" is the process of analyzing a video clip or film shot to determine where in 3-D the camera went, what its field of view was, and where parts of the set were. The 3-D path of a large moving object can be determined as well.

Why bother? What can I do with it?

The camera tracking information lets you add 3-D animated effects into live-action footage, such as

  • Animated 3-D character insertion (mascots, beasts, flying pixies, you name it). A favorite for commercials.
  • Vehicle insertion. Aircraft, boats, cars, trucks, spacecraft, things you wish you could get live, but couldn't afford or don't exist.
  • Architectural Previsualization (ground level or helicopter)
  • Virtual set extensions (boy, that's a fancy newsroom!)
  • Virtual product placement (did George Washington really drink XXX?)
  • Crash reconstructions (from a boom, or vehicle P.O.V.)
  • Matte paintings (ah, so that's what sunset on Mars looks like)
  • Transparent effects. A broad catch-all for effects you really can't see, often fix-its: insert snow, puddles, volumetric smoke, a big tree to cover a phone pole, a vase to replace a microphone left in-camera.

Can't I do some of these already?

You can do some 2-D effects using a compositor, if the camera is locked off, on a tripod, or if the amount of perspective shift is small.

Camera tracking is used with the much more kinetic moving-camera shots that today's audiences crave. Even on simple shots you may want to use camera tracking, as it can offer better speed and accuracy.

Describe the workflow for using SynthEyes.

Shoot. In SynthEyes, track features using the automated system or supervised trackers, solve for the camera and environment information, export to your existing 3-D animation or compositing system (Lightwave, 3ds max, Maya, Hash A:M, Inferno...). Using your 3-D system, create effects to insert. Render. Composite. Edit.

How do I shoot?

Most camera tracking shots are shot from a Steadi-Cam, dolly (or wheel-chair cam), boom or crane, camera truck, or aircraft. SynthEyes works fine with hand-held shots, though you'll have to decide if your camera stability is adequate for your production. 

We have a do-it-yourself project write-up showing how to build a tripod-mounted camera dolly & track with parts from Home Depot. It gives surprisingly good results from a simple construction.

If you are planning to shoot hand-held DV to track, we recommend that you check out some simple inexpensive do-it-yourself camera stabilizers, search the web and also these:

If you're doing a helicopter shoot and don't have a fancy copter-mounted camera stabilizer, here's our hard-earned advice. Make sure the day is cloudless, or don't fly. Keep the camera shutter short (sports mode). If at all possible, use a stabilizer. There's a nifty-looking Kenyon gyro stabilizer that looks good for copter work, it can be rented by the week.

What shots will it work on? 

To determine 3-D information, a shot must exhibit perspective changes as it evolves. Some features must be closer to the camera than others, and the relationship must change over time. If you could paint your shot onto a big cyclorama and re-shoot it from a camera on a tripod, there is no perspective. SynthEyes offers a special mode for tripod shots, but the result is 2.5-D, not 3-D.

There must be enough features to track (no empty white rooms or green backdrops). The features must be in the field of view long enough for their 3-D position to be determined. They can't be all bunched up in one portion of the image. SynthEyes 2008 has a special feature for selectively tracking only features located on the screen.

What's this about features?

A feature is a small spot that can be tracked from frame to frame, including scuff marks on floors, clumps of weeds, light switches, the corner of a box, etc. 

What shots won't work?

One of the common novice shooting problems is to use a hand-held videocamera and wave it around while primarily standing still.. This results in a non-3-D shot, where the camera is effectively on a tripod. It does not produce true perspective shifts. Furthermore, as the camera whips around, trackers fly on- and off-screen rapidly. True 3-D can not be determined. See path-ology.

Any "rules of thumb?"

Make the camera focus on a single area as it moves to approach or circle this focus of attention.

Make sure the shot is busy enough that there are features to track. Make sure that they extend over most of the frame (especially when tracking an object).

The client wants to shoot an actor against blue-screen with some markers, and insert him/her into a 3-D environment. Anything to look out for?

Beware of the client zooming in for the head shot, and taking most/all the trackers out of the frame. You need to keep 10-12 in view for a decent track (mathematically there can be many fewer, but that doesn't make it a good idea.)

The pure blue screen shots are dangerous just because the background is so featureless that if you mess up and have trackers out of frame, you can wind up in big trouble. And there's a tendency to put some marks on a back wall way behind the actor, focus on the actor, and wind up with a very flat shot with little/no perspective on the tracking marks. When this is the case a 2-D track is the right/only thing (using SynthEyes's tripod mode). Also the tracking marks are out of focus, sometimes to the point of invisibility.

For a higher probability of success, you want the shot to encompass the presenter from a distance, so that tracking marks can be seen not only on the back wall, but on the floor and possibly side walls, depending on camera motion. Having trackers visible at a variety of depths makes tracking much more sane. You don't have to use every tracking mark, so feel free to have plenty. Odds are that a moving-camera shot will move the field of view around so that only a fraction of them are visible at a time, unless it has been knowledgably planned. If there's a desk or other props in the scene, that can be helpful, as long as it has some trackable features. 

You can have things hanging from the ceiling (but not waving around) and mounted on light stands.

The cleverest tracking mark is a small square or circle the same color as the screen, but darker. You can still key it out, but SynthEyes can still find it. Be sure to test your setup in advance of the main shoot to make sure the marks aren't too small or too close to the background to be visible if they are out of focus or washed out a bit.

What's this about tracking objects?

If there's a large object moving around in the camera field of view, you can track the object in 3-D, even if the camera was locked off. In your animation package, you'll see the object moving relative to the camera. With the right scene, the camera can be moving too, carrying the object along.

Can I track an actor's head as an object?

Yes! It takes a bit of care and experience and some advance preparation as well. Use a tight shot of the head, with a shorter focal length lens --- by shooting up close there will be more perspective to get 3-D. There have to be plenty of trackable points on the head at all times. They have to be on parts of the head that don't move relative one another---trackers on the jaw, mouth, eyebrow are bad with the actor talking. Usually the tracking features are hand tracked off tiny details on the face and head. Sometimes drawing some dots on the actor helps too. Use high-resolution source such as an HDTV camera or film ---- VHS camcorder source is probably a non-starter. Progressive-scan DV is a good idea.

You can track all the moving features of the actor's face using SynthEyes's motion-capture capabilities, but then you will need two cameras. See the manual.

If SynthEyes can track and solve automatically, why does it have supervised  capabilities also?

It's like cruise control on the car: handles the boring stuff, but switch to manual for heavy traffic. SynthEyes lets you mix and match, even in the same shot.

What do I do in my 3-D animation package?

You'll open the scene as exported by SynthEyes, to find a camera moving along the computed path, with the correct field of view, plus a collection of tracker points, which are nulls or points or axes, depending on the package. Your shot will appear as the camera's background.

You'll add computer-generated objects into this scene, using the tracker points and original shot as guides. You'll add matching lights (or use ones located and imported from SynthEyes), and generally adjust your materials and texture maps to match the scene.

Often, you'll add stand-in geometry (planes, boxes, cylinders) to act as shadow catchers to help sell the integration of the effects with the scene. Sometimes you may use camera mapping to achieve more complex matching of perspective.

Can I really do this?

Yes! Like using any sophisticated program, such as an animation package, compositor, or Photoshop, more experience helps.

Start with the tutorials in the free demo, then move on to some of your own footage. Learn to recognize shots that can be tracked easily. Then you're ready to try some projects with clients.

If possible, offer on-set supervision as a service: a little cost up front may save the client much more later (and you head-aches). You can run SynthEyes on a laptop on-set, so if you're shooting DV, you may be able to track shots immediately before the shoot is wrapped. (SynthEyes reads DV in AVI or QT movies).

How can I find out more about camera tracking?

We have some tutorials and links on our learning page. Also, Tim Dobbert has written a book on matchmoving (see link at right) that provides a lot of information about the field. It doesn't describe SynthEyes's operation in particular, but looks at the techniques involved without focusing on the details of operating any particular program.

 Matchmoving: The Invisible Art of Camera Tracking

Will SynthEyes make my project look like a $150M budget?

Not by itself, that takes skill and practice too. But SynthEyes can help you take you there, and stay there. 

Did this help?

Let us know!