Good Question!
Here are some more subtle good questions that people have asked, at Siggraph and on the forum.
Kinds of Shots
Architects
Motion Capture
Technical Issues
Kinds of Shots
Q: How were the animals made to talk in Racing Stripes using SynthEyes?
A: Those are "moving-object" shots. They require only a single live-action camera. A collection of features on the animal's head are identified and tracked which move rigidly together as a unit. SynthEyes is easily configured to determine the object motion, instead of a camera motion, and the solved result is a "null" and a path that it moves and rotates along. The null represents the motion of the animal's skull. After exporting from SynthEyes, a new head scan mesh is parented to the null; it then matches the motion of the live-action head exactly. The head mesh can then be animated normally to make the animal talk and show expressions.
Q: So that means I can do facial capture from one camera, right?
A: Sorry, no. Read more in the motion-capture section below.
Q: We know SynthEyes does moving-object shots. What if the camera is moving too?
A: No problem. By tracking both background and object features, the camera motion and object motion will be determined. After you export/import to your animation program, you'll see the camera moving on its path, the object on its own path.
Q: Yeah, but actually the camera is on a tripod, so you can't get 3-D from it --- but we want 3-D for the object!
A: No problem. Set up the camera in tripod mode, and do the regular object tracking, and you'll get the 2-D pan/tilt camera motion and the 3-D object motion.
Q: How do I tell SynthEyes what is the background, what is the object?
A: Using animated roto-splines, an alpha channel from your external program, or by assigning supervised trackers to the correct object.
Q: How many separate moving objects can be in the shot?
A: As many as you want, in theory. But, if the objects are too small, you won't have enough resolution or parallax shift available to track them. A handful is more typical in real life.
Q: Why are there 2-D exports to 3-D compositing programs from a 3-D tracker?
A: There were a number of customer requests for these exporters, for the following reasons:
- After a 3-D solve, SynthEyes can output the re-projected image coordinates of the computed 3-D tracker location. Since this effectively takes many trackers into account, it can be very smooth.
- To generate tracks of features that go off the edge of the image or that are obscured, so that a four-corner-pinning setup can be used. It's a nice trick. You could often use a 3-D export instead, but using four-corner-pinning may be faster and more familiar to the compositor user.
- To use multiple SynthEyes tracks to generate an extremely stable overall track to drive shot stabilization setups in the compositing program.
- To use SynthEyes's excellent tracking capabilities instead of their compositor's.
Q: I'm a compositor, and even though I use 3-D compositing, I want the camera to stay still and everything else to move. Can I get that?
A: Yes. Set up the shot as a moving-object shot and the camera will stay put. You can do this at any time, see the camera-to-object conversion tutorial.
Architects
Q: My architectural previews often have telephone poles, trees, etc, in the foreground. Do I have to rotoscope them, or can I use SynthEyes's new object-mesh-building capabilities to model them and front project their original textures onto them?
A: Either way is doing things the hard way: it may be possible but can be quite hard especially on things like poles that are very narrow, or fuzzy like trees. For a pole, you could just put some trackers there and use the trackers as a reference for a cylinder object, say, but still it's a lot of trouble. Usually, it's much more expedient to composite new stuff over anything too inconvenient in the foreground --- get some trees from Archvision, put in some new sidewalks, etc. You don't want a pole blocking your project anyway. Drop a flower pot or mailbox on the stump. Use the mesh-building features for shadow-catching ground planes, and larger occluding objects that have internal and edge details that can be tracked.
Q: I'm an architect; I want to match a single still. Can SynthEyes do that?
A: Only if you have some measurements. There used to be some other applications designed for single-still matching, but they relied on specific geometry, such as parallel lines, that SynthEyes doesn't address. For the truly dedicated, there might be a way to do it in SynthEyes, but it's complex and I can't recommend it.
A much better approach, when possible: have the photographer get a bunch of shots of the building or site from different angles, even though you don't plan to use them. SynthEyes can match these collections; see the section of the manual on "Multi-shot tracking." And this way, you can be a hero and impress the client with a view from a different angle.
Q: I already know the exact coordinates of a lot of points. Can I use that?
A: Yes. SynthEyes can import a simple text file of tracker names and coordinates with that data, or you could even write an import script for your favorite format.
Motion Capture
Q: What's the difference between object tracking and motion capture?
A: In object-tracking, the points must all be rigidly connected to one another, and move around as a group; you need only one camera. In motion capture, the points can all move around independently (arms, legs, facial points), but two cameras are required.
Q: How many cameras can be used for motion tracking?
A: Any number of cameras can be used, there's no particular limit, but you need at least two.
Q: How many cameras are needed for full-body capture?
If you need full-body coverage as an actor goes twisting all around, you will need at least 3, probably 4. The catch is that as you have more cameras, you have some more calibration and tracking to do.
Q: What happens if markers disappear for a while?
A: There's no problem with markers disappearing from time to time. Since this is supervised tracking, you will just key the tracker's enable track off and then back on when it reappears, for each camera view.
When SynthEyes solves, it checks each camera on each frame for each tracker, and computes the best location as long as the tracker was seen by two or more cameras on that frame.
If it was seen by one or no cameras on a frame, no location will be generated, and ultimately there will be no key on that frame after it is exported to your downstream animation package. Consequently, (maybe after adjusting the interpolation method in your 3-D app), you should wind up with a usable spline-interpolated path on any few frames where the tracker was hidden.
Q: How accurate is it going to be?
A: There are a lot of factors based on the setup of the cameras, type of cameras, and action involved. As a rough idea, take the horizontal field of view at the subject and divide it by the horizontal resolution of the footage and multiply by two. For example, an actor is 1 meter away from the camera. Using a ruler, you measure the horizontal field of view of the image at that distance to be 750 millimeter (0.75 meter). You are shooting digital video with a horizontal resolution of 720 pixels, so with the cameras a decent amount away (0.5 meter, say), a rough accuracy estimate is 750mm / 720 pixels * 2 which is about 2 mm. If the images are good so that you have decent subpixel accuracy, you'll do better. If there is a lot of blur, or the cameras are closer together, you'll do worse. The 2mm gives you a rough starting point.
Q: Can any of the cameras move during motion capture?
A: Yes! But this is pretty tricky. You will still need to know the position of each camera in the world at each instant. Effectively, you will need to have a decent number of features in the environment that you have tracked from each camera --- moving camera(s) and still cameras --- such that the camera positions can be determined without reference to the moving motion-capture points. The moving cameras will reduce the accuracy somewhat since their position can't be calculated exactly, so keep as many locked off as possible.
Q: What types of projects are SynthEyes's motion-capture capabilities intended for?
A: If you want to make Polar Express 2, you'll want a big, fancy, expensive motion capture system, which offer specialized capabilities designed for high accuracy, high update rate, and a high volume of shots. But if you need a bunch of hero shots for an indie film, TV commercial, or demo reel, SynthEyes may be exactly the solution you need.
Q: How can I learn more about motion capture with the demo version?
A: Check out the face-tracking example in the manual. The files from the example are in facecapt.zip on the downloads page.
Technical Issues
Q: What is the world size used in solving?
A: The world size is for the benefit of the internal mathematics, which are more accurate if all the numbers stay somewhere near 1, even if they are raised to a few powers, ie 1.1^5 +1 vs 30000^5 +1. The first expression has an accurate result, the second does not. The world size is used to scale down the tracker and camera coordinates so they stay 1-ish and the math stays accurate. The world size doesn't directly affect the numbers at all, only indirectly impacts the accuracy.