What about Stereo 360VR?

NOTE: The discussion below applies to stereo 360VR. The stereo VR180 format is completely different: it does have two separate nodal points. VR180 footage can be tracked by converting it (with STmaps in the image preprocessor) to two conventional images with 90-120deg field of view, and processing it with SynthEyes's normal (NON-360VR) 3D stereo features. It's even easier because you know the inter-ocular parameters you used for the conversion!

The (monocular) equirectangular 360VR format has wonderful mathematical properties that make it easy and mathematically correct to spin images around however you like, most typically for stabilization.

There's an assumption that "stereo 360VR" is just the same, and can be manipulated like the regular stereoscopic imagery we're familiar with in movies and 3D television at home.

Unfortunately, that's not the case. Stereoscopic 360VR is pretty much an oxymoron, an inconsistent combination of two different things. First, there's (monocular) 360VR, which relies on having a single well-defined perfect nodal point. Second, there's stereoscopy, which relies on having two different nodal points, ie one located at each eye.

The nature of the difficulty becomes clear when you consider the viewer turning their head to look in a different direction. The nodal points need to move as well, but that's a problem—we only have two images at most. If they are correct monocular 360VR images, then the nodal points can't be arbitrarily moved by the VR viewing software. The stereo effect would only occur looking straight ahead, which is hardly useful.

Instead, stereo 360VR uses two non-nodal images. Each column of the image represents a viewing plane, but all the viewing planes don't pass through the same nodal point: they are offset by half the inter-ocular distance, forming a cylinder (yes, picture needed). This produces a number of consequences, as follows.

For regular stereo, the visual effects industry has learned to take great pains controlling the stereoscopic viewing parameters to produce an acceptable viewing experience and minimize viewer discomfort. Unfortunately, the images produced in stereo 360VR are intrinsically non-physical, and the stereo 360VR format does not contain the information necessary, even in theory, to correct that. That doesn't bode well for the stereoscopic 360VR format in the long term. A combination of nodal monocular 360VR and rendered inserts might work out better.

What can I do? Obviously stereoscopic 360VR footage is being produced, and SynthEyes customers are being asked to work with it. To stabilize it, one of the image streams can be processed as if it was monocular footage, even though that isn't the case. The resulting stabilization can easily be applied identically to both image streams. When doing the tracking, emphasize using features farther from the camera over those that are near.

To minimize artifacting, stereo 360VR footage should be shot from platforms that are as stable as possible. To the extent that that doesn't happen, artifacts will occur. As with monocular 360VR footage, shooting with synchronized cameras with global shutters is greatly preferred!

SynthEyes easily is the best camera match mover and object tracker out there.

Matthew Merkovich

More Quotes