What about Stereo 360VR?

NOTE: The discussion below applies to stereo 360VR. The stereo VR180 format is completely different: it does have two separate nodal points. VR180 footage can be tracked by converting it (with STmaps in the image preprocessor) to two conventional images with 90-120deg field of view, and processing it with SynthEyes's normal (NON-360VR) 3D stereo features. It's even easier because you know the inter-ocular parameters you used for the conversion!

The (monocular) equirectangular 360VR format has wonderful mathematical properties that make it easy and mathematically correct to spin images around however you like, most typically for stabilization.

There's an assumption that "stereo 360VR" is just the same, and can be manipulated like the regular stereoscopic imagery we're familiar with in movies and 3D television at home.

Unfortunately, that's not the case. Stereoscopic 360VR is pretty much an oxymoron, an inconsistent combination of two different things. First, there's (monocular) 360VR, which relies on having a single well-defined perfect nodal point. Second, there's stereoscopy, which relies on having two different nodal points, ie one located at each eye.

The nature of the difficulty becomes clear when you consider the viewer turning their head to look in a different direction. The nodal points need to move as well, but that's a problem—we only have two images at most. If they are correct monocular 360VR images, then the nodal points can't be arbitrarily moved by the VR viewing software. The stereo effect would only occur looking straight ahead, which is hardly useful.

Instead, stereo 360VR uses two non-nodal images. Each column of the image represents a viewing plane, but all the viewing planes don't pass through the same nodal point: they are offset by half the inter-ocular distance, forming a cylinder (yes, picture needed). This produces a number of consequences, as follows.

• The view in any direction can be determined and used to build up stereoscopic images as part of a 360 VR viewer.
• There is no physically localizable "image plane" like there is for a conventional camera image. The closest thing to an image plane is actually a partial section of the cylinder, edge on to the viewing direction. When the view is nearly vertical, the "image plane" is even more chaotic.
• Due to the cylindrical scheme, stereoscopic 360VR images cannot be rotated arbitrarily, except around the vertical axis. Rotating the images for stabilization unavoidably corrupts them: this is not a software limitation, but reflects the underlying limitations of the format. Artifacts will be most visible towards the poles.
• With conventional stereo, each eye, taken individually, is a normal monocular image. This is not true for stereo 360VR: individual eyes are not nodal monocular 360VR images. The closer a feature is to the camera, the more problematic it is. Of course, that's usually true due to the stitching in monocular 360VR anyway.
• The non-nodal nature of stereo 360VR footage means that a lot of the math used in normal perspective and 360VR simply isn't available.

For regular stereo, the visual effects industry has learned to take great pains controlling the stereoscopic viewing parameters to produce an acceptable viewing experience and minimize viewer discomfort. Unfortunately, the images produced in stereo 360VR are intrinsically non-physical, and the stereo 360VR format does not contain the information necessary, even in theory, to correct that. That doesn't bode well for the stereoscopic 360VR format in the long term. A combination of nodal monocular 360VR and rendered inserts might work out better.

What can I do? Obviously stereoscopic 360VR footage is being produced, and SynthEyes customers are being asked to work with it. To stabilize it, one of the image streams can be processed as if it was monocular footage, even though that isn't the case. The resulting stabilization can easily be applied identically to both image streams. When doing the tracking, emphasize using features farther from the camera over those that are near.

To minimize artifacting, stereo 360VR footage should be shot from platforms that are as stable as possible. To the extent that that doesn't happen, artifacts will occur. As with monocular 360VR footage, shooting with synchronized cameras with global shutters is greatly preferred!