< Previous | Contents | Manuals Home | Boris FX | Next >
Rolling shutter is an imaging problem produced in many popular "CMOS" cameras. This includes company-specific proprietary variations such as 3MOS
HyperMOS, etc, as well as other cameras such as RED that have other sensor names. It occurs just as much in expensive cameras as cheap ones. Only cameras that specifically state "global shutter" in their specifications, or "CCD" cameras (which are rarely seen these days) do not suffer from this problem.
The rolling shutter problem arises because there is no consistent shutter period for the entire image. The top lines of the image are "taken" over a physically different stretch of time than the bottom lines. The lines at the top are taken earlier, the lines at the bottom much later.
Consider a 30 fps 1080p camera. The top line is read out then reset; it begins accumulating light and charge for the next 1/30th of a second. As soon as the first line has been read out and reset, the second line is read out and reset, and it then begins accumulating light for 1/30th second. That continues for every line in the chip.
Generally, for a 30 fps camera, it will take about 1/30th of a second to read out, one after another, all 1080 lines.
That means that by the time the bottom, 1080th, line is read out, almost a full 1/30th of a second has gone by. It will be accumulating light for a period of 1/30th of a second with virtually no overlap with the 1/30th of a second that the top line is integrated over! The top line of the next frame will be read out and begin integrating in just an instant. So the last line is closer to the next frame than the first!
This wreaks havoc on the geometry of the image. Depending on how the camera is moving during the 1/30th of a second, and what it is looking at, an alarming number of geometric distortions will be introduced.
To the extent that the camera is panning left or right, the image will be skewed horizontally: a vertical pole will become slanted. To the extent that the camera is tilting vertically, images will be squashed or stretched vertically. If the camera is pushing in or pulling back, keystone distortions will result.
Consider a shot from a camera panning to follow a truck. The background suffers rolling shutter distortion, but the truck (stationary in the image) does not! Each moving object is affected differently, depending on its motion across the image.
If the camera is vibrating, the image turns completely to jello. (We rather famously lost an expensive helicopter shoot to this effect.)
In short, rolling shutter, and CMOS cameras, are pretty much a disaster for visual effects. If at all possible, we recommend using cameras with a global shutter for visual effects shots, despite their higher cost.
Unfortunately, there's no way to eliminate this problem. You can reduce it, work around it, but not eliminate it. Contrary to the claim of some, a short shutter time does not reduce rolling shutter. If you think about the explanation above, you'll see why. A short shutter time that reduces blur will make it harder to hide mistakes, as well.
Improperly shot footage with CMOS cameras will be objectionable even to lay human observers, because it does not correspond to our perception of the world— except in cartoons!
For professional shooters, the usual tactic for CMOS is to make sure that the camera motion is slow in all directions, so that there is comparatively little motion from frame to frame (wasn't it supposed to be "moving pictures"?). And CMOS cameras should always be shot from hard mounts such as dollies, cranes, etc.
Important : Always use a global shutter camera for aggressive hand-held shooting such as combat POV footage. See the paragraph and video about jello shots above if you doubt that.
Moving a CMOS camera slowly does not eliminate the rolling-shutter problem. It may reduce the geometric distortion sufficiently that you cannot see it. However, in match-moving we typically match shots down to the sub-pixel level, so something you can't see may still be a ten pixel error! That's bad.
There are several possible approaches to deal with that:
1. try to shoot to keep the rolling-shutter low, and try to cheat the tracks and inserts to allow for the inevitable errors,
2. use a third-party software tool to try to correct the footage, or
3. compensate for the rolling shutter in the solve, producing a solve for an 'ideal' camera.
The first choice can work out for small amounts of distortion and modest tracking requirements. The solver will adjust the solve to best match the distorted data, which winds up allowing inserts to appear at a correspondingly distorted location. Long shots that loop back on themselves and other shots with high self-consistency requirements will be substantial problems.
The second choice, a third-party tool, can be OK for simple images too. But, keep in mind that rolling-shutter causes an unrecoverable loss of data in the image. Any repair can be only a best guess at the missing data, and for that reason you will commonly see artifacts around edges, and residual distortion.
The third choice, producing a solve for an ideal camera, is the approach we make available in SynthEyes, using the rolling-shutter controls on the Shot Setup panel, and the Advanced Lens Controls.
When you turn on rolling shutter compensation, the solver will correct the tracker data (not the images) based on the motion of each tracker, so that the tracker's position is the position it had at the vertical center line of the image.
To do that, you must turn on rolling shutter compensation and supply a single number, which is the portion of the frame time required to read out the image data. An online tutorial shows how to measure rolling shutter.
Alternatively, the solver can compute rolling shutter from suitable shots , when enabled from the Advanced Lens Controls panel. To be suitable, a shot must have sufficiently rapid motion, and well-enough distributed trackers, for the rolling shutter to be computed.
Important: If you enable a measured rolling shutter value on the Shot Setup panel, it overrides any solver value, and prevents the solver from changing it. When the shot setup panel is specifying the value, the rolling shutter spinners on the lens distortion control panels will be grayed out.
The rolling shutter value is the ratio of the readout time to the frame time, for example, a 30fps camera with a 27 msec readout has a rolling shutter fraction of 0.81 (=27/33.33). Camera manufacturers are moving to reduce the readout time to reduce the rolling shutter problem, so a 5 msec readout at 30 fps would be a rolling shutter factor of 0.15 (quite modest). Some measured values can be found on the web site.
Gory detail : on footage shot on mirror stereo rigs, the footage from the camera that is mirrored vertically will have rolling shutter in the reverse direction, so the rolling shutter value should be set to the negative of the usual value.
Eliminating the rolling shutter distortion in this fashion can provide very substantial improvements in the quality of the solve.
In order to composite CGI generated for the ideal camera with the original rolling- shutter-based footage, you must render the footage with a rolling shutter also. At present, that capability is not widely available, for example in Houdini, Lightwave and Renderman, but we expect it to be more widely available in the future. It is essentially a modified form of motion blur.
At present, you can likely simulate the effect by rendering at a multiple of the frame rate, then combining the subframes in varying amounts depending on the vertical position in the image.
Rolling shutter compensation also makes it more difficult to assess the quality of tracking within SynthEyes, as the predicted 3-D position of a solved tracker is based on the ideal camera, with no rolling shutter. So there will be apparent errors where there are none.
©2024 Boris FX, Inc. — UNOFFICIAL — Converted from original PDF.