www.ssontech.com

imagesh · Joined: 05 Apr 2007 Posts: 17

Hi all... first post... been really enjoying getting up to speed with Syntheyes and still so much to learn even before the new version was released!

Client tasked me with coming up with a method of combining multiple live action takes of a single talent, to be cloned into dozens. Wanted the camera to freely pan and tilt, no budget for motion control.

Been knocking around an idea to do just that, so did a quick proof of concept test and wanted to share. Don't judge the rotoscope or color balancing... these issues would be the same with motion control... actual shoot is on a set with controlled lighting and greenscreen...

The raw takes used...

http://ftp.datausa.com/imageshoppe/outgoing/PAN_TILT/SOURCE_TAKES.mov

The end result...

http://ftp.datausa.com/imageshoppe/outgoing/PAN_TILT/TAKES_BLENDED.mov

Thanks to Syntheyes and some other steps, I think I've come up with something VERY useful and cool as a technique...

Regards,

Jim Arthurs

eoffermann · Posted: Thu Apr 05, 2007 11:14 pm Post subject:

Very nice - a bit dizzying! but good work.

imagesh · Joined: 05 Apr 2007 Posts: 17

Thanks, just a test, I was trying to break the process by doing something far more radical than the actual shoot would be... so a full 360 degree spin on the master shot did the trick. If that works, anything will. Even the motion blur on each element is correct.

A total of seven nodal tracks, each one nulled out to zero camera motion via front projection mapping on the cameras, while a master camara played back the main shot's tracking.

So much to learn in SynthEyes... glad to see a NAB presence... I'll certainly be there asking questions!

Jim Arthurs

CineGobs · Joined: 16 Jan 2007 Posts: 29 Location: Denmark

Looks amazing IMHO. Shocked

Bo

ShadowMaker SdR · Joined: 17 Jun 2005 Posts: 69 Location: The Netherlands

I have no idea how you did this - really none whatsoever - I really like it, but if you wouldn't mind lifting a little bit of the veil, that would be great.
I just don't get how you combine motions that are completely each other's opposites.

imagesh · Joined: 05 Apr 2007 Posts: 17

Thanks everyone... this is a VERY powerful technique, I have no doubt somewhere it's been done before, but since I've brute forced it out on my own, I have no problem sharing. It truly can take the place of MOCO pan/tilt in a great number of cases.

I'll put together an example movie of how the footage is treated and post back with it, as a picture (or movie) is worth a thousand words...

Regards,

Jim Arthurs

imagesh · Joined: 05 Apr 2007 Posts: 17

Hi all... I've put up an example movie showing the process of processing one take to match up with the master take.

http://ftp.datausa.com/imageshoppe/outgoing/PAN_TILT/LW_DEMO.mov

Basically, here are the steps;

- Un-distort the footage.
- Track every take for tripod in Syntheyes. Export data to Lightwave camera.
- Build 3D sphere in LW, use camera data from each take to drive front projection mapping from each take, projecting onto inside of sphere.
- Use tracking data from master take to drive additional camera, which "re-photographs" each projected take with correct motion of master take.

The 3D step could be eliminated by using a composting package with good 3D support... I just used what I know best to prototype it out...

Glad to answer any questions...

Jim Arthurs

gordonrobb · Joined: 17 Mar 2005 Posts: 281

This is just amazing. I am not sure that I understand it, but am definately going to try it. Superb work.

gordonrobb · Joined: 17 Mar 2005 Posts: 281

One question (so far), when you say, solve each shot, do you mean seperately, or as a multiple shot solve?

imagesh · Joined: 05 Apr 2007 Posts: 17

Thanks gordonrobb...

Yes, each take is a completely stand-alone tripod solved SynthEyes project.

I then export to a Lightwave scene file and copy the camera motion onto a unique camera in a master Lightwave scene, which then is used to front projection map the image sequence into a sphere. I can decide at that point when in time I want the take to start, simply by moving the camera's keyframes around in time, and changing the sequence start frame.

When I get back from NAB I hope to have the time to shoot a better demo of this techique, using a "real world" problem as the basis. Something like the "guy jumps from out of the way of the speed car" kind of shot.

Gorf · Joined: 10 Mar 2007 Posts: 111

Really cool effect. How did you avoid the instances of the actor "colliding" with each other?

You could use this technique to go the other way too, I suppose, and extract the main actor, a bit like Harry Potter's cape...

imagesh · Joined: 05 Apr 2007 Posts: 17

Hi Gorf... I'm not sure I understand what you mean by colliding... I simply had the guy walk in different patterns, trying not to repeat any exact path. Later, once the original pan/tilt was removed, I could slip and slide each take in time, which helped to avoid collisions.

Yes, you can use this to create a "clean" plate for erasing all or part of an actor. This was originally how I was going to use the technique, then I realized that you could combine many takes into one, not just a clean take...

Regards,

Jim Arthurs

Gorf · Joined: 10 Mar 2007 Posts: 111

Hi, Jim. By "collisions" I meant two instances of the actor intersecting, but it didn't occur to me that the paths they follow probably do not cross.

I also thought that you'd be restricted as to where each actor can "occur", but I realise now that as long as you're following the actor for his entire path, and you have enough "buffer" at the end of the clip, you can slide his path forwards along the clip. This, in itself, is enough to avoid intersections.

For example - there is one clip (other than the master clip) where the actor passes the white barn door. You can slip that clip so that he passes the door at any point the door appears on the master clip.

imagesh · Joined: 05 Apr 2007 Posts: 17

Yes, exactly! As you can see in the file I posted that shows all the raw takes, I panned and followed the actor for a good length of time and distance, sometimes the pan is to opposite direction of the master take... and that doesn't matter after the motion is removed. Obviously you have some motion blur issues to monitor, but since the camera is panning with the actor, he's relatively sharp in frame, and you could simply roto him tighter for each take, pulling out more of the background if the blur is objectionable.

Then, I can slip and slide the time start of each of those takes to best fit the timing I desire in the master...

Try that little trick with motion control. Smile

Jim Arthurs

Gorf · Joined: 10 Mar 2007 Posts: 111

It's a shame Robert Rodriguez didn't have a copy of SynthEyes handy when he made Once Upon A Time in Mexico. The army withdrew their support when they found out the General character was a baddie.

He could have used your technique to take a handful of more modern vehicles (instead of the fleet of mostly WWII surplus he had to use) and "replicate" them.