Moving Object from AprilTag Corner Trackers

Shows a box with 3 AprilTags being tracked using Corner Tracker mode, combined as a single moving‐object track. Bonus: a coordinate system setup along the box's diagonal, resulting in the box becoming the hilt of a sword.

Script for Search Engines:

Hi, this is Russ Andersson. In this tutorial we're going to show how you can use the Corner Trackers option to combine data from multiple AprilTags. As a quick reminder, AprilTags are an open source fiducial scheme from the University of Michigan; there are multiple families of tags and each of the many tags in a family can be automatically found and identified in an image. Here, I've taped 3 AprilTags onto a box that I'm holding. It's not an exact thing; I probably should have taped them flatter. Note that the outer white boundary is necessary so that the the outer black square can be detected reliably. The black boundary cannot be stretched to the edge of box; it needs the white boundary in order to be located. Let's get started. Since we are going to be tracking a moving object, we are going to start by going up to the solver panel and disabling the camera. We don't want it to be moving around. Now, we can add a moving object that's going to hold the trackers we're going to create. But let's actually go to the tracker panel to get a better view, and we'll fire up the AprilTag detector. Now we do know that the trackers are 25H9 tags (we could identify them if we needed to), and we are going to switch Corner Trackers mode, which is the subject of this tutorial. With that, we're ready to start tracking ... and now you see why this is the Corner Tracker option. We've got these trackers, each has a name based on the number of the tag itself and a corner number, so that those trackers are always assigned repeatably based on the tag number and which corner it is. If I scrub through the shot you can see them track, and then somewhere down here, you'll see that actually it loses track briefly, where two of those go off the edge, and you'll see that all of the trackers are lost at that time, because we're tracking the entire tag first, and then generating the trackers based on that. Now, the trackers that we've generated are just standard supervised trackers, so if we wanted to we could go and continue to track those ones that are still visible into the areas where the tag as a whole is not. I can select one of the trackers there and I am going to hit the 5 key to bring us into pan-to-follow mode. And now between this view and the one up here you can take a look at how the tracking goes, and it does a reasonably good job here. Now if you look at some of these trackers that are further out you'll see that when the tag is being looked at kinda more edge on, you'll see that the corners can drift away a bit more. Some of that may be some motion blur or defocus also. So, you know, the tracking is reasonable but does have some limitations. Now let's go and solve this, which is done quickly enough, and now we have an object-mode track there and you can see that moving around and you'll notice that the error is fairly high at around 3 pixels. Now you can lower that error by using some of the SynthEyes corner auto-trackers; that does take some more work to monitor them and combine them together from different sections of the shot. So, AprilTags gives you a quick way to do this. But in either case, no matter how you do it, it turns out that rolling shutter is pretty significant in this shot, and it is adding on about another pixel of error in this shot. So we do have something we could use here. Now we could be boring and just put a coordinate system on one of the faces of the cube, but let's have some more fun. We'll start out by adding an additional tracker. We're going to add that down here, and let's just dial its size down quite a lot. We're going to make it a white spot tracker. Just jump onto that spot there. And we also want to increase the size of the search area. So with that done, we'll make one other adjustment, just to take advantage of the tracking we've already done to help to predict where the thing is going to go. We're using the Use the Other Tracker option. Now you can see we can track into the shot pretty nicely without any great thought. And eventually the cube does turn enough that it loses track, so we're just going to back up at that point, shut the tracker down, lock it up, and call it a day. So we have that additional tracker available. Now we're ready to do our coordinate system setup. And to do that, we're going to take these three trackers that are on the far back corners and make them be on the XY plane. So there's a plane right behind the box and we'll take that tracker that is in the front, and here I'll just quick do a solve here again. Here's the tracker in the front, and it is going to be on the Z axis. So now, that sets the position of the origin as well, where that Z axis drops down onto that XY plane. I also want that tracker to have a positive Z coordinate, so that the coordinate system will be coming up out of the image towards us. I have one other degree of freedom to nail down, really two, but I'm going to link these two trackers together and set up a distance between them of ten units. So that's setting up the scale of the coordinate system. But I want to do double duty with this, in that I want that to be roughly parallel to the Y axis. Now the height of each of these trackers over that XY plane should be roughly the same, but not necessarily exactly. So to eliminate any possibility of a conflict there, and get as good a coordinate system setup as possible, I'm going to say I want to have it on the same XZ plane which has the effect of making those two have the same Y axis (no), same Y value. So that constrains all the degrees of freedom of the coordinate system and now we're in a position of being able to go and run it again and we have our coordinate system setup with our object. And now we can do something with it. Let's go to the 3D panel. We're going to go to creation mode and Object coordinate systems, using that object, our moving object. And we'll create a cone that's going to sit right on our corner tracker and we're going to make the diameter of the cone roughly match up with those three other trackers that are the corners of the box over there. And having done that, we're going to start increasing the size of our cone, and we're going to do one other thing to it, which is to just move it upwards to match those three trackers here at the base of that corner. And now if we take a look at what we've got, our box has become the hilt of a very cheap low-budget sword. That's what we've done with this; a little more interesting than just a box. To wrap this up, the AprilTag corner mode can be used for camera or object tracking, whenever you want to combine the results from multiple AprilTags, but you don't have so many tags that you can use them in Center mode, where there's only one tracker per tag. Thanks for watching.