I Spent the Night Tuning the Wrong Layer

We put a gimbal camera in a room and taught it where things were. Pan 163 was a bottle. Pan 82 was an air-conditioning unit. Useful, until someone picks the camera up — then every number is wrong by an unknown amount and the map is fiction.

Restoring that is a real problem with a real name. We didn't know the name for most of the night.

The tuning trap

The approach seemed obvious. Store a small picture of something distinctive — a junction box, a piece of wall art. After the camera moves, find that picture again, and the difference tells you how far it turned.

It worked when the camera barely moved. It failed the moment it moved properly. So we did what you do: we tuned.

We tried a different feature detector. We changed the matcher from ORB to SIFT because SIFT is scale-invariant and scale seemed to be the problem. We adjusted inlier thresholds. We widened the search. We measured noise floors and re-derived cutoffs. Each change was defensible and each was measured, which is exactly what made it feel like progress.

Then we ran a controlled comparison. Same room, same movement, same photographs — the only change being what we stored. Instead of a small crop of a landmark, we stored the whole view.

landmark stored as a crop, matched into a wide frame :  2 of 5
landmark stored as a full view, matched view-to-view : 10 of 10

With the original detector. The one we had spent hours replacing.

The bottleneck was never the matching. It was that we were asking the matcher to find a postage stamp inside a landscape — a task nothing is designed for. Every threshold we carefully calibrated was calibrated around a decision one level above it that we never questioned, because it wasn't presented as a decision. It was just how we'd built it in the first ten minutes.

That is the trap. Tuning feels like progress because each step is measurable. You can watch a number improve while remaining inside a bad frame. The question that would have saved the night — what shape is this problem in? — never gets asked, because the shape isn't a parameter and nothing in your dashboard is pointing at it.

The literature had this handled

Late on, we were asked a fair question: is this not common? Is there no research?

Yes. Extensively.

The geometry dates to Willebrord Snellius around 1615 — the three-point resection problem, how a surveyor finds their position from angles to three known landmarks. Its modern form is Perspective-n-Point, solved directly since 1841, and it ships in OpenCV as one function call. The contemporary pipeline is well established: retrieve similar views, match them with learned features, solve pose against a 3D model.

The literature also hands you a warning we would never have derived. If the observer and three landmarks all sit on a common circle, the problem has infinitely many solutions — the "danger circle." Surveyors are taught to avoid it. We had been cheerfully arranging landmarks in exactly that configuration.

We got to a working answer by hill-climbing. The field got there four centuries ago and left signposts. Half an hour of reading would likely have skipped a night of experiments — and the experiments that stung most were the ones confirming things already in a textbook.

Refusing to answer is a feature

One part we would keep unchanged.

At one point the system reported a correction of +86 degrees. The truth was about +111. It described that answer as "usable for coarse aiming."

Its confidence came from two measurements that agreed closely with each other. Both had been flagged unreliable for separate reasons. Both were taken at the edge of the sensor where the maths is weakest. Three of five landmarks were missing entirely.

A consistency metric computed over untrustworthy inputs is not evidence. Two bad numbers can agree perfectly.

So the system now refuses. It checks whether it recovered enough landmarks, whether each measurement was taken where the maths holds, whether the geometry is self-consistent — and when it isn't, it declines to produce a number and says which check failed. It will still show you the number it would have given, labelled clearly as not to be applied.

Refusing correctly is harder to build than answering, and much harder to demonstrate, because the output looks like nothing happened. But a confident wrong correction propagates silently into everything downstream. A refusal stops there.

Two things that surprised us

Measuring how far it moved, not just how far it turned. When a camera rotates, everything in view shifts equally. When it moves, nearby objects shift more than distant ones. That difference is a signal. Fit it and you separate the rotation from the translation — recovering how much the camera turned even though it also moved, plus which direction it went.

It needs one real-world measurement to become metric. We used a tape measure across the floor. One number, taken once, and every subsequent movement reads in feet. Two independent calibrations — different distances, different positions, different directions — agreed to within one percent. Distances the system then reported to walls matched hand measurements to about three inches.

Honest error bars beat precise-sounding ones. Asked to measure an unknown movement, it reported 5.9 feet and said the real answer was likely between 4.7 and 7.4. The truth was 5.0.

Off by eleven inches, and inside its own stated range. That is a better outcome than a confident 5.9 with no interval, because you can plan around a system that knows how wrong it might be. You cannot plan around one that is sometimes silently wrong by a foot.

What we'd tell ourselves at the start

Before optimising anything, ask what shape the problem is in. Tuning inside a bad representation produces measurable, satisfying, worthless progress.

Look up the prior art before hill-climbing. Not for the algorithm — for the failure modes. The danger circle is not something you discover by experiment; you discover it by reading, or by being mysteriously wrong forever.

And build the refusal. Any measurement system will eventually be asked a question it cannot answer. The useful ones say so.