Relaxation-Free Robust Vanishing Point Estimation in Manhattan World

In man-made environments, the parallel edges of a scene form three mutually orthogonal families, known as the Manhattan structure, and their projections meet at three vanishing points that fix the rotation between the camera and the scene. Recovering that rotation from a single calibrated image is difficult, because a line detector returns far more segments than the scene structure actually contains. The two halves of the task are moreover entangled, since a segment can be assigned to a direction only once the directions are known, while estimating them in turn presupposes that assignment. A single criterion, the truncated multi-selection error over the rotation group, settles assignment and estimation together. Our main contribution is MMVP, an iterative algorithm that minimizes this criterion directly, in the original problem dimension and without any relaxation (in the sense of losing tightness). Built on the majorization–minimization principle, the method assembles its surrogate in three stages, freezing the association at the current iterate, majorizing the truncation by an outlier process guided by a homotopy, and dominating the quadratic form over the rotation group by a linear function whose minimizer is available in closed form at a cost linear in the number of lines. We prove that, at every fixed level of the homotopy, the objective values it produces are non-increasing and bounded below, hence convergent, and that the iterates admit limit points, every one of them stationary. A deterministic voting-based initialization completes the pipeline, and no stage of the procedure draws a random sample. Experiments on synthetic scenes and on two real datasets, against geometric and learning-based estimators alike, show that MMVP outperforms them on association F${}_1$ and recall, and that no competing method reaches a higher angular accuracy at any threshold.