HiP3R: Hierarchical Pose Graphs for Long-Horizon 3D Reconstruction

Feed-forward models accurately estimate camera motion and 3D geometry from short video windows. However, computational and memory costs prevent direct inference on sequences containing thousands of frames. Long videos must therefore be processed as overlapping chunks. This division produces independent local reconstructions with inconsistent coordinate frames and duplicate estimates of shared cameras. Achieving global consistency requires propagating constraints from adjacent overlaps and distant revisits across the sequence. This requirement naturally motivates pose graph optimization. However, the choice of graph level creates a trade-end. Chunk-level graphs efficiently resolve scale drift and close large loops. Yet, their rigid transformations cannot correct internal errors or conflicting frame predictions. Conversely, frame-level optimization offers local flexibility, but scaling it globally is computationally expensive and can break metric scale. HiP3R resolves this trade-off without retraining the underlying models, using a hierarchical pose graph. HiP3R first optimizes a robust chunk-level $\mathrm{Sim}(3)$ graph to establish the global layout and scale. It then holds this layout fixed and estimates one shared $\mathrm{SE}(3)$ camera per frame, turning overlapping window predictions into a single trajectory. As a post-processing method that does not retrain the backend, HiP3R applies to both chunk-based and streaming models. Across seven benchmarks, HiP3R reduces VGGT-Omega average trajectory error by $74\%$, from $18.555$ to $4.906$ m, and relative tracking drift by $94\%$, from $0.9878$ to $0.0622$ m. It also decreases 3D surface reconstruction error by $64\%$, from $0.0759$ to $0.0272$ m. When paired with LingBot-MAP, HiP3R reduces trajectory error by $52\%$, from $8.288$ to $4.017$ m, and achieves the lowest overall error. It executes faster than existing alignment methods on long sequences.