acceptodds
Under review as a conference paper at ICLR 2027

Agentic Relative Camera Pose Estimation via Learned Ranking and Verification

Abstract

A wide range of approaches have been developed for camera pose estimation, including correspondence-based methods, end-to-end pose regression, and recent 3D geometric foundation models. Our key observation is that no single estimator is optimal for diverse challenges, such as wide baselines, lack of texture, appearance changes, and occlusions. Further analysis reveals substantial performance variation across both benchmarks and individual image pairs, with different estimators exhibiting complementary strengths. We introduce **PoseAgent**, an agentic framework for relative camera pose estimation that dynamically orchestrates pose estimators through learnable ranking and verification. Given an image pair, a profiling agent first extracts appearance, semantic, and geometric features relevant to pose estimation, e.g., scene type. A learned ranking agent then predicts the relative competence of multiple pose estimators given the image-pair profile. The top-ranked estimator is executed, and its predicted pose is assessed by a learned verification agent that estimates the corresponding pose error. When verification fails, PoseAgent adaptively invokes lower-ranked estimators until a candidate is accepted or the execution budget is reached. For pose verification, our verification network predicts pose errors more accurately than prior models. For pose estimation, PoseAgent improves AUC@5 up to 4.2% over the strongest standalone estimator on each of ARKitScenes, MegaDepth, ScanNet++, and RealEstate10K. On ARKitScenes, PoseAgent also outperforms VLM-based agents, which include a VLM ranker with the same verifier and fallback policy. These results demonstrate the effectiveness of our learned ranking and verification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Camera Pose Estimation, Agent, Model Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cG4mOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PoseAgent selects among nine relative pose estimators using learned ranking and verification. An image-pair profiler provides features for an XGBoost ranker, while a pose-conditioned verification network predicts candidate errors and controls fallback. The framework improves over the strongest standalone estimator on four datasets, with additional experiments supporting the complementary roles of ranking and verification.

Critical strengths

The analysis shows useful complementarity among pose estimators, providing a clear motivation for pair-specific selection.

The component comparisons are reasonably thorough. Ranker-only, verifier-only, and fixed-order baselines help establish that learned ranking and verification both contribute to the final performance.

Critical weaknesses

The accuracy–cost trade-off remains unclear. AUC@5 gains over the strongest estimator range from 0.9 to 4.2 percentage points, while profiling alone takes roughly 0.6–0.7 seconds. Table 7 does not report total latency including estimator execution, or the average number of estimators invoked. Comparisons at matched runtime would better establish the practical value.

Generalization of the full pipeline is insufficiently tested. The ranker and verifier are trained on all four evaluation datasets, using scene-disjoint splits. The additional standard-benchmark experiments test the verifier with fixed execution orders, rather than the complete learned pipeline. They therefore provide limited evidence that the ranking policy transfers to a new dataset.

Questions for authors

What are the end-to-end latency and average number of estimator calls, and how does PoseAgent compare with a simple two-estimator cascade at matched runtime?

How does the full pipeline perform when both the ranker and verifier are trained without any data from the target dataset?

I think the topic of this paper is generally interesting. I'm also curious if the author has considered integrating the agentic loop and the novel view synthesis-related iterative refinement [1][2][3]? Could the author conduct some such discussions?

[1] iNeRF: Inverting Neural Radiance Fields for Pose Estimation. IROS 2021

[2] Neural Refinement for Absolute Pose Regression with Feature Synthesis. CVPR 2024

[3] GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting. ICLR 2025

Reviewer_zNbXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PoseAgent addresses relative camera pose estimation by selecting and verifying heterogeneous estimators for each image pair. A profiling stage extracts appearance, semantic, and geometric features, which an XGBRanker uses to order nine candidate estimators. A Pose Verification Network (PVN), built on frozen Reloc3r features with trainable pose-conditioned modules, predicts rotation and translation-direction errors. These predictions determine whether to accept a pose or execute another estimator, with a default budget of four executions.

The ranker and PVN are trained on pooled data from ARKitScenes, MegaDepth, ScanNet++, and RealEstate10K. On held-out test scenes, the complete framework improves AUC@5° over the strongest standalone candidate by 0.9–4.2 percentage points. Component studies compare ranker-only, verifier-only, fixed-order, and VLM-based alternatives. Additional experiments on ScanNet1500 and MegaDepth1500 evaluate verification-guided fallback under fixed estimator orders, providing evidence for the verifier beyond the main evaluation splits, although they do not test the learned ranker.

Critical strengths
  1. The analysis of estimator complementarity and the oracle gap provides a clear motivation for selection at the image-pair level.
  2. Gains over the strongest standalone candidate are consistent across four datasets. Table 2 provides useful comparisons with ranker-only, verifier-only, and fixed-order variants.
Critical weaknesses
  1. The candidate pool limits the comparison. Four candidates are SuperGlue variants and two are RoMa variants, while MASt3R and RoMa v2 are absent. Complementarity among the selected models is useful, but the benefit relative to stronger standalone alternatives remains unclear.
  2. Transfer of the full system is not demonstrated. Table 4 uses fixed estimator orders on ScanNet1500 and MegaDepth1500, omitting the learned ranker. Since the ranker and PVN are trained on pooled data from the four main datasets, these additional experiments establish less about full-system transfer than the paper's broader claims suggest.
  3. The accuracy–cost trade-off is incompletely reported. Table 7 lists profiling, ranking, and verification time without separately accounting for candidate-estimator execution. Profiling alone takes approximately 0.6–0.7 seconds per pair, so end-to-end latency matters when judging the practical value of fallback.
Questions for authors
  1. Candidate selection and stronger alternatives. What criteria determined the nine-estimator pool, and were availability or implementation constraints responsible for excluding MASt3R and RoMa v2? A comparison with these standalone methods, and ideally a candidate pool containing them, would clarify whether orchestration retains an advantage as individual estimators improve.
  2. Transfer of the complete framework. Can you evaluate the frozen ranker, PVN, and fallback policy together on ScanNet1500 and MegaDepth1500, without benchmark-specific retuning? Please compare learned ranking with the fixed orders in Table 4 and clarify scene overlap with ranker/PVN training and Reloc3r pretraining, so that the scope of generalization is clear.
  3. End-to-end computation. Does Table 7's “Verifier” time represent one call or the accumulated calls per pair? Please include candidate-estimator execution in mean and P95 latency, specify cached or reused features, and report the mean number of executed estimators and the fraction of pairs reaching the four-execution budget.
  4. Design choices and fallback failures. How much do the ranking objective and added PVN modules contribute over best-estimator classification and a simpler pose-conditioned verification head using the same features and supervision? For pairs that trigger fallback, please also report how often the final estimate improves or worsens relative to Rank-1, with representative failure conditions.
Reviewer_hXC6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PoseAgent, a closed-loop agentic framework for relative camera pose estimation from image pairs. It treats a set of off-the-shelf pose estimators as candidate tools. A profiling agent extracts image features to rank these candidates prior to execution. The top-1 estimator is then executed, and a Pose Verification Network (PVN) predicts its rotation and translation errors. If verification fails (predicted error exceeds thresholds), a fallback policy invokes the next estimators according to the predicted ranking. This process is iterated util satisfy certain criteria, or exceeds a budget.

Experiments are conducted with a fixed set of pose estimators and several in-domain datasets. The results show improvements over the best standalone estimator and over VLM-based agent baselines.

Critical strengths
  1. Despite some concerns regarding rigor/overclaims, the paper is overall well written: the motivation is clear, the pipeline design is sound, the results are promising, and the figures are well designed.

  2. An agentic framework for camera pose estimation is a worthwhile but currently underexplored research direction. This work presents a straightforward yet effective pipeline and provides our community a useful baseline. Despite the limited validation in this paper, I really like the research problem setting.

Critical weaknesses
  1. Lack of rigor
  • The analysis in Section 3.1 needs more rigor. Reloc3r achieves the best performance on ScanNet++, and RoMa achieves the best performance on MegaDepth. However, ScanNet++ is part of Reloc3r’s training data, and RoMa is trained on MegaDepth. Although the authors claim to use a non-overlapping test split, in-domain pose estimation can naturally be more accurate than out-of-domain pose estimation.
  • The term “agentic framework” is somewhat overclaimed in this paper. There is no planning, tool discovery, re-planning, language reasoning, etc. The proposed loop is essentially a handcrafted fixed cascade: rank, execute, verify, and move to the next ranked model.
  1. Other claims not well supported
  • "even classical estimators outperform recent models in many cases" is not well supported by the candidate set. The candidate set contains no classical SIFT/SURF/ORB pipeline. By "classical", I assume the authors mean handcrafted methods, but all candidate estimators are learning-based. In addition, line-based and point-line hybrid pose estimation methods are only discussed in Section 2 but are not included in the experiments.
  • The claim that PoseAgent treats an "arbitrary set" of estimators as callable tools is not supported. The ranker and PVN are trained on the outputs/errors of the same nine estimators used at test time. Adding or removing an estimator would require retraining. Dealing with "arbitrary image pair" is also kind of an overclaim, since there is no evaluation on driving scenes (like KITTI) or aerial images (like ACID). Therefore, the actual scope is limited and the evaluation is far from comprehensive. Although it includes nine candidate estimators, these actually correspond to five methods trained on different datasets.
  1. Other experimental issues
  • The profiling features and candidate pose estimators are handcrafted. There is also a mismatch: for example, ORB and LoFTR are used in profiling features but not as pose estimators.
  • The ablation study is incomplete. The necessity of several network design choices (e.g., the camera/register tokens, the PRoPE-conditioned self-attention) is not sufficiently ablated.
Questions for authors

Please respond to my major concerns in the weaknesses section. Additional questions below:

  1. A key advantage of an agentic framework is supposed to be generalization, but PoseAgent appears to break this property. Does PoseAgent generalize to out-of-domain datasets and new pose estimators? What is the impact of each profiling feature?

  2. While the pose estimators run in real time, PoseAgent introduces additional computation and latency. Oracle@9 produces the best results, which means all pose estimators are executed. One could instead execute all pose estimators and directly use a median pose, or run a RANSAC-like consensus mechanism over all estimated poses, which may produce better results.

  3. PoseAgent is noticeably worse than its upper bound (Oracle@2 and Oracle@9), which suggests that PoseAgent does not work that well.

Comments on the paper

2 comments
Area_Chair_hQJAOpenReview ↗

reminder

Dear reviewers,

This is a gentle reminder to read the authors’ rebuttals and respond in the discussion.

Thank you for your time and effort throughout the review process, and for your contributions to the community.

Best Regards,

AC

AuthorsOpenReview ↗

Summary of Rebuttal to AC

Dear Area Chair,

We appreciate the reviewers like the research problems setting, find the paper generally interesting and well written, motivation clear, and results promising. We also thank the reviewers for their constructive feedback and summarize our rebuttal and revision below.

1. Out-of-domain Generalization and Full-pipeline Evaluation (hXC6, zNbX, cG4m). We further tested on out-of-domain datasets including Cambridge and ACID. We evaluated the full pipeline, including ranker, PVN, and fallback policy, on ScanNet1500, MegaDepth1500, Cambridge Landmarks, and ACID. PoseAgent outperforms the strongest standalone estimator on each benchmark at all three AUC thresholds (Table R3).

2. Stronger Candidate Models (hXC6, zNbX). We added suggested MASt3R and RoMa v2, replacing the two original RoMa variants. With this updated pool, PoseAgent still outperforms the strongest standalone estimator and the median-pose baseline on all four main datasets at all three AUC thresholds (Table R2). The expanded per-pair analysis across eight test sets further shows that estimator complementarity persists with these stronger candidates (Table R1). It also shows that our framework can accomodate additional estimators that provide compatible pose outputs and verifier inputs.

3. Accuracy and Latency (hXC6, zNbX, cG4m). We report controlled end-to-end measurements covering profiling, ranking, estimator execution, and verification. PoseAgent takes 878–1236 ms per pair on average, invoking 1.71–3.10 estimators, and reduces mean latency by 26–46% relative to executing all nine estimators and selecting with PVN (Table R4). Reloc3r–MASt3R cascades using the same PVN and fallback policy are faster but less accurate on all four datasets, showing the accuracy gain obtained at additional computational cost (Table R7).

4.Effectiveness of Ranking and Fallback (zNbX). Learning to rank achieves higher Top-1 AUC@5° than best-estimator classification on three datasets, with comparable performance on MegaDepth (Table R5). Among the 2,931 pairs whose final estimator differs from Rank-1, the final pose improves in 63.5% and worsens in 36.5% (Table R6). We also discuss representative failures involving inaccurate translation-direction error predictions.

5. Framework Scope (hXC6). We clarify that PoseAgent uses “agentic” to describe its closed perceive–act–verify tool-use loop, without claiming an LLM-based agent. The original VLM comparisons support the learned ranker as an effective selection policy for this task. We also clarify that the ranker is candidate-specific, whereas PVN takes no estimator identity as input and was reused without retraining when MASt3R and RoMa v2 were introduced. We also clarify the term "classical estimators" and the scope of the evaluated estimator families.

Finally, following cG4m’s suggestion, we discuss incorporating iterative pose refinement as an additional tool when verification fails. We appreciate the Area Chair’s and reviewers’ consideration of these additional results and clarifications.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.