acceptodds
Under review as a conference paper at ICLR 2027

CAPAgent: Keeping Spatial Agents Reliable When Their Perception Models Change

Abstract

Spatial embodied agents rely on external perception models for depth, pose, and grounding to make task decisions, and these models are often replaced after the agent is trained. Similar readings from different models can differ in how far they can be trusted, so the right action can change after a replacement. This paper asks how one trained agent can make correct decisions with unseen perception models under fixed task rules and interfaces, without updating its weights. The agent judges reliability from quality cues, such as how widely depth values spread around a doorway, but a cue rises both at real scene edges and when the model is noisier. We show that an agent trained on one output at a time, from many scenes but few models, learns mostly the scene-driven meaning of these cues and underreacts when a new model raises them. We first build CapSwap-Bench, a benchmark that replays actual model replacements on matched scenes, with switch pairs where the right action changes and retain pairs where it should not. We then present CAPAgent, which trains on outputs of two perception models for the same scene and query, so that their difference isolates the model change. The core idea is to learn how its decision margins should move within such pairs, crossing a task threshold only when the right action changes; at deployment, CAPAgent still reads one output at a time. When three tool models are swapped for unseen ones, CAPAgent-8B reaches an average score of 70.6 on five standard spatial benchmarks, retains 96% of its original-tool score, and matches the 27B SpatialCLI agent. On CapSwap-Bench, it acts correctly on 84.7% of switch pairs and 92.3% of retain pairs. Anonymous code repository: https://anonymous.4open.science/r/CAPAgent-8E4C/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
spatial agents, cross-model generalization, tool learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_nAxwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CAPAgent, which aims to maintain reliable spatial decisions when external perception models are replaced. It trains on outputs from different models for the same scene, supervising changes in decision margins while using only one output at deployment. The paper also introduces CapSwap-Bench and evaluates transfer to unseen perception models on standard spatial benchmarks.

Critical strengths
  1. Comparisons include matched single output training, domain generalization methods, and alternative pair losses. Ablations distinguish the benefits of matching scenes from those of the proposed objective.

  2. The paper reports improved standard benchmark performance and fewer errors against ground truth, including under matched deferral budgets. These controls help establish that the gains are not simply due to more conservative behavior.

Critical weaknesses
  1. Generalization depends strongly on informative quality cues and the range of reliability seen during training. Transfer is weaker when replacement models are more reliable, and the systematic depth scaling test reveals substantial failures when errors barely affect the cues. This limits the broader claim of reliability under unseen model replacements, although the paper clearly acknowledges this limitation.

  2. Calibration quality remains a concern for decisions near task thresholds. The agent’s predicted intervals achieve 85.2% coverage on unseen providers, below the nominal 90% target. Reference actions also depend on a calibrated radius and a fixed decision rule. While evaluations against ground truth strengthen the results, their sensitivity to the calibration level and the task’s risk tolerance remains unclear.

Questions for authors
  1. How sensitive are the gains and method rankings to the reference coverage level, for example 80%, 90%, and 95%?

  2. Where is the remaining undercoverage concentrated, particularly among unseen providers and decisions close to task thresholds? Can the authors report coverage and commitment errors for these subsets?

  3. Does including stronger providers during training improve transfer to reliability improvements without reducing performance on degraded providers?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nAxw

Thank you for the careful review, and for noting that the matched-deferral controls separate real gains from more conservative behavior. We added three analyses on the controlled core, following the paper's protocol.

Response 1 (W1, Q3): Does training with stronger providers improve transfer without hurting degraded ones?

Yes. Adding one stronger provider to training raises switch accuracy on a held-out stronger model by 10–12 points, and degraded providers stay within 0.3 points or improve. Scale errors that the cues do not show need an independent check, and a simple label-free one works.

The submitted policy trains with one depth treatment, ZoeDepth-NK, which is noisier than the reference (2.62× its radius), so it never sees a swap toward a more reliable model. In two training runs with the paper's steps and batch size, we added UniDepthV2 or MoGe-2 to training and tested on the other. Numbers are switch accuracy (%).

Depth treatments in trainingUniDepthV2MoGe-2DA-V2 Large (1.15×)Metric3D v2 Large (1.52×)Metric3D v2 Small (3.41×)Retain
ZoeDepth-NK (paper)58.561.562.266.978.892.3
+ MoGe-270.2in training67.168.178.592.6
+ UniDepthV2in training71.865.967.579.092.4

So the weak transfer to better models reflects what the training pool covered, and one extra provider closes most of the gap.

The scaled-depth test (Table 23) is a different case: a uniform 6% scale leaves the cues almost unchanged. It can be caught without labels. At swap time we run the old and new models on 20 unlabeled frames and take the ratio ρ of their median depths (1.057 for the scaled model, 0.96–1.04 for the six real replacements). Widening each radius by |ρ − 1|·p, where p is the measured value, raises accuracy on the scaled model's switch outputs from 15.2% to 62.9%, close to recalibration with 20 labeled units (64.3%). On the real swaps it changes switch accuracy by at most 1.1 points.


Response 2 (W2, Q1): How sensitive are the gains and rankings to the reference coverage level and risk tolerance?

The ranking and CAPAgent's lead of about 16 points are stable from 80% to 95%. The lead also holds at every task tolerance, in switch accuracy and in false proceeds against ground truth. A higher level only trades fewer false proceeds for more re-measurements.

We rebuilt the references at 80% and 95% and retrained the three main policies at each level (six runs of 3.8 hours). FP and RM are CAPAgent's depth false proceeds against ground truth and its re-measure rate (%).

Reference levelSwitch, Single-outputSwitch, GARSwitch, CAPAgentFPRM
80%67.572.983.62.418.1
90% (paper)68.473.984.71.623.1
95%68.974.084.31.229.6

On unseen providers, CAPAgent reaches 71.1 / 72.9 / 72.0 at the three levels, against 51.6 / 52.8 / 53.5 for single-output training.

Across the three task limits δ = 5% / 15% / 30% (Appendix D.5), switch accuracy rises from 64.5 / 69.5 / 73.6 to 82.1 / 85.3 / 88.3, and depth false proceeds against ground truth fall from 4.7 / 2.8 / 1.5 to 2.6 / 1.4 / 0.8.


Response 3 (W2, Q2): Where is the remaining undercoverage concentrated?

The undercoverage comes mainly from one unseen provider, Metric3D v2 Small. Across distance to the threshold, coverage is nearly flat. Commitment errors do concentrate near the threshold, and CAPAgent halves unsupported commitments there (11.6% → 5.7%).

Coverage of the true value by each policy's own range (target 90) and unsupported commitments, by unseen provider (%):

Unseen provider (radius ratio)Coverage, Single-outputCoverage, CAPAgentUnsupported, Single-outputUnsupported, CAPAgent
DA-V2 Large (1.15)84.390.23.01.5
MASt3R (1.31)71.286.15.82.5
Metric3D v2 Large (1.52)75.987.65.02.2
Metric3D v2 Small (3.41)58.276.99.14.7

Metric3D v2 Small accounts for about two thirds of the shortfall. It is the only replacement noisier than anything in training, so the agent's radius stops growing near the widest value it has seen (0.124 m against a calibrated 0.157 m; Appendix G.10). A wider training pool addresses this, as in Response 1.

By distance to the nearest threshold (near / middle / far third), CAPAgent's coverage is 84.3 / 85.3 / 86.0%. Unsupported commitments are 11.6 / 4.4 / 1.2% for single-output training and 5.7 / 2.0 / 0.5% for CAPAgent. If an application needs more caution, scaling the predicted half-width by 1.15 at inference raises unseen coverage to 89.4% and cuts depth false proceeds from 1.6% to 1.2%, with 2.9 points more re-measurements.

The three analyses are summarized in the new Section 5.7 of the revised PDF, marked in blue (Table R1b–d).

Thank you for addressing my questions. I appreciate the clarifications. After considering your response, I’ve decided to keep my original rating.

Thank you for reading our rebuttal and for your continued support of the paper. Your questions on calibration sensitivity, near-threshold coverage, and training with stronger providers helped make our analysis much more complete, and we really appreciate the time and care you put into the review.

Reviewer_JdxjOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the reliability of spatial embodied agents when their perception tools are replaced by unseen models. The authors introduce CAPSwap-Bench to evaluate decision consistency under real perception model swaps and propose CAPAgent, which learns decision-margin changes through same-scene paired training. The method is supported by theoretical analysis and extensive experiments, demonstrating improved robustness across spatial reasoning benchmarks without retraining for unseen perception models.

Critical strengths

The problem of maintaining agent reliability under perception model replacement is practically meaningful. The manuscript is well-structured, clearly written, and provides detailed implementation settings, supporting reproducibility and practical deployment.

CAPSwap-Bench addresses the limited evaluation of real perception model replacements in existing benchmarks. CAPAgent introduces same-scene paired learning with decision-margin supervision, supported by theoretical analysis and comprehensive experiments.

Critical weaknesses

The framework relies on predefined task-specific decision thresholds and rules, with some benchmark thresholds constructed using ground-truth measurements. Although effective for structured spatial decisions, its applicability to more complex spatial reasoning and embodied decision-making tasks without explicit thresholds remains unclear.

Questions for authors

(1) When CAPAgent encounters a new task type or environment beyond the evaluated settings, does it require redefining decision rules or retraining the model to maintain performance?

(2) How could the framework be extended to tasks without explicit decision thresholds, such as robotic manipulation, navigation, or object search that require real-world exploration and multiple tool interactions?

(3) Does the proposed framework have the potential for continual self-improvement when a perception model is replaced or a new task scenario is encountered? For example, could the spatial agent interact with the environment during an initial adaptation phase and leverage trial-and-error feedback to progressively refine its reliability estimates, decision thresholds, and decision rules? Such feedback-driven adaptation could help the agent better accommodate unseen perception models and new task scenarios, improving its decision-making performance over time without requiring full retraining.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jdxj

Thank you for the positive assessment of the problem, CapSwap-Bench, and the paired training. Your questions ask how far the framework reaches beyond the evaluated setting. We answer each below, with evidence from the submission and two new results.

Response 1 (W1, Q1): Does a new task type or environment require new decision rules or retraining?

No retraining is needed for a new task type: the decision rule is written as a one-line check at call time. The same agent also carries over to new environments (DA-2K). Ground truth sets thresholds only when we build the benchmark.

Only CapSwap-Bench sets thresholds from ground truth (θ = c*(1 ± δ), Appendix D.5), to control how close each decision is to its threshold. The agent never sees c*. In the agent loop, the threshold comes from the question: 0 for relations such as "closer" or "left of", or the value the question states.

The rule is an input and is not stored in the weights. A check names a measurement p, a threshold θ, and a scale s (Appendix E.4), so a new decision means writing one such line. Two pieces of evidence:

  • None of the five benchmarks in Table 1 contributes a question or an image to training, and none is framed as a threshold task. The agent writes the checks on its own.
  • The check type depth_at_least has no training example in check format (Appendix C.1). CAPAgent-8B still uses it in 7.4% of its checks on SpatialCLI-Bench with swapped tools and answers those 74 questions with 85.1% accuracy (86.2% on the whole benchmark).

For a new environment: all margin supervision for the Table 1 agents comes from indoor scans, and CAPAgent-8B still reaches 89.4 on DA-2K (outdoor, transparent, and non-real scenes) after the swap, against 86.6 for the single-output agent.


Response 2 (Q2): Extending to tasks without explicit thresholds: manipulation, navigation, object search

Navigation, object search, and multi-tool interaction are partly covered in the paper already. For tasks with no threshold, a planner can use the predicted range directly, for example in a costmap, a grasp check, or the choice of the next view.

Already in the paper:

  • Navigation: passage clearance and outdoor free distance are navigation decisions. The replay track runs them over three steps with a moving camera, and final-step accuracy rises from 79.8% to 88.6% (Appendix G.8).
  • Object search: the grounding track decides whether to act on a detected object, declare it absent, or look again (Case 2 in Appendix L).
  • Multiple tool calls: in SpatialCLI's loop the agent averages 3.0 calls per question, and 64% of the questions CAPAgent-8B gains on SpatialCLI-Bench involve a re-query.

For tasks without a threshold: the head first predicts a plausible range around a measurement, and the threshold enters only when the range has to become an action. A planner can use the range directly: inflate an obstacle by the predicted radius in a costmap, compare the range of an object's width with the gripper opening, or choose the next view where the range should shrink most. Outputs that do not reduce to a single number, such as masks or plans, are the extension we name in Section 6.


Response 3 (Q3): Can the agent keep improving from trial-and-error feedback after a swap?

Yes. After a swap, the agent can refine its reliability estimate from a few labeled scenes or from its own re-measurements, without full retraining. We would keep thresholds and rules fixed, because they define the task.

With labeled feedback, we fine-tuned both agents on N labeled scenes of each new provider. Unseen switch accuracy at N = 0 / 5 / 20 / 50 is 52.8 / 58.1 / 67.4 / 76.9 for the single-output agent and 72.9 / 75.0 / 79.3 / 84.6 for CAPAgent, so fine-tuning after a swap and paired training are complementary.

Without labels, the agent's own re-measurements serve as feedback. After a RE-MEASURE, the provider runs again on the next keyframe, and the new reading either falls inside the range the agent predicted earlier or misses it. These misses update one number online, a multiplier on the predicted half-width, with the rule of adaptive conformal inference (Gibbs & Candès, 2021), where the new reading takes the place of a label. No weights change, and ground truth is used only to score the result. On the 120 replay episodes of Metric3D v2 Small, the unseen model with the lowest coverage, the multiplier settles at 1.21: coverage of the true clearance rises from 77.4% to 86.3%, and unsupported commitments fall from 4.6% to 2.8%. For the other three depth models in the replay track, it stays between 1.00 and 1.06.

Thresholds and rules specify the task (how much clearance the robot needs), and a loop that is free to move them could quietly relax a safety requirement.

The fine-tuning and re-measurement results are in the new Section 5.7 of the revised PDF, marked in blue.

Thank you for the detailed and thoughtful response. I appreciate the authors' efforts to clarify the role of decision thresholds and task-specific rules, as well as the additional experiments on few-shot adaptation and online reliability calibration. These new results provide valuable evidence for the robustness and adaptability of CAPAgent under unseen perception model replacements and further strengthen the paper's contributions.

The rebuttal has addressed several of my concerns. Nevertheless, the framework still relies on predefined task-specific decision criteria, where decisions must be formulated in terms of measurable quantities, explicit thresholds, and fixed rules. This may limit its applicability to more complex embodied tasks where the relevant measurements or decision thresholds are difficult to specify in advance, such as those involving multiple physical properties, uncertain task requirements, or open-ended interactions. Furthermore, its ability to adapt to entirely new task scenarios without predefined decision rules remains to be explored.

I acknowledge the practical significance of the problem and the comprehensive experimental validation. I think this work makes a meaningful contribution to improving the reliability of spatial embodied agents under perception model replacement.

Thank you for taking the time to follow up and for your positive assessment of the new results. Extending the framework to more open-ended embodied interactions is a natural next step that we plan to explore in future work.

Reviewer_Y2YeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether spatial agents can continue to make correct decisions when their perception models are replaced, i.e., whether embodied agents can perform stably under different harnesses. To this end, the paper constructs CapSwap-Bench, in which models from different sources produce outputs for the same query in the same scene, and LightGBM is then used to calibrate an error radius for each model. The paper further proposes CAPAgent, built on Qwen3-VL-8B and jointly trained under supervision derived from the LightGBM outputs. At deployment, the agent does not need to know which model is behind a tool: from a single anonymous output and its quality cues alone, it can judge whether a reading is trustworthy. This improves the agent's performance on unseen tools.

Critical strengths
  1. The paper studies whether spatial agents can perform stably under different harnesses and formulates this setting as a measurable problem. This distinguishes it from prior work, and the problem has strong practical relevance in real-world applications.
  2. The paper proposes same-scene, cross-source paired training. Within the same scene, changes in quality cues can only come from the model itself, so paired training enables the agent to distinguish whether a cue change is caused by the scene or by the model. The paper also explains this theoretically through Proposition 1. With this training, CAPAgent's accuracy on unseen replacement models improves substantially.
  3. The experimental design is solid. The baselines cover domain generalization and several paired learning methods, and the paper includes fairly detailed ablations. Overall, the experiments are reasonably thorough.
Critical weaknesses
  1. Both the training targets and the main evaluation metric depend on the quality of LightGBM itself. LightGBM only guarantees an average coverage of about 90%, coverage is even lower on unseen switch cases, and conditional coverage under different scene conditions is not guaranteed. It is questionable whether LightGBM alone can fully characterize the gap between the models' predictions and the real scene across different scenes, which raises particular concerns about out-of-domain generalization.
  2. The providers' pretraining data may overlap with the evaluation data. The paper only ensures that the agent has not seen the evaluation scenes and does not state whether each provider's training set includes them. If such overlap exists, both training and evaluation take place on data familiar to the providers. The relationship between cues and errors learned by the agent may then not hold at deployment, when providers face unfamiliar environments, which could affect the agent's actual performance.
  3. The paper argues for CAPAgent's practical value on the grounds that retraining requires preparing new data for every replacement. However, the only baseline is full retraining from scratch, and it actually outperforms CAPAgent. In practice, a more common approach is to lightly fine-tune an existing agent on a small amount of data from the new tools, which might substantially improve performance on new models.
Questions for authors
  1. Could the authors report, separately for different scene features, the coverage of the reference radii and CAPAgent's accuracy scored against GT? Are the strata with lower coverage exactly where CAPAgent's errors are concentrated?
  2. Could the authors clarify whether each provider's training data includes the scenes actually used to evaluate the agent? If so, do the current gains still hold after grouping providers into in-domain and out-of-domain? Could the authors further report how much gain the proposed method can guarantee on scenes that neither the providers nor the agent have seen?
  3. If Single-output-8B or SpatialCLI-8B were lightly fine-tuned on a small amount of data from the new tools, could it match CAPAgent's performance? Would the resulting model generalize better to scenes that neither the providers nor the agent have seen?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Y2Ye

Thank you for the thorough review, and for recognizing the problem's practical relevance and the experimental design. We added one analysis for each of your three concerns.

Response 1 (W1, Q1): Do the conclusions depend on LightGBM? Coverage and GT error by scene feature

Our two main results are scored without LightGBM. Among the scene features we checked, only glass and mirrors show clearly lower coverage and higher errors, and CAPAgent still cuts GT error there from 10.5% to 6.4%.

Table 1 uses each benchmark's own answers (CAPAgent-8B leads Single-output-8B by 2.7 points [1.8, 3.6]), and Table 17 uses mesh and LiDAR ground truth (false proceeds fall from 3.0% to 1.6%). Held-out reference coverage is 88.9–91.1% for all 15 providers (Table 11). It is lower on unseen switch cases (85.9%) because switch pairs are the cases with larger errors.

By scene feature, on the passage outputs of the three unseen depth replacements (n = 1,530). GT error is a false proceed or an unnecessary stop.

StratumReference coverage (%)GT error, Single-output (%)GT error, CAPAgent (%)
Edge energy: low third90.65.53.1
Edge energy: high third88.87.44.3
Glass or mirror: no90.25.93.3
Glass or mirror: yes86.510.56.4

CAPAgent's errors are partly concentrated where coverage is lower: glass or mirror regions are 11% of the outputs and 20% of its errors.

Refitting the references with a three-layer quantile MLP, without retraining any policy, keeps 95.9% of the reference actions, and switch accuracy against them is 67.6 / 73.2 / 83.5 for single-output / GAR / CAPAgent (68.4 / 73.9 / 84.7 with LightGBM).


Response 2 (W2, Q2): Do provider training sets include the evaluation scenes, and do the results hold where they do not?

Some providers were trained on these datasets, and the results do not depend on it. The gain is +20.2 switch points on the 250 pairs where the replacement was trained on the scene dataset and +19.7 on the 157 where it was not, and the agent's ranges stay as reliable. On scenes new to both providers and agent, CAPAgent leads (grounding 58.6% → 70.9%, DA-2K 86.6 → 89.4).

The table marks whether each provider's reported training data include the dataset. ZoeDepth-NK, the depth model the agent trains with, was trained on neither, so the agent already learns the cue–error relationship on scenes unfamiliar to its provider. Unseen switch accuracy (%):

ReplacementScenes (trained on?)PairsSingle-outputCAPAgentGain
Metric3D v2 SmallScanNet (yes)13555.979.3+23.4
Metric3D v2 SmallScanNet++ (no)8454.878.0+23.2
Metric3D v2 LargeScanNet (yes)5151.067.6+16.6
Metric3D v2 LargeScanNet++ (no)3248.465.6+17.2
MASt3Rboth (yes)6450.867.2+16.4
DA-V2 Largeboth (no)4147.662.2+14.6

Metric3D v2 tests your concern directly: the same model is on both sides. On ScanNet++, which it has not seen, its median error is 11% larger than on ScanNet. The agent's ranges are 9% wider there and cover the true value on 81.8% of outputs, against 82.6% on ScanNet (single-output: 66.2% and 67.6%).

Scenes new to both: the ScanNet++ rows (UniDepth was not trained on it either), grounding (no detector saw ScanNet++, Table 22), and DA-2K (Flickr images, Table 1). Evaluation scenes are held out from agent training (Appendix D.2).

A formal guarantee needs labels from the new provider. Section 4.2 states the condition under which the measured gain transfers, and Table 23 tests it.


Response 3 (W3, Q3): Can light fine-tuning on the new tools match CAPAgent, and does it generalize better?

A fine-tuned agent needs 20 to 50 labeled scenes per new provider to match what CAPAgent does with none. It does not generalize better: it stays below CAPAgent on unseen scenes (DA-2K), and its gain stays with the tuned provider.

We fine-tune on N labeled scenes (capture units) per new provider (none in the evaluation core). Unseen switch accuracy (%):

Labeled scenes per provider (N)052050
Single-output + fine-tuning52.858.167.476.9
CAPAgent + fine-tuning on pairs72.975.079.384.6

In SpatialCLI's loop, SpatialCLI-8B fine-tuned on 10% of the regenerated trajectories averages 69.4 (one run), between no adaptation (66.5) and CAPAgent-8B without new data (70.6). On DA-2K, which neither the providers nor the agents have seen, it scores 87.7 against 89.4 for CAPAgent-8B.

After tuning on Metric3D v2 Small (N = 50), the single-output agent rises only from 47.6% to 49.2% on DA-V2 Large (CAPAgent untuned: 62.2%). So every new swap needs new labels, which do not exist on the day a model is replaced.

The revised PDF adds Section 5.7 (one page, in blue) with the main new results.

Reviewer_Y2YeOpenReview ↗

Reply to Response

Thank you for the detailed response and the additional experiments. The stratified coverage and error analysis, the split by whether each provider was trained on the evaluation scenes, and the fine-tuning comparison address my concerns.

Thank you for your follow-up comment. We are glad that the additional experiments addressed your concerns, and your questions on reference calibration, provider training data, and light fine-tuning helped strengthen the paper. We appreciate your time and careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.