- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CAPAgent, which aims to maintain reliable spatial decisions when external perception models are replaced. It trains on outputs from different models for the same scene, supervising changes in decision margins while using only one output at deployment. The paper also introduces CapSwap-Bench and evaluates transfer to unseen perception models on standard spatial benchmarks.
Critical strengths
-
Comparisons include matched single output training, domain generalization methods, and alternative pair losses. Ablations distinguish the benefits of matching scenes from those of the proposed objective.
-
The paper reports improved standard benchmark performance and fewer errors against ground truth, including under matched deferral budgets. These controls help establish that the gains are not simply due to more conservative behavior.
Critical weaknesses
-
Generalization depends strongly on informative quality cues and the range of reliability seen during training. Transfer is weaker when replacement models are more reliable, and the systematic depth scaling test reveals substantial failures when errors barely affect the cues. This limits the broader claim of reliability under unseen model replacements, although the paper clearly acknowledges this limitation.
-
Calibration quality remains a concern for decisions near task thresholds. The agent’s predicted intervals achieve 85.2% coverage on unseen providers, below the nominal 90% target. Reference actions also depend on a calibrated radius and a fixed decision rule. While evaluations against ground truth strengthen the results, their sensitivity to the calibration level and the task’s risk tolerance remains unclear.
Questions for authors
-
How sensitive are the gains and method rankings to the reference coverage level, for example 80%, 90%, and 95%?
-
Where is the remaining undercoverage concentrated, particularly among unseen providers and decisions close to task thresholds? Can the authors report coverage and commitment errors for these subsets?
-
Does including stronger providers during training improve transfer to reliability improvements without reducing performance on degraded providers?
3 comments
Thank you for addressing my questions. I appreciate the clarifications. After considering your response, I’ve decided to keep my original rating.
Thank you for reading our rebuttal and for your continued support of the paper. Your questions on calibration sensitivity, near-threshold coverage, and training with stronger providers helped make our analysis much more complete, and we really appreciate the time and care you put into the review.
Response to Reviewer nAxw
Thank you for the careful review, and for noting that the matched-deferral controls separate real gains from more conservative behavior. We added three analyses on the controlled core, following the paper's protocol.
Response 1 (W1, Q3): Does training with stronger providers improve transfer without hurting degraded ones?
Yes. Adding one stronger provider to training raises switch accuracy on a held-out stronger model by 10–12 points, and degraded providers stay within 0.3 points or improve. Scale errors that the cues do not show need an independent check, and a simple label-free one works.
The submitted policy trains with one depth treatment, ZoeDepth-NK, which is noisier than the reference (2.62× its radius), so it never sees a swap toward a more reliable model. In two training runs with the paper's steps and batch size, we added UniDepthV2 or MoGe-2 to training and tested on the other. Numbers are switch accuracy (%).
So the weak transfer to better models reflects what the training pool covered, and one extra provider closes most of the gap.
The scaled-depth test (Table 23) is a different case: a uniform 6% scale leaves the cues almost unchanged. It can be caught without labels. At swap time we run the old and new models on 20 unlabeled frames and take the ratio ρ of their median depths (1.057 for the scaled model, 0.96–1.04 for the six real replacements). Widening each radius by |ρ − 1|·p, where p is the measured value, raises accuracy on the scaled model's switch outputs from 15.2% to 62.9%, close to recalibration with 20 labeled units (64.3%). On the real swaps it changes switch accuracy by at most 1.1 points.
Response 2 (W2, Q1): How sensitive are the gains and rankings to the reference coverage level and risk tolerance?
The ranking and CAPAgent's lead of about 16 points are stable from 80% to 95%. The lead also holds at every task tolerance, in switch accuracy and in false proceeds against ground truth. A higher level only trades fewer false proceeds for more re-measurements.
We rebuilt the references at 80% and 95% and retrained the three main policies at each level (six runs of 3.8 hours). FP and RM are CAPAgent's depth false proceeds against ground truth and its re-measure rate (%).
On unseen providers, CAPAgent reaches 71.1 / 72.9 / 72.0 at the three levels, against 51.6 / 52.8 / 53.5 for single-output training.
Across the three task limits δ = 5% / 15% / 30% (Appendix D.5), switch accuracy rises from 64.5 / 69.5 / 73.6 to 82.1 / 85.3 / 88.3, and depth false proceeds against ground truth fall from 4.7 / 2.8 / 1.5 to 2.6 / 1.4 / 0.8.
Response 3 (W2, Q2): Where is the remaining undercoverage concentrated?
The undercoverage comes mainly from one unseen provider, Metric3D v2 Small. Across distance to the threshold, coverage is nearly flat. Commitment errors do concentrate near the threshold, and CAPAgent halves unsupported commitments there (11.6% → 5.7%).
Coverage of the true value by each policy's own range (target 90) and unsupported commitments, by unseen provider (%):
Metric3D v2 Small accounts for about two thirds of the shortfall. It is the only replacement noisier than anything in training, so the agent's radius stops growing near the widest value it has seen (0.124 m against a calibrated 0.157 m; Appendix G.10). A wider training pool addresses this, as in Response 1.
By distance to the nearest threshold (near / middle / far third), CAPAgent's coverage is 84.3 / 85.3 / 86.0%. Unsupported commitments are 11.6 / 4.4 / 1.2% for single-output training and 5.7 / 2.0 / 0.5% for CAPAgent. If an application needs more caution, scaling the predicted half-width by 1.15 at inference raises unseen coverage to 89.4% and cuts depth false proceeds from 1.6% to 1.2%, with 2.9 points more re-measurements.
The three analyses are summarized in the new Section 5.7 of the revised PDF, marked in blue (Table R1b–d).