acceptodds
Withdrawn from ICLR 2027

Ask What If Before Committing: Counterfactual Stability for Vision-Language Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language navigation in continuous environments often employs online topological planners to rank candidate goals on an evolving map built from partial observations. Yet a high policy score does not reveal whether a goal remains supported when the computation behind that score changes. We propose CTPNav, a framework that measures decision-time counterfactual stability by combining mean support with a penalty for response dispersion under controlled internal interventions. At the representation level, standardized edge responses gate the learned attention residual, while their dispersion adjusts the association radius for the next map update. At the action level, CTPNav probes goal competition, geometric bias, and instruction grounding to assess the stability of candidate goals. The resulting responses provide detached targets for policy learning and guide a bounded inference-time correction of action scores. On R2R-CE Val-Unseen, CTPNav achieves 61% SR and 51% SPL. On RxR-CE Val-Unseen, it achieves 54.53% SR, 45.98% SPL, and 63.19% nDTW. These results demonstrate the value of intervention-response stability for learning and topological planning. Our code is available at https://anonymous.4open.science/r/CTPNav-AD2F.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Navigation, Embodied AI, Counterfactual Reasoning, Topological Planning, Decision Reliability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dFedOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CTPNav, which evaluates the stability of navigation decisions by observing how candidate actions respond to controlled interventions on action competition, graph geometry, and instruction grounding. These responses are used to regulate graph reasoning, adjust map construction, provide an auxiliary training target, and slightly refine action scores at inference. Experiments on R2R-CE and RxR-CE show improvements over the underlying navigation baselines.

Critical strengths

The paper presents an interesting attempt to look beyond a policy’s raw confidence by examining how its decisions respond to controlled internal changes. The method is integrated into graph reasoning, mapping, training, and inference, and is evaluated on two standard VLN benchmarks with detailed ablations, paired comparisons, and runtime analysis.

Critical weaknesses

My main concern is whether the proposed interventions actually measure decision stability in a meaningful way.

A central challenge in VLN is to align language with visual observations and scene geometry, yet the method deliberately weakens one of these sources and treats changes in the resulting action scores as instability. I am not convinced that this interpretation is always valid. For example, if the instruction requires the agent to turn right, masking the most relevant instruction token may cause it to go straight, but this could simply show that the original decision was correctly grounded in the instruction rather than that it was fragile. Conversely, an intervention may accidentally change an incorrect prediction into the correct one, which also does not provide clear evidence of reliability. Since these interventions can remove information genuinely needed for the action, the meaning of the measured response dispersion remains unclear.

In addition, the evaluation mainly shows improvements on standard VLN benchmarks rather than robustness under meaningful perturbations, while the gain from online verification itself is small and not statistically significant.

Questions for authors
  1. Could the authors explain more clearly why sensitivity to the proposed interventions should indicate an unreliable decision? In particular, how does the method distinguish brittle dependence from valid dependence on instruction or geometric evidence?

  2. Why does the competition intervention equalize only the top two actions rather than the top-k actions or all candidates in the shortlist?

  3. Why does the grounding intervention mask only the single most aligned token, and how sensitive is the method to the number and selection of masked tokens?

Reviewer_xoV4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CTPNav, a vision-language navigation framework for continuous environments that measures the stability of candidate navigation decisions under controlled internal interventions. The method applies a mean-minus-dispersion criterion at two levels: edge-level graph reasoning and action-level goal scoring, using the resulting responses both for training-time distillation and inference-time score correction. Experiments on R2R-CE and RxR-CE show improved performance over strong topological-planning baselines, along with ablations analyzing the roles of the intervention channels and the dispersion term.

Critical strengths
  1. The paper tackles a real issue in online topological VLN, namely that a high action score does not tell you whether that score is robust to small changes in the underlying computation. This is a sensible problem to study, and the framing in Figure 1 (Page 2) makes the intuition easy to grasp: the same state can admit multiple plausible goals, and the relevant question is not only “which one is highest now?” but also “which one survives mild counterfactual stress tests?”
Critical weaknesses
  1. The novelty of this paper is limited, as counterfactual learning has already been extensively explored in vision-and-language navigation [A-C]. In addition, the paper lacks real-robot experiments. [A] Counterfactual vision-and-language navigation via adversarial path sampler, ECCV 2020. [B] Counterfactual vision-and-language navigation: Unravelling the unseen, NeurIPS 2020. [C] Towards a Causally-inspired Evolving World Model for Vision-and-Language Navigation in Continuous Environments, TPAMI 2026.
  2. The paper’s title and motivation emphasize online “ask what if before committing,” but the strongest gains come from training-time distillation and evaluation-time mapping, not from the actual online verification step. This is the sharpest mismatch in the paper. In Table 3(a) (Page 8), once Edge and Map are on, Verify alone gives only +0.24 SR and +0.20 SPL, whereas Distill alone gives +0.82 SR and +1.03 SPL. The paper also states on Page 9 that the paired Verify-only SR change is not significant. In other words, the method’s most marketable component, the decision-time correction before execution, is not the component doing most of the work. The paper is honest about this, which I appreciate, but it also undercuts the headline narrative. If the main takeaway is really that intervention-derived targets help training and that adaptive graph resolution helps evaluation, the paper should say that more directly.
  3. The performance gains are meaningful but modest, while the inference overhead is noticeable. On RxR-CE Val-Unseen in Table 2 (Page 7), the gain over DGNav is +0.75 SR and +1.61 SPL. That is fine, but not dramatic. Meanwhile, Table 11 (appendix) reports latency increasing from 78.8 to 119.6 ms per high-level decision, roughly a 50% increase, due to extra graph passes. Because Verify itself contributes only a small, statistically weak part of the overall improvement, the practical case for always paying this online cost is not fully convincing. A clearer Distill-only deployment comparison in the main paper would help.
Questions for authors
  1. The novelty issue.
  2. Can the authors provide multi-seed results for the full method and the within-pipeline baseline on both R2R-CE and RxR-CE? Right now, the main tables and several ablations are single-seed. Since the reported gains are modest, even a 3-seed mean ± std for the core comparisons would increase my confidence substantially.
  3. Baseline comparisons in the reported tables Although the paper includes relevant topological-planning baselines such as ETPNav and DGNav, the comparison with prior work is still not sufficiently comprehensive for a submission to ICLR 2027. In particular, Table 2 includes only three baselines on RxR-CE, despite the existence of several relevant and competitive methods, including HNR, ETP-R1, and more recent approaches such as GA-VLN. The authors should clarify the baseline selection criteria and provide a broader comparison with recent methods, taking into account differences in training data and evaluation protocols. This would help establish the competitiveness of CTPNav beyond improvements over its immediate architectural predecessors.
Reviewer_LgXCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates decision reliability in vision-language navigation in continuous environments. The paper proposes CTPNav, a framework based on decision-time counterfactual stability. The central idea is to evaluate candidate actions or graph relations under controlled internal interventions and quantify their stability using the mean response minus a dispersion penalty. Experiments on R2R-CE and RxR-CE demonstrate improvements over several established VLN methods.

Critical strengths
  1. The paper investigates an interesting aspect of decision reliability in topological navigation.

  2. The experimental evaluation is relatively comprehensive.

  3. The implementation and reproducibility details are generally well documented.

Critical weaknesses
  1. The central assumption of this work is that candidate actions receiving consistent support across selected internal interventions are more reliable navigation decisions. However, the connection between internal response stability and actual navigation correctness remains primarily empirical and indirect. Please clarify the interpretation and limitations of internal response stability as a measure of decision reliability.

  2. The paper provides a useful mean-only comparison, demonstrating that incorporating dispersion yields additional gains. Nevertheless, this does not fully establish that the improvements require the proposed counterfactual-stability interpretation, rather than arising from the regularizing effects of using multiple perturbed predictions.

  3. One of the main contributions is the use of counterfactual responses to refine action preferences before execution. However, the experimental results show that this component has a relatively small independent effect. With Distill enabled, adding Verify increases SR from 54.30% to 54.53% and SPL from 45.78% to 45.98% on RxR-CE Val-Unseen.

  4. How sensitive is the method to the choice and strength of internal interventions? In particular, how does the geometric attenuation factor affect the stability estimates and downstream performance?

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.