acceptodds
Withdrawn from ICLR 2027

Disentangling the Execution Gap from the Information Gap with Conditional Computation in Vision-and-Language Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-and-language navigation turns a spoken instruction into a route through a real building, so a robot can act on directions in a space it has not been given as a map. In continuous environments the agent sees only first-person frames and moves through low-level steps and turns. Recent video-language navigators process image histories and emit actions. Yet these tasks still differ in their information requirements: some require finding or grounding a target, whereas others directly specify distances, angles, or a return to the starting point. A single aggregate success rate cannot tell whether a failed navigator lacks the ability to execute or lacks the information needed to determine the correct action. We route each episode by how completely its instruction already states a self-motion program. When language specifies that program, we introduce Proprioceptive Navigation (PNav), a conditional computation architecture that holds an origin register outside the frozen policy and passes the program to an external executor. Without training, PNav reaches a new SOTA success rate on NavSpace and leaves trajectories outside its takeover scope unchanged. Further closed-loop and perceptual-aid tests indicate that the tested readout, control and bypass-perception modules recover only part of the gap to a privileged-information reference. These results suggest that VLN failures have different causes: when language already specifies an action-relevant motion program, scaling the visual model alone may target the wrong bottleneck, while other instructions still need a usable goal condition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-and-Language Navigation, Multimodal Agents, Embodied AI, Conditional Computation, Program Execution, Spatial Reasoning, Robotics and Autonomy, Benchmark Diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tcBMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper finds the execution gap and information gap in VLA navigation, and addresses these two gaps with the proposed PNav, which executes a metric or return program from an external origin register and otherwise leaves control with the frozen policy. It also improves the results on NavSpace, showing the value of the method.

Critical strengths
  1. No additional training is needed for this result, and the improvement is substantial.
  2. The method is tested on multiple benchmarks, such as SNav, StreamVLN and LLaVA-Video. The consistent improvements support the effectiveness of external execution.
Critical weaknesses
  1. PNav mainly works for instructions with explicit distances and angles, or for returning to the starting point. However, it is hard to get accurate structured commands in the real world. The proposed method is hard to generalize to other scenarios.
  2. The paper does not provide enough comparison with other VLM/VLA based approaches. For example, the VLM can also generate structured motion commands and then let the same deterministic controller execute them. In this way, it can still keep the language and visual understanding ability of the VLM while having the benefit of precise execution. Without this comparison, it is not clear why using a rule-based parser is a better choice, especially for more complex navigation instructions.
  3. In Table 4, PNav achieves almost the same Precise navigation performance across SNav-7B (54.2%), StreamVLN (54.2%), and LLaVA-Video (53.2%), even though their original performance is quite different. Does this mean that the improvement is mostly coming from the deterministic executor rather than the VLM itself? It would be better to further investigate how much the underlying VLM actually contributes to the final performance.
Questions for authors
  1. Why does the paper use a rule-based parser instead of letting the VLM generate structured motion commands for the deterministic executor? Have the authors considered comparing these two approaches?
  2. How well can PNav handle more complex navigation instructions that require both visual understanding and precise execution, especially when obstacles block the planned path?
1 comment
AuthorsOpenReview ↗

Thank you for the positive assessment and helpful suggestions

Thank you for the constructive review. The suggested comparison between a rule parser and a VLM generated structured program was especially useful. We have now performed this comparison while holding the executor fixed. The results also clarify why different backbones approach similar Precise performance after adding PNav.

1. Rule parser versus VLM generated program.

We evaluated all 231 NavSpace instructions independently labeled by humans as fully executable.

Frozen VLA without executor: 13.4% SR.

Rule parser: 85.3% valid, 76.2% exact, 54.5% SR.

Frozen VLM program generator: 94.8% valid, 68.4% exact, 50.6% SR.

Human gold program from instruction only: 100% exact, 71.0% SR.

The VLM can therefore generate a structured program and use the same executor successfully. The fixed parser is one implementation of the program interface rather than a requirement of PNav. The VLM has higher coverage, while the rule parser has higher program accuracy. The human program reaches 71.0%, showing that better program extraction can further improve the system.

There are 143 instructions where the rule parser and VLM produce the same correct program. Their trajectories are identical in 141 cases, and success is 101/143 versus 100/143. Once the correct program is supplied, the source of that program has almost no effect on the resulting behavior.

2. Why use a conservative rule parser?

We evaluated an independent set of 300 instructions written by people who did not know the parser grammar.

On 100 fully executable instructions, exact program accuracy is 74% for the rule parser and 67% for the VLM. On 100 mixed instructions, the VLM recovers the executable prefix slightly more often, 64% versus 59%, but incorrectly treats the complete instruction as executable in 18% of cases, compared with 7% for the rule parser. On 100 grounding required instructions, erroneous complete takeover occurs in 10% versus 2%.

We therefore do not argue that rule parsing is inherently better. Its current advantage is conservative and auditable delegation. A learned program generator could replace it if its routing precision improves.

3. Does the executor explain most of the gain?

For fully executable instructions, yes. We believe this supports rather than weakens the execution gap hypothesis.

Across all 1,228 NavSpace instructions, two independent annotators labeled semantic executability using only the instruction. Raw agreement was 90.8% and Cohen's κ was 0.82.

Fully executable E: 13.4% to 54.5%, a gain of 41.1 points.

Mixed M: 28.3% to 26.1%.

Grounding required G: 31.6% to 32.5%.

Thus the large gain appears exactly where language already determines the motion program. This also explains why SNav, StreamVLN, and LLaVA Video all approach about 53% to 54% on Precise after PNav, despite very different frozen performance. Once the program is specified, persistent execution becomes the dominant missing capability. The visual model remains necessary for the much larger set of mixed and grounding tasks.

The existing OctoNav intervention supports the same interpretation. Coordinate parsing alone leaves SR at 5.7%, while parsing plus execution raises it to 19.7%. A structured representation alone is not sufficient.

4. Complex instructions that require both precise motion and visual reasoning.

We implemented residual handoff. The executor completes only the metric prefix, then returns control to the frozen visual policy with the remaining instruction and preserved RGB history.

On 100 independently written mixed instructions:

Frozen: 29%.

Recognition routing: 25%.

Semantic routing: 30%.

Residual handoff: 36%.

Human program and handoff: 41%.

We also evaluated 90 eligible Precise episodes. Frozen succeeds on 10/90, or 11.1%, while residual handoff succeeds on 26/90, or 28.9%. The exact McNemar test gives p = 0.00372. Among the 83 cases with a nonempty residual, success improves from 10.8% to 22.9%.

This shows that precise execution and visual reasoning can be composed within one episode rather than competing for control over the whole trajectory.

5. Obstacles.

We agree that PNav is not a general obstacle planner. It does consume collision feedback, so blocked forward actions are not integrated as successful motion. In the 90 episode experiment, 27 trajectories encounter at least one blocked action and nine encounter two consecutive blocked actions. We will make clear that this is feedback corrected execution, not visual route replanning.

We especially appreciate the suggestion to compare VLM generated programs with the fixed parser under the same executor. This experiment helped us separate program extraction from execution much more cleanly, and we believe it strengthens both the analysis and the scope of the paper.

Reviewer_4K1wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates execution-related and information-related failures in vision-and-language navigation. It introduces PNav, a wrapper that parses metric and return instructions into programs and executes them using an external origin register, while otherwise retaining a frozen navigation policy. The study combines paired NavSpace evaluations with offline instruction categorization, additional backbone and cross-corpus experiments, and diagnostic probes for target-related information.

Critical strengths
  1. The statistical analysis includes paired significance testing and a scene-cluster bootstrap, strengthening the empirical assessment.
  2. Experiments with additional backbones suggest that the execution mechanism is useful beyond a single navigation policy.
  3. The paper explicitly distinguishes runtime inputs from privileged offline diagnostics and acknowledges several limits of its conclusions.
Critical weaknesses
  1. The main deployment configuration does not fully implement the semantic routing principle emphasized in the paper. Equation 5 requires a recognized program and sufficient budget, but not residual closure. Appendix B calls closure diagnostic, and Appendix C states that residual handoff is disabled in the main run. A full-benchmark comparison between recognition-only routing, closure-gated routing, and residual handoff is needed to connect the reported gains to the proposed principle.

  2. The offline partition does not independently identify information requirements. The metric endpoint-compatible class is defined using the compiler’s ideal endpoint and the hidden scoring goal, while remaining cases enter the grounded remainder. This can conflate grounding requirements with parsing failures or endpoint incompatibility. This is not an allegation of runtime goal leakage; it is a limitation of the diagnostic interpretation.

  3. The negative evidence does not sufficiently disentangle causes of unsuccessful grounding. Weak readouts may reflect representation, training, memory, or viewpoint-selection limitations. Proposition A.2 establishes a conditional ambiguity result, but the paper does not establish how frequently its identical-history, conflicting-goal assumptions hold in the benchmark. The authors appropriately qualify their claims, yet the central mechanistic distinction remains underdetermined.

  4. Generalization evidence is limited by coverage and sample size. Parser triggers cover only 5.22% of R2R-CE and 3.35% of RxR-CE; the latter comparison improves from 13 to 16 successes. The additional paraphrase and human-label studies are useful but small.

Questions for authors

Can the authors evaluate closure-gated routing on the complete paired benchmark and independently annotate semantic executability? Can they separate language parsing from execution by feeding model-generated and externally parsed programs to the same executor?

Reviewer_w7NMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies vision-and-language navigation in continuous indoor environments, where an agent follows an instruction using first-person RGB frames and emits low-level actions. It argues that benchmark success rates mix two kinds of failure: the agent cannot execute a stated motion, or it lacks the scene or target information needed to choose the right action.

The main contribution is PNav, a wrapper around a frozen video-language navigator (SNav-7B). A fixed grammar parses the instruction. If the instruction states a metric program (a sequence of turns and distances) or a RETURN command, a deterministic executor takes over. It tracks pose from its own emitted actions and collision feedback in an “origin register,” then runs the program to completion. Otherwise, the frozen policy keeps control for the whole episode. The decision is made once from the instruction text and does not change during the episode.

On 1,228 NavSpace episodes, PNav raises the six-axis macro success rate from 27.6% to 35.7%. Nearly all of the gain is on the Precise axis (11.4% to 54.2%). Other axes are identical to the baseline, and Structure rises from 13.0% to 19.0%. The paper also reports that episodes without takeover keep their original trajectories under deterministic execution.

Critical strengths

Paired statistics and reporting on the main result. The main comparison runs both arms on identical episodes with the same checkpoint, decoding, and success criterion. It reports exact McNemar tests and Newcombe intervals, and the scene-cluster bootstrap gives a similar macro difference (8.13 points). This is careful evaluation practice and makes the Precise result credible as a measurement. It matters for my assessment because the effect is large and statistically clear on the axis where it appears.

Honest negative results. The paper reports that the tested identity, bearing, distance, and visibility readouts do not reach a usable goal condition, and it compares them against a privileged-information reference.

Critical weaknesses

The performance gain is concentrated on one axis. Of the 8.1 point macro improvement, all improvement is attributable to the "Precise" and "Structure" conditions. These are the cases that are listed in Table 5 in compiler signal, so these are the tasks for which there were samples that could be parsed from the instruction. The statement "The comparisons use parser-triggered subsets from R2R-CE (Anderson et al., 2018; Krantz et al., 2020) and English RxR-CE (Ku et al., 2020; Krantz et al., 2020), target-instance episodes from REVERIE-CE (Qi et al., 2020; Li et al., 2022), and coordinate episodes from OctoNav (Gao et al., 2026)." states that only instructions that could be parsed for direct execution were used for the comparison in table 5. Therefore, results must be interpreted understanding that the comparison is only valid for instructions that can be parsed directly into a set of movement commands. This makes the method benchmark-specific to those that have tasks with explicit motion commands to be followed.

It is unclear if the executor is open loop with respect to the environment. Section 3.3 describes an arbitration bit that controls whether the executor or frozen policy is used. However, it is unclear if this changes over the course of an episode. It appears that after takeover, the executor executes the plan to completion. The executor corrects only its own dead-reckoned pose and never appears to pass control to the frozen policy. This further limits utility to only when instructions are exact parseable movement instructions, which are vanishingly rare in practical applications.

Together, these imply that the paper's contribution is limited to a subset on benchmarks that are not readily applicable to practical applications.

Questions for authors

[Clarification] For each axis, what fraction of episodes does the takeover condition actually trigger on, and what is the success rate of control versus PNav within the taken-over subset alone? Table 2 gives a partial answer for the metric-compatible slice, but not for each axis.

[Clarification] How often do NavSpace Structure and other axes contain mixed instructions, where a metric prefix is followed by a grounded clause? The paper says the residual-closure check rejects takeover on these, which means the executor cannot help even with the parseable prefix. What would the success rate be if the executor handled the prefix and the VLA handled the remainder?

[Clarification] Section 5.2 states "For grounding-dependent instructions, geometric execution becomes useful only after the interface exposes a sufficiently reliable target condition." Does your method test cases where the frozen policy guides to the point where a sufficiently reliable target condition is met and then hand off to the executor?

1 comment
AuthorsOpenReview ↗

Thank you for the helpful assessment and suggestions

Thank you for the careful review. We agree that PNav itself is selective. Our original presentation may have made the paper look mainly like a method judged by how many NavSpace axes it improves. That is not the central contribution we intended.

The main contribution is an empirical diagnosis of two different failure regimes in vision and language navigation. We use PNav as a controlled positive intervention for the execution regime, and negative experiments to test whether the same intervention can repair tasks that still require target or scene information. PNav is intentionally simple to make its source easier to identify.

1. The gain is predicted by semantic executability rather than benchmark axis.

We independently annotated all 1,228 NavSpace instructions. Two annotators saw only the instruction and did not see parser outputs, hidden goals, trajectories, or model results. Raw agreement was 90.8% and Cohen's κ was 0.82.

Human classnFrozenPNav
Fully executable, E23113.4%54.5%
Mixed, M18428.3%26.1%
Requires grounding, G81331.6%32.5%

The effect is concentrated in E, with a 41.1 point gain. M changes by minus 2.2 points and G by only plus 0.9 points. This is the core prediction of the paper. Explicit execution helps when language already specifies the required motion, but it does not solve a task whose action still depends on grounding.

The recognition router triggers on 392 of 1,228 instructions. It recognizes 85.3% of E, 60.9% of M, and 10.2% of G. Recognition is therefore not equivalent to semantic executability, which explains why unconditional takeover can hurt mixed instructions.

2. The concentration on Precise and Structure is consistent with this diagnosis, not evidence of a benchmark specific trick.

Precise contains explicit metric programs at very high frequency, so it is exactly where the intervention should have the largest effect. The other axes require more grounding and show little or no net gain.

The same effect appears across backbones. On Precise, SNav reaches 54.2% with PNav, StreamVLN 54.2%, and LLaVA Video 53.2%, despite very different frozen performance. Once language determines the motion program, the original visual backbone becomes much less important for that component.

The executor also transfers beyond NavSpace. R2R CE, RxR CE, REVERIE CE, and OctoNav improve when a suitable program is present. This tests transfer of the execution mechanism, not broad language coverage.

We also tested 300 independently rewritten instructions written without knowledge of the parser grammar. The same semantic pattern remains.

3. We implemented the reviewer's suggestion of executing a metric prefix and then returning control to the visual policy.

On 100 independently written mixed instructions, Frozen obtains 29%, recognition routing 25%, semantic routing 30%, and prefix execution followed by residual handoff 36%. A human program and handoff reaches 41%.

This shows that mixed instructions need not be a binary choice between the executor and the visual model. Semantic routing avoids harmful takeover, while composition improves over the frozen policy.

We also tested 90 eligible Precise episodes. Frozen succeeds on 10 of 90, or 11.1%, while prefix execution followed by handoff succeeds on 26 of 90, or 28.9%. The exact McNemar test gives p = 0.00372. Among 83 cases with a nonempty residual, success improves from 10.8% to 22.9%. This is an eligible subset, not a replacement for the complete benchmark result.

4. We agree that the current executor is not a general obstacle planner.

It does not visually construct a new route around arbitrary obstacles. However, it is not blind dead reckoning. Its register uses committed actions and collision feedback, so blocked forward actions are not counted as successful motion. In the new 90 episode experiment, 27 trajectories contain at least one blocked forward action.

These observations show that execution uses environment feedback, but they do not establish general obstacle avoidance.

The reverse composition proposed in the review also remains open on NavSpace. Our current target readouts are not reliable enough to support visual navigation first followed by geometric handoff, so we do not claim that result.

The contribution is therefore not that deterministic control should replace visual navigation. It is that different navigation failures require different interfaces. When language already fixes the motion, persistent execution can be the missing capability. When the action still depends on scene or target information, the visual policy remains necessary.

We appreciate that these comments pushed us to clarify the paper's central contribution and scope. We hope the new analyses make clear that PNav is a controlled intervention for diagnosing the execution regime rather than a universal replacement for visual navigation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.