acceptodds
Under review as a conference paper at ICLR 2027

Aligned-VLN: Modeling Navigation Progress over Ordered Instructions

Abstract

Many recent Vision-and-Language Navigation (VLN) methods based on vision-language models directly predict actions from natural-language instructions and visual observations. While action supervision specifies what action to execute, it provides limited guidance about which instruction content should guide decisions at different navigation stages. In long-horizon navigation, a locally feasible action may be premature under the instruction order, making action selection dependent on execution progress. We refer to this challenge as instruction-progress mismatch. We propose Aligned-VLN, an action-chunk VLM policy that models navigation progress relative to the ordered instruction. Our key idea is to learn an instruction-conditioned latent alignment over the original instruction tokens. The alignment yields an instruction-relative progress coordinate and an attended instruction representation, which are fused with a learned navigation-state representation into a progress context for action decoding. This context conditions the decoder through layer-wise progress attention and final readout injection. The latent alignment is learned from weak trajectory-derived supervision without token-level execution annotations, with monotonic supervision encouraging progress estimates to follow trajectory order. At each decision step, Aligned-VLN predicts an action chunk with one VLM forward pass, without generated progress descriptions, predefined sub-instructions, or an external progress model. On R2R-CE Val-Unseen, Aligned-VLN achieves 63.5% SR and 56.3% SPL. on RxR-CE, it achieves 59.6% SR and 51.5% SPL. Separate R2R-only controlled experiments show improvements over an action-only policy using the same backbone and action interface.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-and-Language Navigation, Vision-Language Models, Multimodal Learning, Embodied AI, Instruction Following, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fdwmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Aligned-VLN augments Qwen2.5-VL-3B with progress and action query tokens to model navigation progress through instruction-token alignment. Content matching and a Gaussian positional bias jointly produce an alignment distribution, whose expected token position and attended instruction representation form a progress context for action decoding. Training combines action imitation with trajectory-derived progress, alignment, increment, and monotonicity losses, using the fraction of completed forward primitives as weak supervision. Each forward pass predicts three action symbols, with the first executed before observing and replanning.

Critical strengths
  1. Meaningful problem and concrete mechanism. For multi-stage instructions, locally feasible actions can lead to errors such as premature turns. Linking navigation progress to the instruction content relevant at the current stage is a well-motivated direction.

  2. Informative design comparisons. Full improves over PositionOnly by 2.72 SR and 2.63 SPL percentage points, while removing monotonicity supervision reduces SR by 2.72 points. These results suggest benefits from content-based scoring and temporal-order supervision.

  3. Simple deployment pipeline. Progress estimation and action prediction share a single VLM forward pass, without generating progress text or invoking a separate progress model. Demonstrations on two robot platforms provide preliminary evidence of deployment feasibility, although inference speed and deployment success rates are not quantified.

Critical weaknesses

W1. Training and validation primarily assess trajectory progress rather than semantic instruction progress. The current evidence does not directly establish whether the model tracks the semantic execution stage of the instruction.

W2. The incremental contribution of token alignment is small and insufficiently isolated by the ablations. ScalarOnly improves SR over ActionOnly by 6.96 percentage points, compared with Full’s total gain of 7.56 points. The former accounts for approximately 92% of the latter, although this is a descriptive ratio rather than a decomposition of independent causal contributions. Full improves over ScalarOnly by only 0.60 SR points, equivalent to approximately 11 additional successful episodes out of 1,839. Its 1.67-point SPL improvement may nevertheless have practical value.

W3. The main results lack key controls under the same training regime, the training corpus is insufficiently specified for reproduction, and comparisons with the latest state-of-the-art methods are missing. The main experiments combine R2R, RxR, and ScaleVLN, whereas the ablations use only R2R. These ablations support component benefits in the R2R-only setting but do not establish whether token alignment is similarly necessary for the main results. Aligned-VLN exceeds the reported SR of the 3B NaVIDA model by 2.1 points on R2R and 2.2 points on RxR. However, as the authors acknowledge, cross-paper protocols may differ, so these differences cannot be interpreted as controlled evidence of gains from progress alignment.

Questions for authors
  1. Could the authors provide a small-scale semantic-stage or premature-action diagnostic to demonstrate the added value of token alignment beyond trajectory-progress prediction and a scalar baseline?

  2. How does the proposed method differ from the instruction-attention/progress coupling in Self-Monitoring? Which gains arise from the shared expected-position readout rather than standard auxiliary scalar-progress supervision?

  3. What are the size, mixture proportions, and conversion rules of the aggregated training corpus, and how is the model initialized? How does ScalarOnly perform under the same aggregated-training configuration?

4 comments
AuthorsOpenReview ↗

Response to Reviewer fdwm

Thank you for suggesting that we look at when the agent turns.

W1 and Q1: Semantic stages and early turns

Figure 7 (Appendix D) shows the doorway sequence in episode 898 on the R2R-CE unseen validation split. Panels A and B are consecutive expert observations. Panel C is a Habitat rendering after the expert's turn. It illustrates the timing problem in Fig. 1. At the bedroom doorway, the expert moves forward 25 cm before turning right 15°. Full matches the expert's action at both A and B. ActionOnly and PositionOnly predict the turn one step early.

We use the fraction of completed forward steps to supervise the expected token position and center the soft target over tokens (Sec. 3.5). Figure 3(b) shows the weights moving from the opening clause to the destination clause on a held-out route. On 2,290 expert replay states with a clear FGR2R stage away from a clause boundary, summing Full's weights by clause matches the label 53.0% of the time. The expected position alone matches 48.7% of the labels. The fraction of forward steps from the complete expert route matches 60.4%.

W2: What token alignment adds to scalar progress

ScalarOnly uses route progress as a cue to where the agent is in the instruction. With that cue, it scores 6.96 SR points above ActionOnly on R2R. Full uses the same token weights to estimate the current instruction position and pool the features the decoder uses to choose its next action (Appendix A.5).

Full improves over ScalarOnly by 0.60 SR and 1.67 SPL points on R2R, and by 3.01 SR and 2.61 SPL points on RxR (Table II). On RxR, nDTW rises from 0.543 to 0.593. Full, PositionOnly, and ScalarOnly train on the same RxR trajectories and monotonic pairs.

Table II. Ablations with one training source at a time

VariantR2R SRR2R SPLRxR SRRxR SPLRxR nDTW
ActionOnly31.8128.977.265.740.310
NoMono36.6533.4619.2516.700.585
ScalarOnly38.7734.4019.5816.140.543
PositionOnly36.6533.4421.2517.470.578
Full39.3736.0722.5918.750.593

The R2R scores come from Table 2 of the manuscript. We retrained the RxR variants on RxR only.

Q2: Self-Monitoring and shared weights

Self-Monitoring uses instruction attention to predict progress for ranking trajectories or backtracking. In Full, the decoder gets both an expected position and pooled instruction features from the same token weights (Eqs. 3–4). Self-Monitoring, Secs. 2.2–2.4.

Full and PositionOnly both share weights across the two readouts. Full also scores tokens by content. Relative to PositionOnly, Full gains 2.63 SPL points on R2R (95% paired interval over routes: [0.03, 5.22], with checkpoints fixed). On RxR, it gains 1.34 SR and 1.28 SPL points. To measure what sharing contributes, we would keep both readouts and compute each from its own distribution. To test progress as auxiliary supervision alone, we would keep the loss and remove the decoder's progress input.

W3 and Q3: Recent baselines and pooled training

Table I. Published scores on the unseen validation splits

MethodR2R-CE SRR2R-CE SPLRxR-CE SRRxR-CE SPL
Aligned-VLN (paper Table 1)63.556.359.651.5
NaVIDA61.454.757.449.6
MapDream59.854.459.449.2
Progress-Think (three actions)60.153.6——
Progress-Think (one action)58.051.1——

Sources: NaVIDA, Tables 1–2 and Secs. 3.3, 4.1; MapDream, Table 1 and Sec. 4.2; Progress-Think, Tables 1, 2, and 5.

Progress-Think reports RxR-CE in a transfer setting. We execute the first predicted action symbol and replan after the next observation. NaVIDA executes the first two predicted chunks and trains with DAgger and inverse-dynamics supervision. MapDream trains with non-oracle samples and reinforcement fine-tuning.

For Table 1 of the manuscript, we trained Full with Qwen2.5-VL-3B on pooled trajectories from R2R, RxR, and ScaleVLN. Table III gives the counts from the archived source lists before conversion to action examples.

Table III. Source records before action-level conversion

SourceTrajectory–instruction records
R2R10,819
RxR19,996
ScaleVLN155,098
Total185,913

Each symbol encodes one to three consecutive forward steps or same-direction turns. We fill unused target slots with STOP and label progress by the fraction of forward steps completed (Appendices A.3–A.4). Table II compares Full and ScalarOnly separately on R2R and RxR. Table 1 of the manuscript reports Full after pooled training.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Thank you for the clarification and additional experiments. The RxR ablations and semantic-stage analysis strengthen the empirical evidence. However, it remains unclear whether the method tracks instruction semantics beyond trajectory progress, and concerns about novelty and the attribution of the gains remain insufficiently addressed. I therefore maintain my original rating.

Thank you for carefully reviewing the additional experiments. You observed that ScalarOnly already achieves most of the SR gain on R2R. This is precisely why it is the relevant baseline for assessing Full: its backbone already encodes the complete instruction, and its action decoder already receives scalar progress. Against this baseline, Full gains a further 0.60 SR points and 1.67 SPL points on R2R. On RxR, both models were retrained with the same backbone, trajectories, and monotonic training pairs. Full raises SR from 19.58 to 22.59 and SPL from 16.14 to 18.75. These comparisons measure the navigation gain of the complete design over scalar progress conditioning.

In Full, the token weights that determine the expected instruction position also determine which instruction features the action decoder reads. The fraction of expert forward actions completed provides weak positional supervision for the alignment distribution. For a given positional bias, the match between the current navigation state and each token’s content can still change the relative token weights (Eq. 3). The model uses these weights to compute both the expected instruction position and the pooled instruction features. Gradients from action imitation also reach the weights through the decoder (Eq. 4 and Sec. 3.5). Figure 3(b) illustrates this readout on one unseen route, where the weights shift from the words about turning and leaving the pantry toward those about reaching the patio and stopping.

PositionOnly addresses the contribution of content matching to navigation more directly. It can already shift its weights with predicted progress and pool instruction features by position. Against this baseline, Full gains a further 1.34 SR points and 1.28 SPL points on RxR. The gain over ScalarOnly establishes the performance advantage of the complete design in these experiments, while the PositionOnly comparison supports the additional navigation value of matching instruction tokens to the current state. We respectfully ask you to reconsider your assessment of the method’s incremental contribution and your rating in light of these two comparisons.

Reviewer_aLHhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Aligned-VLN, which formulates navigation progress as a latent alignment over ordered instruction tokens, using a vision-language model (VLM) backbone (Qwen2.5-VL-3B by default). The model outputs an expected instruction-relative progress coordinate and instruction-attended representations, conditioning the action decoder at each step. Experiments on R2R-CE and RxR-CE are reported, along with some ablation analysis.

Critical strengths
  1. Attempts to formalize instruction-progress mismatch as a modeling problem and provide an efficient, continuous latent alignment. Adopts a more compact (3B) vision-language backbone and achieves strong results under certain settings.

  2. Adopts a more compact (3B) vision-language backbone and achieves strong results under certain settings.

Critical weaknesses
  1. Progress modeling is not new in VLN. Many previous works (e.g., Progress-Think) propose progress estimation techniques, and even release code/models. The authors do not provide direct comparison or qualitative discussion with these well-known, open-source works, despite clear technical and conceptual overlap.

  2. No results are reported versus MapDream (ICML 2026) or similar spatially- and memory-augmented VLN models, which are now strongest baselines for progress and evidence-aware navigation. Table 1 even misses recent work like MapDream.

  3. If I understood correctly, a large portion (possibly >50%) of the reported performance gains come solely from adopting the Qwen2.5-VL-3B backbone, rather than the Aligned-VLN architecture itself. The ablation results and Table 2 confirm that simply swapping the backbone leads to significant performance jumps, which casts further doubt on whether the claimed improvements actually stem from the proposed technical contributions. Moreover, the paper lacks extensive "backbone swap" experiments, i.e., re-running other baselines with the same VLM.

  4. total amount of training data used is not made sufficiently explicit. From the appendix, it seems that models are trained on a "large-scale aggregated trajectory corpus" (including R2R, RxR, ScaleVLN, etc), but the absolute training sample counts—and those of all compared methods—are not clearly and comparably reported. This is crucial, as many recent methods differ by orders of magnitude in training data, and reported metrics could be confounded by data scale, rather than architecture.

Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer aLHh

Thank you for asking how Aligned-VLN compares with Progress-Think and MapDream.

W1: Progress-Think

Progress-Think predicts the completed part of the instruction as text and feeds it to the policy. In a single forward pass, Aligned-VLN uses the original instruction tokens to compute a position and pool features for the next action (Secs. 3.3–3.4). Table I shows Progress-Think's R2R-CE results when it executes one or three actions per decision. Progress-Think, Sec. 3 and Table 5.

W2: MapDream

Aligned-VLN's SPL is 1.9 points above MapDream's reported score on R2R-CE and 2.3 points above it on RxR-CE (Table I). MapDream uses non-oracle samples and reinforcement fine-tuning.

Table I. Published scores on the unseen validation splits

MethodR2R-CE SRR2R-CE SPLRxR-CE SRRxR-CE SPL
Aligned-VLN (paper Table 1)63.556.359.651.5
NaVIDA61.454.757.449.6
MapDream59.854.459.449.2
Progress-Think (three actions)60.153.6——
Progress-Think (one action)58.051.1——

Sources: NaVIDA, Tables 1–2; MapDream, Table 1 and Sec. 4.2; Progress-Think, Tables 1, 2, and 5.

Progress-Think reports its RxR-CE result in a transfer setting.

W3: Backbone

In Table 2 of the manuscript, Full and ActionOnly use the same Qwen2.5-VL-3B backbone and train on the same R2R trajectories. Full scores 7.56 SR and 7.10 SPL points above ActionOnly. NaVIDA also uses Qwen2.5-VL-3B. StreamVLN uses LLaVA-Video-7B with Qwen2-7B. NaVIDA, Sec. 4.1; StreamVLN, Sec. 4.2.

W4: Training data

We trained the final model in Table 1 on pooled trajectories from R2R, RxR, and ScaleVLN. Table III counts the archived trajectory–instruction records before conversion to action examples. For the ablations in Table 2, we trained all variants on the same R2R trajectories.

Table III. Source records before action-level conversion

SourceTrajectory–instruction records
R2R10,819
RxR19,996
ScaleVLN155,098
Total185,913

If these clarifications address your main concerns, we would be grateful if you could reconsider your rating.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Reviewer_RuQxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets instruction-progress mismatch in VLM-based VLN-CE policies: action supervision does not tell the policy which part of the ordered instruction should govern the current decision. Aligned-VLN derives a single distribution over the original instruction tokens from learnable progress queries in a Qwen2.5-VL-3B backbone. Two readouts come from this one distribution: an expected token position, used as a progress coordinate, and an attended instruction feature. Both condition an action-chunk decoder, and the whole model is trained with trajectory-derived weak supervision. It reports 63.5 SR on R2R-CE and 59.6 SR on RxR-CE, and gains 7.56 SR over an action-only baseline in controlled R2R-only ablations.

Critical strengths
  • The motivation is clear. The design is simple and runs in a single forward pass, with no generated progress text, no sub-instruction annotations and no external progress model. Coupling “where am I” and “what to follow” through one distribution is elegant.
  • Results are competitive for a 3B backbone and improve over NaVIDA, the closest comparable baseline, on both benchmarks.
  • The controlled ablations are well organized, the numbers are internally consistent, and the paper is candid about its limitations.
Critical weaknesses
  • Attribution of the core contribution. Most of the gain over ActionOnly (6.96 of 7.56 SR) is already achieved by ScalarOnly. Full adds only +0.60 SR, from a single seed and without a CI for this pair. PositionOnly falls 2.12 SR below ScalarOnly, which suggests run-to-run noise of a similar size. Multi-seed results and paired CIs are needed.
  • Missing ablations. The “one distribution, two readouts” design is never compared against separate distributions or a single readout. There is also no variant where the decoder directly attends to all instruction tokens, and none where the progress losses are kept but the decoder conditioning is removed. Finally, the large-scale setting behind Table 1 has no ActionOnly counterpart.
  • Alignment evaluation and novelty. The replay analysis only correlates predictions with their own training target. RxR pose-trace word alignments or FGR2R annotations could test semantic alignment directly. Self-Monitoring (Ma et al., 2019a) already couples textual attention with progress estimation, so the paper should state more clearly what is new.
Questions for authors
  • What are the parameter count and latency overhead of the alignment module and the decoder, compared with NaVIDA?
  • Does ScalarOnly’s decoder still receive P^ℓ? What is the learned value of γ, which is unconstrained?
4 comments
AuthorsOpenReview ↗

Response to Reviewer RuQx

Thank you for suggesting more direct tests of the shared readouts.

W1: Training runs and confidence intervals

We used our compute budget to compare the components in Table 2 and trained each variant once. The R2R training file contains 157,762 records across action examples and monotonic pairs. Progress-Think and MapDream also report one set of scores per ablation. Progress-Think, Table 4; MapDream, Table 3.

Full scores 0.60 SR and 1.67 SPL points above ScalarOnly on R2R. The gains are 3.01 SR and 2.61 SPL points with RxR-only training. On R2R, ScalarOnly's SR is 2.12 points above PositionOnly's. ScalarOnly gives the decoder a scalar progress embedding. PositionOnly also gives it instruction features weighted by token position.

The Full–ScalarOnly gains are differences between aggregate scores; the paired bootstrap uses episode-level scores. Table IV gives the intervals for Full against ActionOnly and PositionOnly. We held the checkpoints fixed and resampled 613 routes covering 1,839 instructions. These intervals measure variation across routes; repeated training would measure variation across seeds.

Table IV. R2R paired route bootstrap intervals for score differences (95%)

ComparisonΔSR (points)ΔSPL (points)
Full − ActionOnly (Appendix A.6)[4.79, 10.33][4.54, 9.66]
Full − PositionOnly[−0.05, 5.55][0.03, 5.22]

W2: Ablation design

We used the same R2R trajectories in Table 2 to compare progress conditioning, token content matching, and temporal supervision. We then used the rebuttal compute budget to repeat the comparison on RxR. There, Full scores 22.59 SR and 18.75 SPL, compared with 7.26 and 5.74 for ActionOnly. Training ActionOnly on the pooled corpus would give the corresponding control for Table 1. NaVIDA also limits the cost of its ablations by leaving out ScaleVLN. NaVIDA, Sec. 4.3.

In Full, the decoder gets its progress coordinate and instruction context from the same token weights. Giving each readout its own distribution would isolate weight sharing. The single-readout, direct-attention, and auxiliary-only controls would test which inputs help the decoder. The R2R PositionOnly run used a next-progress consistency loss with a weight of 0.2. We document this setting in Appendix D of the revised manuscript.

W3: Semantic stages and Self-Monitoring

We projected the FGR2R clause labels onto the expert routes and kept 2,290 replay states with a single label away from clause boundaries. Summing Full's token weights by clause matches the labels more often than using its expected position alone. Table V also gives the result for the fraction of forward steps computed from the complete expert route.

Table V. Agreement with the projected FGR2R stage

Stage estimateAgreement
Full: token weights summed by clause53.0%
Full: expected token position48.7%
Forward-step fraction from the complete expert route60.4%

Self-Monitoring feeds instruction attention into a progress predictor. Full reads both the expected position and the pooled instruction features directly from the same token weights and gives both to the action decoder (Secs. 3.3–3.4). Self-Monitoring, Secs. 2.2–2.4.

Q1: Parameter count and latency

Full has 15.7% more parameters than the released NaVIDA checkpoint when tied weights are counted once. It adds 588.3M parameters, including 8.4M for alignment scoring and 542.0M for the action decoder. NaVIDA checkpoint.

Table VI. Model size and mean latency per model call

ModelTotal parametersMean latency
NaVIDA3.75B501.24 ms
Full4.34B108.47 ms

In this test, Full's mean latency is 78.4% lower. Both models ran on an H100 with 80 GB of memory, using BF16, FlashAttention 2, and batch size 1. Each call used nine 308 × 252 frames and 729 visual tokens. Full predicts three action symbols in one forward pass; NaVIDA decodes greedily, up to 16 new tokens. We used 5 warm-up and 30 timed calls for Full and 3 warm-up and 15 timed calls for NaVIDA. We timed model calls only.

Q2: ScalarOnly inputs and learned γ

ScalarOnly uses P^ℓ to build the route state. The decoder receives progress only through the scalar embedding derived from that state (Appendix A.5). We initialize γ at 1. The saved PositionOnly and NoMono checkpoints both give γ = 1.0.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Thank you for the rebuttal, which partially addresses my concerns. However, the backbone configuration and the additional parameters still leave me unconvinced that the performance gains can be attributed to the alignment mechanism itself. Therefore, I will maintain my original score.

Thank you for the FGR2R evaluation. However, Table V does not support the semantic-alignment interpretation. The learned alignment (53.0% / 48.7%) agrees with the clause labels less often than the simple forward-step fraction (60.4%), which suggests it captures little beyond the trajectory-derived prior it is trained toward. I accept the clarified distinction from Self-Monitoring, but the related-work description of Ma et al. (2019a) as a scalar-only signal should be corrected. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.