acceptodds
Under review as a conference paper at ICLR 2027

MemVLN: Streaming Episodic Memory for Vision-Language Navigation

Abstract

Vision-language navigation (VLN) requires an embodied agent to follow language instructions through 3D environments. Maintaining persistent episodic memory remains a challenge for VLM-based streaming agents in long-horizon navigation. We present MEMVLN, a streaming episodic memory framework that maintains a bounded bank of compressed perceptual features annotated with timesteps and action-derived relative poses, and produces compact memory tokens via cross-attention. We pair the proposed memory with a frozen pretrained geometry encoder. To learn the memory module, we design a masking-based trajectory summarization task with multi-granularity sub-trajectory decomposition, encouraging route reconstruction exclusively from episodic memory. Under navigation training restricted to the official R2R/RxR trajectories, MEMVLN achieves state-of-the-art performance on most reported metrics on standard VLN-CE benchmarks. Long-horizon analyses further characterize when episodic memory is useful. For physical evaluation, we collect 30 instruction-conditioned episodes in one physical scene, use 24 as demonstrations for post-training, and hold out six for testing. On the held-out episodes, MEMVLN improves success over the VLM baseline from 17% to 40%, and post-training further raises it to 47%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision and Language Navigation, Robot Learning, Deep Learning, Vision Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ibTyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MemVLN, an episodic memory-augmented framework designed for streaming visual input. The paper introduces a memory module comprising a Memory Bank and a Memory Perceiver. The Memory Bank continuously stores historical visual features, timestamps, and relative poses estimated based on executed actions, while utilizing token merging for spatial compression. When the memory bank exceeds its capacity limit, the system merges certain historical frames based on the average patch-level cosine similarity between adjacent frames. The experiments and analyses are comprehensive and support the potential of the framework, though I believe several details require further elaboration.

Critical strengths

This paper focuses on the problem of continuous historical memory in streaming VLN, addressing a critical bottleneck that current VLM-based navigation systems may face during long-horizon navigation. The modular design avoids directly concatenating all historical images into the language model's context and can, in principle, be integrated with existing streaming VLMs. Furthermore, by masking recent local visual tokens, the model is forced to reconstruct navigation instructions using memory tokens. This constraint is more direct in enforcing the retention and utilization of historical information compared to simply training the memory module passively via standard action prediction tasks. The experimental results and analyses are comprehensive and include real-world robot deployment tests.

Critical weaknesses

The main technical components of this paper including explicit historical feature storage, spatial-temporal positional encoding, and token merging are built upon well-established technical foundations. The authors need to clarify the following aspects:

  1. MemVLN does not achieve higher efficiency as claimed, and the module focuses on controlling context growth rather than building a more efficient model, and it may even incur additional inference overhead.
  2. Compared to existing explicit or implicit memory methods, what substantial innovations does this work introduce in terms of memory representation, writing mechanism, reading mechanism, and training objectives?
  3. It is necessary to verify whether the performance gains stem primarily from the new memory structure, or rather from the geometric encoder or the auxiliary supervision.
  4. The moudule of merging adjacent frames with the highest cosine similarity may lead to the loss of critical information, such as missing a crucial intersection in a hallway during a specific frame.
Questions for authors

In addition to the points raised in the weaknesses, I would like the authors to address the following questions:

  1. The authors should explain why increasing the memory bank capacity leads to performance degradation.
  2. How is the total loss for the navigation task and trajectory summarization task specifically defined? Does the auxiliary supervision genuinely improve the memory representation, or does it merely alter the output distribution of the language model?
  3. As cumulative errors accumulate during continuous navigation, do these errors directly affect the spatial encoding and historical localization within the memory?

Overall, the paper presents a reasonably sound framework, but several key issues remain. If the authors can adequately address my concerns, I am willing to raise my score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ibTy

1. Novelty: memory representation and supervision

Our main contribution is a learned implicit memory representation for VLN, paired with task-specific supervision. Compressed frame features are stored in a bounded episode bank. Learned queries, grounded in the current timestep and action-derived pose, read this bank into a fixed set of latent memory tokens for the policy.

The distinction from retaining encoder KV caches is the dedicated readout and its training objective. Masking local visual tokens during whole-trajectory and sub-trajectory summarization requires route reconstruction from memory at multiple temporal scales. This learns navigation-relevant representations without prescribing a semantic or topological map. Table 3 and General G1–G2 show navigation gains from memory and benefits from earlier episode history.

2. Auxiliary supervision and the total loss

For navigation and trajectory summarization, we use the same cross-entropy loss, averaged over supervised output tokens:

L(x,y)=−1∣y∣∑tlog⁡pθ(yt∣x,y<t).L(x,y) = -\frac{1}{|y|}\sum_{t}\log p_{\theta}(y_t \mid x, y_{ < t}).

Here xx denotes the task-specific conditioning inputs: the task prompt, visual representations and memory tokens. The target yy is the action sequence for navigation or the route instruction for summarization. Training processes each episode in temporal order, including its navigation and summary examples. Both tasks use the same loss, with no separate auxiliary-loss weight. Masking local visual tokens in summary examples directs reconstruction gradients through the memory reader and the LLM adapters. The drops under Zero-memory and Recent16 show that the trained policy uses its memory readout and earlier history; see General G2 for the comparisons.

3. Inference cost

The computational benefit is bounded context growth. The memory adds 300 tokens plus 7 prompt/special tokens, independent of trajectory length. Table 8 reports 503.2 ms for Baseline and 691.0 ms for Full on an RTX 4090; this 37.3% overhead includes both geometry and memory.

4. Frame merging and capacity

Merging similar frames can discard a brief but important cue, as noted in our limitations. Frame merging is one choice for maintaining the bounded bank. FIFO (first in, first out) is an alternative that removes the oldest frame entry when the bank exceeds capacity, retaining the most recent NbN_b frames without merging their features. A separately trained model using this FIFO policy gives the following results:

Separate training configurations with Nb=100N_b=100 and Nm=300N_m=300.

ConfigurationR2R SRRxR SR
Original Full — FRME55.055.4
FIFO configuration53.354.8

Task-aware retention and adaptive sampling are directions for improving consolidation.

When bank capacity increases, the reader still produces 300 tokens. More redundant or irrelevant entries may make retrieval and compression harder, which could explain the degradation in Table 4.

5. Accumulated pose error

Pose errors perturb the spatial embeddings used for retrieval. Visual features and timestamps remain available, and removing pose embeddings entirely causes a modest drop while still outperforming Zero-memory. Memory therefore remains useful without explicit pose encoding. See General G2 for the No-pose settings and results.

The authors’ response emphasizes bounded context, but bounded context does not necessarily imply higher inference efficiency. In fact, most existing VLN methods also employ bounded-context strategies. Therefore, I do not find this argument sufficient to support the claimed efficiency advantage of MemVLN. I am also not convinced by the authors’ characterization of MemVLN as an implicit memory method. In my view, it is more accurately described as a hybrid architecture that combines explicit feature storage with implicit latent-variable retrieval.

More importantly, the authors’ response still does not fully address the concern regarding the auxiliary supervision. The auxiliary task simultaneously updates both the Memory Reader and the LLM adapters. Therefore, it remains unclear whether the observed performance improvement actually results from a better memory representation, or is partly caused by changes in the output distribution or behavior of the language model induced by the auxiliary objective. I encourage the authors to provide more details about the auxiliary supervision.

Thank you for your follow-up comment.

Auxiliary supervision. Yes, the shared LLM adapters can contribute to the improvement. We deliberately train them alongside the Memory Reader so that the reader learns which route information to retrieve and the policy learns how to use it when choosing an action. Adding trajectory summarization improves navigation in both R2R comparisons.

R2R val-unseenSR without summarizationSR with summarizationGain
Geometry only (Tables 3 and 9)49.2%50.9%+1.7 pp
Geometry + memory (Table 3)49.7%55.0%+5.3 pp

The geometry-only policy gains 1.7 points from predicting the route instruction from unmasked local views, even though it has no reader. In the full model, we mask the local visual tokens during summarization, so the reader’s output is the only visual evidence for predicting the instruction. We attribute the 5.3-point gain to adding this supervision to the memory-equipped policy. We do not claim that the reader alone accounts for it.

Memory representation. We agree that MemVLN combines an explicit feature bank with a learned, implicit representation of route history.

Inference cost. We do not claim that MemVLN is faster than prior VLN methods. The reader supplies 300 memory tokens per decision regardless of trajectory length (§3.2). Table 8 reports mean policy latencies of 503.2 ms for Base and 691.0 ms for Full on the same RTX 4090.

Reviewer_LaHROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MemVLN adds a streaming episodic memory to Qwen3-VL-4B. Each frame’s feature, timestep, and action-derived relative pose are stored in a bounded bank; when full, the two most similar consecutive frames are merged. A perceiver with learnable queries reads the bank into 300 memory tokens for the LLM. A frozen StreamVGGT encoder adds geometry features. The memory is trained with trajectory summarization: local visual tokens are masked, so the memory must carry route information, with FGR2R sub-trajectories providing multi-scale supervision. Trained only on official R2R/RxR, MemVLN reaches 55.0% SR on R2R-CE and 55.4% on RxR-CE. On G1, it improves SR on six held-out episodes from 16.7% to 40.0%, and to 46.7% after post-training.

Critical strengths
  1. The design is simple. It can be added to an existing VLM without changing the LLM. The memory adds a fixed number of tokens (307) to the input, and this number does not grow with route length.
  2. Masking the local visual tokens during trajectory summarization is a clear way to force the memory to carry route information. Using sub-trajectories for multi-scale supervision is also reasonable.
  3. The model is trained only on the official R2R/RxR trajectories. This makes the setting data-efficient and easier to reproduce.
Critical weaknesses
  1. The memory module shows no measurable benefit on RxR, which contradicts the paper’s main claim. RxR is the more long-horizon benchmark. In Table 9, the Nb=0N_b = 0 variant (no memory, with geometry and auxiliary supervision) reaches 54.9 / 49.0 / 64.5 on RxR, while the full model reaches 55.4 / 48.4 / 64.6. The differences are within noise, and SPL is lower with memory. In Table 3, the geometry-only variant (56.3 SR / 49.7 SPL) also outperforms the full model. The RxR gains over JanusVLN therefore appear to come from the Qwen3-VL backbone and StreamVGGT, not from the proposed memory.
  2. The memory bank ablation suggests training variance may be as large as the reported effects. From the training data statistics in the Appendix (≈334K action samples over 10,819 R2R episodes), each episode averages about 31 steps, so with capacity 100 the bank is rarely full and frame merging is rarely triggered. Nb=100N_b = 100 and Nb=200N_b = 200 should behave almost the same, yet Table 4 reports 55.0 vs. 50.2 SR and 5.19 vs. 6.24 m NE. This 4.8-point gap most likely comes from training randomness. All main results come from a single run, and most claimed gains are in the 2–5 point range, including the +2.2 SR over JanusVLN.
  3. The SOTA claim relies on a narrow regime. The paper restricts comparison to methods using only official R2R/RxR trajectories. In the broader landscape, StreamVLN’s full configuration reaches 56.9 SR / 51.9 SPL on R2R, and JanusVLN with extra data reaches 60.5 SR on R2R and 56.2 SR on RxR, both above MemVLN’s 55.0 / 55.4. The paper’s Table 1 lists StreamVLN at 45.5 SR, which is a weaker configuration than the one reported in the original paper. Even within the narrow regime, StreamVLN without extra VLN data already reaches 52.8 SR on R2R with an older backbone; MemVLN’s 55.0 with the newer Qwen3-VL-4B plus StreamVGGT is a modest step. Combined with Table 3, where the geometry-only variant already matches the full model on RxR, most of the gain comes from the backbone and geometry encoder, not from the proposed memory. A clearer presentation would list both regimes side by side rather than restricting the SOTA claim to the narrow one.
Questions for authors
  1. Does the pose embedding act as near-perfect odometry in simulation? In Habitat, pose computed from discrete actions is nearly ground truth, yet the method is listed as single-RGB. Please provide: an ablation without the Fourier pose embedding; results with pose noise at the level measured on G1; and the dead-reckoned vs. true pose error on G1.
  2. Does the +5.3 SR gain from auxiliary supervision on R2R require the memory? The current Nb=0N_b = 0 control also masks the local tokens, so summarization becomes a pure language task. Please add a variant without memory in which summarization uses the unmasked local tokens. If this variant captures most of the gain, the auxiliary task mainly acts as regularization plus extra FGR2R supervision.
  3. At inference, does the model use memory information from beyond the local window? Please run two interventions: set memory tokens to zero, and replace them with memory from another episode. Report results on all episodes and separately on episodes longer than 16 steps (the local window). If performance barely changes, or drops only on short episodes, the memory functions as extra capacity overlapping with the local window, not as long-horizon episodic memory.
3 comments
AuthorsOpenReview ↗

Response to Reviewer LaHR

1. Memory, geometry and the backbone

Memory improves both datasets with Qwen3 fixed, and our Qwen2.5 baseline is already close to our Qwen3 baseline. These gains cannot be attributed solely to a stronger base model; they are additional gains over an already competitive SFT baseline.

StreamVGGT’s historical cache may overlap with explicit memory; R2R also has denser summary supervision and sub-trajectory targets absent from RxR. These factors may explain the smaller incremental gain on RxR. See General G1–G2 for the memory-use evidence and G3 for the backbone and baseline comparisons.

2. Bank capacity and training variation

Nb=100 and Nb=200 were each trained separately on joint R2R/RxR data. The 31-step R2R average does not describe the full trajectory-length distribution or closed-loop rollout lengths. Increasing capacity changes the history available to the same 300-token reader. The 4.8-point gap in Table 4 compares two capacity settings; it is not an estimate of seed variance.

3. Action-derived poses

Poses are integrated from issued actions, without ground-truth poses or external odometry, so collisions and imperfect execution can introduce discrepancies. Removing pose embeddings causes a modest drop, and No-pose remains above Zero-memory: memory still helps without explicit pose encoding. See General G2 for the comparison.

4. Visual input in the Nb=0 control

The Nb=0 control in Table 9 uses unmasked local visual tokens during summarization. It already implements the requested no-memory, visually grounded control. Full improves R2R SR by 4.1 points and SPL by 3.3 points over this control.

5. Memory beyond the local window

Earlier explicit history contributes to navigation: Recent16 lowers SR and SPL with the local RGB window unchanged. The larger degradation under Zero-memory also shows reliance on the memory readout. Memory gains increase with reference trajectory length. See General G1 for the length analysis and G2 for the interventions and subset protocol.

6. StreamVLN and JanusVLN comparisons

StreamVLN reports 45.5% R2R SR when trained on R2R/RxR trajectories alone (v1, Table 3). Its 52.8% result also uses DAgger, VideoQA and MMC4. The table below distinguishes these settings and those with extra navigation data.

Reported SR (%); StreamVLN v1 and JanusVLN v2.

MethodTraining configurationR2R SRRxR SR
StreamVLNOfficial trajectory stage45.5—
StreamVLN+ DAgger and VL data52.848.6
StreamVLN+ ScaleVLN; spatial pruning56.952.9
JanusVLNOfficial R2R/RxR trajectories52.851.4
JanusVLNWith extra navigation data60.556.2
MemVLNOfficial R2R/RxR trajectories55.055.4

MemVLN uses the official trajectories and summaries derived from them. Among models trained on these data alone, it achieves the highest SR on both datasets.

Thanks for the detailed rebuttal. Three concerns remain.

On W1. G1’s “+Memory” column matches Table 3’s “+mem” row (no geometry). Our concern was that once geometry is added, memory’s marginal contribution becomes negligible (Table 3: mem+geo 56.1 vs. geo-only 56.3 on RxR). The length-binned results do not include +geo vs. +geo+mem, so the central question is not addressed.

On W2. We did not claim the 4.8-point gap is seed variance; we claimed it suggests run-to-run variation may be comparable to the reported effects. Attributing the gap to capacity differences does not rule out this possibility. Seed-level results on a fixed capacity would be the most direct evidence.

On Q2. The rebuttal states the Nb=0 control uses unmasked local tokens, but §3.3 says “all local visual tokens H_t are masked” during summarization, with no exception. If Nb=0 is a special case, this changes the summarization objective, not just the presence of memory, and should be stated clearly.

I will keep my current rating. I am open to raising it if the revision adds (a) length-binned results with geometry held fixed, (b) seed-level variance for the main results, and (c) a precise description of the Nb=0 summarization objective.

Thank you for your follow-up. Our evaluation server was briefly offline for maintenance, which delayed our reply.

W1. Effect of memory across trajectory lengths

We compared the original Table 3 results for +Geometry and +Geometry+Memory on the same 3,669 episodes in the RxR unseen validation split. Both variants use StreamVGGT. We grouped episodes by the number of actions in the ground-truth trajectory, including turns and the terminal stop action.

Each metric is shown as +Geometry → +Geometry+Memory. Changes in success rate (ΔSR) are in percentage points and are calculated before rounding.

\small \begin{array}{ccccccc} \textbf{Actions} & \textbf{N} & \textbf{SR (}\mathbf{\\%}\textbf{)} & \textbf{SPL (}\mathbf{\\%}\textbf{)} & \textbf{nDTW (}\mathbf{\\%}\textbf{)} & \textbf{NE (m)} & \textbf{ΔSR (pp)} \\\\ \hline \le 16 & 138 & 91.30\to 91.30 & 88.90\to 91.15 & 87.84\to 87.18 & 0.93\to 1.01 & 0.00 \\\\ \text{17–48} & 783 & 71.26\to 69.48 & 68.27\to 67.58 & 75.02\to 75.12 & 2.56\to 2.52 & -1.79 \\\\ \text{49–80} & 918 & 62.20\to 60.57 & 58.20\to 56.42 & 68.93\to 69.12 & 4.17\to 4.38 & -1.63 \\\\ \text{81–120} & 933 & 50.59\to 49.95 & 41.67\to 40.81 & 57.99\to 58.47 & 6.89\to 6.76 & -0.64 \\\\ \mathbf{> 120} & 897 & \mathbf{37.79\to 40.91} & 28.61\to 30.33 & 46.00\to 46.97 & 9.92\to 9.35 & \mathbf{+3.12} \\\\ \hline \textbf{Overall} & 3{,}669 & \mathbf{56.31\to 56.12} & 49.69\to 49.40 & 62.55\to 62.96 & 5.87\to 5.74 & \mathbf{-0.19} \end{array}

For trajectories with more than 120 reference actions, adding memory resulted in 28 more successful episodes. In this group, SPL and nDTW also increased, and navigation error decreased. Overall SR was nearly unchanged.

The inference-time interventions in General G2 used the same full-model checkpoint, geometry configuration, and 200 RxR episodes. Setting the memory embeddings to zero reduced SR by 5.50 pp, and retaining only memory entries within the local RGB observation window reduced SR by 3.50 pp.

W2. Variation across training runs

With only two days available for training and evaluation, we did not have sufficient compute to conduct additional runs with different random seeds at a fixed memory-bank capacity.

Q2. Trajectory summarization in the zero-memory control

In the Nb=0N_b=0 control in Table 9, trajectory summarization uses unmasked local visual tokens with fused geometry features. The full model uses memory tokens for summarization and masks the local visual tokens. Both variants predict the same route instruction and use the same token-averaged cross-entropy loss.

We will clarify the inputs to the Nb=0N_b=0 control in Section 3.3 and the caption of Table 9.

Reviewer_e6mVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemVLN, a streaming episodic memory framework for VLN. It maintains a bounded memory bank of historical visual features with temporal and relative-pose information, and uses a cross-attention module to produce fixed-size memory tokens. The framework also incorporates a pretrained geometry encoder and an auxiliary trajectory summarization task to improve memory learning.

Critical strengths
  1. The proposed framework addresses an important problem in long-horizon VLN. The combination of compressed visual history, spatial information, and trajectory-level supervision is reasonable and can be integrated into existing VLM architectures.

  2. The paper evaluates the method on two standard VLN benchmarks and a physical robot. The component ablations and analyses across different instruction types provide useful insights into when navigation memory may help.

Critical weaknesses
  1. The contribution of episodic memory is not consistently demonstrated. Although the overall results are promising, the ablations raise questions about how much of the improvement actually comes from episodic memory. In Tab3, the geometry-only variant achieves better SR and SPL on RxR than the full model (56.3%/49.7% vs. 55.4%/48.4%). More importantly, Tab9 shows that removing memory from the full model reduces RxR SR by only 0.5 points, while SPL slightly improves. These results suggest that the benefits may come largely from the geometry encoder or auxiliary training rather than episodic memory itself. While memory provides clear gains on R2R, its contribution to RxR remains less convincing.

  2. The evaluation does not sufficiently establish long-horizon memory capabilities. The long-horizon analysis groups instructions mainly by punctuation counts and keyword matching, which are indirect measures of memory requirements. It remains unclear whether the model can reliably recall and use information observed much earlier in a trajectory. More targeted experiments involving revisiting locations, recalling previously observed landmarks, or navigating with long observation gaps would provide stronger evidence. Furthermore, the real-world evaluation is limited to six held-out instructions in a single scene, with a collision rate of 46.7%. These limitations make it difficult to assess the practical benefits and generalizability of the proposed memory mechanism.

Questions for authors
  1. Why does the geometry-only variant outperform the full model on RxR SR and SPL? Why does removing memory have such a small effect on RxR performance in Tab9?

  2. Can the authors conduct more targeted evaluations that explicitly require long-term memory, such as recalling previously visited locations or landmarks? A comparison with simpler history-retention methods under a matched token budget would help establish the benefit of the proposed memory design.

1 comment
AuthorsOpenReview ↗

Response to Reviewer e6mV

1. Memory and geometry on RxR

Memory improves the same baseline on both datasets, including RxR. StreamVGGT already retains history through its key/value cache, first-frame anchor and cached geometry features, so overlap with explicit memory may explain the smaller gain over Geometry-only.

R2R also provides denser memory supervision: 6.66 summary examples per 100 navigation examples, versus 1.97 for RxR. Its 11,456 sub-trajectory summaries provide finer-grained supervision that is absent from RxR. This training difference may contribute to the larger gains on R2R. See Table 3 for the component comparisons and General G1–G2 for the trajectory-length analysis and memory interventions.

2. Evidence of long-term memory use

The policy uses its memory readout and benefits from history beyond the local RGB window. Zero-memory degrades navigation, while Recent16 also reduces SR and SPL with the local window and memory-token count unchanged. Memory gains increase with reference trajectory length. See General G1–G2 for the comparisons and fixed-subset sampling protocol.

3. Token budget

Our component ablations keep the backbone and local RGB window fixed. Equalizing total tokens by changing the number of RGB observations would also change the visual information supplied. The interventions show the value of memory content at a fixed readout budget and local RGB-window size. See General G2 for the settings and comparisons.

4. Physical evaluation

Our physical evaluation combines qualitative G1 rollouts with repeated trials measuring both task success and collisions. Across six held-out instructions, each repeated five times, memory alone raises SR from 16.7% to 30.0%. Full reaches 40.0% SR and lowers the collision rate from 56.7% to 46.7% relative to the baseline. These results show that the navigation gains carry over to physical execution; collision reporting also reveals failures that SR alone would miss. Figure 5 and Appendix C.4 provide qualitative examples.

The evaluation covers one scene and six instructions, and both success and collision rates leave room for improvement. Our training scale is also limited: Full uses the official R2R/RxR trajectories, and post-training uses 24 separate real-world demonstrations, raising SR to 46.7%. Broader scene coverage and larger, more diverse training data are directions for improving physical performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their comments. Memory improves SR over the same baseline by 1.7 points on R2R and 4.5 on RxR (Table 3). We analyze its gains and use of earlier history.

G1. Memory gains on longer trajectories

We group all 3,669 English RxR val-unseen episodes by reference action count (forward/turn actions and final STOP), using identical groups for all models. Cutoffs 48/80/120 are the 25th/50th/75th percentiles; 16 is the local window size.

Table G1. SR (%) by reference action count; memory gain over Baseline (percentage points).

Reference actionsNumberBaseline+MemoryMemory gainFull
≤1613889.1388.41−0.7286.96
17–4878367.4367.05−0.3871.01
49–8091854.3656.54+2.1861.33
81–12093339.9847.59+7.6149.73
>12089729.9938.46+8.4736.90
All3,66948.8453.28+4.4455.44

Memory gains reach 7.61 and 8.47 SR points in the two longest groups, complementing our analysis by instruction clauses.

G2. Memory use, history and pose

All settings use the same Full checkpoint without retraining, the same local RGB window (current + up to 15 preceding frames), and 300 memory-token positions.

  • Normal: Original Full inference. FRME maintains at most 100 bank entries from accumulated episode history, which the reader accesses.

  • Recent16: From episode start, retain only entries within the local RGB window before each read, with no cross-frame merging. Older history cannot survive in merged entries; per-frame compression and retrieval are unchanged.

  • Zero-memory: Update and read the bank normally, then zero all 300 memory embeddings before insertion into the LLM, retaining their positions and local RGB input.

  • No-pose: Remove action-derived pose embeddings from stored features and retrieval queries. Keep visual features, timestep embeddings, consolidation and the 300-token readout.

We select 200 episodes without replacement using a fixed seed, proportionally across all 11 scenes (at least one per scene) and the five reference-action bins within each scene. Selection uses only scene and reference-trajectory metadata from the 3,669 English RxR val-unseen episodes; the list is fixed before intervention evaluation.

Table G2. Paired RxR200 results. SR, OSR, SPL and nDTW are percentages; NE is in metres.

ConditionSR ↑OSR ↑SPL ↑nDTW ↑NE ↓
Normal53.5059.5046.6563.575.37
Recent1650.0059.5044.3863.595.56
Zero-memory48.0059.5041.6160.546.19
No-pose51.5060.0045.5863.155.44

Recent16 lowers SR/SPL by 3.50/2.27 points. Zero-memory lowers SR/SPL/nDTW by 5.50/5.04/3.04 points and raises NE by 0.82 m. The policy uses memory, including history beyond the local window.

No-pose lowers SR/SPL by 2.00/1.07 points but remains 3.50 SR points above Zero-memory: memory helps without explicit pose encoding.

G3. Backbone and baseline performance

Table G3 compares published baselines and ablations under different training configurations. Our baseline is competitive before adding geometry or memory.

Table G3. Baseline and ablated configurations. Cells report SR / SPL (%).

Method / configurationBackboneR2R SR / SPLRxR SR / SPL
NaVILA — w/o Label BalancingVILA-8B30.0 / 25.1—
JanusVLN — w/o Dual Implicit MemoryQwen2.5-VL-7B24.8 / 16.8—
Efficient-VLN v1 — 2D tokens onlyQwen2.5-VL-3B42.3 / 38.4—
VLN-R1 — SFTQwen2-VL-7B24.9 / 17.519.5 / 16.7
ActiveVLN — IL / w/o RLQwen2.5-VL-3B38.5 / 33.741.0 / 34.1
MapNav — w/o MapLLaVA-OneVision-7B27.3 / 23.2—
Our baselineQwen2.5-VL-7B42.7 / 37.3—
Our baselineQwen3-VL-4B43.9 / 40.948.8 / 43.9

Our R2R baseline reaches 42.7 SR with Qwen2.5-VL-7B and 43.9 with Qwen3-VL-4B: just 1.2 points apart. High baseline performance is already present with Qwen2.5, reflecting our training configuration rather than simply a newer backbone.

All Table 3 ablations of the main draft keep Qwen3-VL-4B fixed. Memory raises SR from 43.9 to 45.6 on R2R and from 48.8 to 53.3 on RxR. Full reaches 55.0/55.4, gaining 11.1/6.6 points over our baseline. These gains cannot be attributed solely to a stronger base model. Improving on an already competitive SFT baseline strengthens the evidence for our method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.