- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MemVLN, an episodic memory-augmented framework designed for streaming visual input. The paper introduces a memory module comprising a Memory Bank and a Memory Perceiver. The Memory Bank continuously stores historical visual features, timestamps, and relative poses estimated based on executed actions, while utilizing token merging for spatial compression. When the memory bank exceeds its capacity limit, the system merges certain historical frames based on the average patch-level cosine similarity between adjacent frames. The experiments and analyses are comprehensive and support the potential of the framework, though I believe several details require further elaboration.
Critical strengths
This paper focuses on the problem of continuous historical memory in streaming VLN, addressing a critical bottleneck that current VLM-based navigation systems may face during long-horizon navigation. The modular design avoids directly concatenating all historical images into the language model's context and can, in principle, be integrated with existing streaming VLMs. Furthermore, by masking recent local visual tokens, the model is forced to reconstruct navigation instructions using memory tokens. This constraint is more direct in enforcing the retention and utilization of historical information compared to simply training the memory module passively via standard action prediction tasks. The experimental results and analyses are comprehensive and include real-world robot deployment tests.
Critical weaknesses
The main technical components of this paper including explicit historical feature storage, spatial-temporal positional encoding, and token merging are built upon well-established technical foundations. The authors need to clarify the following aspects:
- MemVLN does not achieve higher efficiency as claimed, and the module focuses on controlling context growth rather than building a more efficient model, and it may even incur additional inference overhead.
- Compared to existing explicit or implicit memory methods, what substantial innovations does this work introduce in terms of memory representation, writing mechanism, reading mechanism, and training objectives?
- It is necessary to verify whether the performance gains stem primarily from the new memory structure, or rather from the geometric encoder or the auxiliary supervision.
- The moudule of merging adjacent frames with the highest cosine similarity may lead to the loss of critical information, such as missing a crucial intersection in a hallway during a specific frame.
Questions for authors
In addition to the points raised in the weaknesses, I would like the authors to address the following questions:
- The authors should explain why increasing the memory bank capacity leads to performance degradation.
- How is the total loss for the navigation task and trajectory summarization task specifically defined? Does the auxiliary supervision genuinely improve the memory representation, or does it merely alter the output distribution of the language model?
- As cumulative errors accumulate during continuous navigation, do these errors directly affect the spatial encoding and historical localization within the memory?
Overall, the paper presents a reasonably sound framework, but several key issues remain. If the authors can adequately address my concerns, I am willing to raise my score.
3 comments
The authors’ response emphasizes bounded context, but bounded context does not necessarily imply higher inference efficiency. In fact, most existing VLN methods also employ bounded-context strategies. Therefore, I do not find this argument sufficient to support the claimed efficiency advantage of MemVLN. I am also not convinced by the authors’ characterization of MemVLN as an implicit memory method. In my view, it is more accurately described as a hybrid architecture that combines explicit feature storage with implicit latent-variable retrieval.
More importantly, the authors’ response still does not fully address the concern regarding the auxiliary supervision. The auxiliary task simultaneously updates both the Memory Reader and the LLM adapters. Therefore, it remains unclear whether the observed performance improvement actually results from a better memory representation, or is partly caused by changes in the output distribution or behavior of the language model induced by the auxiliary objective. I encourage the authors to provide more details about the auxiliary supervision.
Thank you for your follow-up comment.
Auxiliary supervision. Yes, the shared LLM adapters can contribute to the improvement. We deliberately train them alongside the Memory Reader so that the reader learns which route information to retrieve and the policy learns how to use it when choosing an action. Adding trajectory summarization improves navigation in both R2R comparisons.
| R2R val-unseen | SR without summarization | SR with summarization | Gain |
|---|---|---|---|
| Geometry only (Tables 3 and 9) | 49.2% | 50.9% | +1.7 pp |
| Geometry + memory (Table 3) | 49.7% | 55.0% | +5.3 pp |
The geometry-only policy gains 1.7 points from predicting the route instruction from unmasked local views, even though it has no reader. In the full model, we mask the local visual tokens during summarization, so the reader’s output is the only visual evidence for predicting the instruction. We attribute the 5.3-point gain to adding this supervision to the memory-equipped policy. We do not claim that the reader alone accounts for it.
Memory representation. We agree that MemVLN combines an explicit feature bank with a learned, implicit representation of route history.
Inference cost. We do not claim that MemVLN is faster than prior VLN methods. The reader supplies 300 memory tokens per decision regardless of trajectory length (§3.2). Table 8 reports mean policy latencies of 503.2 ms for Base and 691.0 ms for Full on the same RTX 4090.
Response to Reviewer ibTy
1. Novelty: memory representation and supervision
Our main contribution is a learned implicit memory representation for VLN, paired with task-specific supervision. Compressed frame features are stored in a bounded episode bank. Learned queries, grounded in the current timestep and action-derived pose, read this bank into a fixed set of latent memory tokens for the policy.
The distinction from retaining encoder KV caches is the dedicated readout and its training objective. Masking local visual tokens during whole-trajectory and sub-trajectory summarization requires route reconstruction from memory at multiple temporal scales. This learns navigation-relevant representations without prescribing a semantic or topological map. Table 3 and General G1–G2 show navigation gains from memory and benefits from earlier episode history.
2. Auxiliary supervision and the total loss
For navigation and trajectory summarization, we use the same cross-entropy loss, averaged over supervised output tokens:
L(x,y)=−∣y∣1∑tlogpθ(yt∣x,y<t).
Here x denotes the task-specific conditioning inputs: the task prompt, visual representations and memory tokens. The target y is the action sequence for navigation or the route instruction for summarization. Training processes each episode in temporal order, including its navigation and summary examples. Both tasks use the same loss, with no separate auxiliary-loss weight. Masking local visual tokens in summary examples directs reconstruction gradients through the memory reader and the LLM adapters. The drops under Zero-memory and Recent16 show that the trained policy uses its memory readout and earlier history; see General G2 for the comparisons.
3. Inference cost
The computational benefit is bounded context growth. The memory adds 300 tokens plus 7 prompt/special tokens, independent of trajectory length. Table 8 reports 503.2 ms for Baseline and 691.0 ms for Full on an RTX 4090; this 37.3% overhead includes both geometry and memory.
4. Frame merging and capacity
Merging similar frames can discard a brief but important cue, as noted in our limitations. Frame merging is one choice for maintaining the bounded bank. FIFO (first in, first out) is an alternative that removes the oldest frame entry when the bank exceeds capacity, retaining the most recent Nb frames without merging their features. A separately trained model using this FIFO policy gives the following results:
Separate training configurations with Nb=100 and Nm=300.
Task-aware retention and adaptive sampling are directions for improving consolidation.
When bank capacity increases, the reader still produces 300 tokens. More redundant or irrelevant entries may make retrieval and compression harder, which could explain the degradation in Table 4.
5. Accumulated pose error
Pose errors perturb the spatial embeddings used for retrieval. Visual features and timestamps remain available, and removing pose embeddings entirely causes a modest drop while still outperforming Zero-memory. Memory therefore remains useful without explicit pose encoding. See General G2 for the No-pose settings and results.