acceptodds
Under review as a conference paper at ICLR 2027

InfiniteVLN: Constant-Memory Streaming Vision-and-Language Navigation

Abstract

Current Vision-and-Language Navigation (VLN) models are constrained by growing context length and high computational cost in long-horizon scenarios, limiting their applicability to real-world robots that require continuous observation and low-latency decision making. In this paper, we propose InfiniteVLN, a novel streaming VLN framework that enables constant-memory navigation through a bounded, instruction-conditioned belief state. Specifically, we formalize VLN task as a Partially Observable Semi-Markov Decision Process (POSMDP) and decouple inference process into an observation step, which updates persistent visual memory, and a decision step, which generates actions based on the current memory state. To support efficient streaming inference, we introduce Asymmetric Context Caching, which stores the globally invariant instructions as a persistent instruction sink while preserving local spatiotemporal consistency within a constant-size KV space. Furthermore, we propose an Instruction-Gated Recurrent Memory module, which leverages the relative attention mass of visual tokens with respect to the instruction to construct a write-gating mechanism, and updates a long-term associative memory matrix via the Delta Rule. This design enables the model to selectively retain critical, instruction-relevant historical evidence. Our approach converts long-horizon VLN from full historical context modeling into instruction-conditioned evidence consolidation under a bounded state, which significantly reduces caching overhead and inference latency. Extensive experiments demonstrate that InfiniteVLN achieves superior performance, memory stability, and computational efficiency across both standard VLN benchmarks and challenging long-trajectory settings.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-and-Language Navigation, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MsagOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

InfiniteVLN aims to reduce memory usage and decision latency in streaming vision-language navigation. It separates observation updates from action decisions and uses a local visual cache together with a fixed-size recurrent memory. The memory is controlled by instruction attention. Its 2B model achieves 65.1% SR and 59.3% SPL on R2R-CE, with a peak GPU memory usage of 5.5 GB. On the constructed long-trajectory dataset, ACC+IGRM achieves 50.1% SR, compared to 41.6% with ACC alone. The method is well designed, but the long-trajectory evaluation protocol is not fully clear, and the real-world robot experiments are still limited.

Critical strengths

The main strength of InfiniteVLN is its focus on efficient streaming navigation. The method provides a practical way to control memory usage while maintaining strong navigation performance. I find the following two aspects particularly convincing.

1.InfiniteVLN has a strong performance in a low usage. It achieves competitive results on R2R-CE and RxR-CE among RGB-only VLM methods, with only 5.5 GB of peak GPU memory usage.

2.InfiniteVLN has a effective long-term memory design. When using IGRM, ACC achieves an SR of 50.1, compared to 41.6 without it, while keeping the same peak GPU memory usage. It also outperforms the full-attention baseline, showing the benefit of the proposed memory design.

Critical weaknesses

I think the main weakness of this paper is the lack of validation beyond standard benchmarks. The long-trajectory and robot experiments are still limited, making it difficult to assess the robustness and actual inference speed of the method in real-world settings.

1.The long-trajectory dataset lacks basic statistics, such as dataset size and trajectory lengths. The robot experiments also lack quantitative results on success rates and inference speed.

2.IGRM improves SR by 8.5 points, but the instruction-based gate only outperforms the best fixed gate by 0.9 points, without any uncertainty estimates. Also, comparing memory usage across models of different sizes makes the efficiency gains less convincing.

Questions for authors

I have a few questions that could help clarify my concerns and affect my overall assessment of the paper.

1.The long-trajectory evaluation lacks some important details. Could the authors provide more information to support the reliability of these results?

2.Although instruction-based gating improves performance, the gain over simpler methods is quite small. How much does instruction information actually contribute to the improvement?

3.More quantitative results from real-world robot experiments would make the evaluation more convincing. Is there any additional evidence on navigation success rates and inference efficiency?

6 comments

We thank the reviewer for the constructive feedback and for recognizing InfiniteVLN’s strong navigation performance and low memory usage. We address each concern below.

W1. I think the main weakness of this paper is the lack of validation beyond standard benchmarks. The long-trajectory and robot experiments are still limited, making it difficult to assess the robustness and actual inference speed of the method in real-world settings.

We conducted a total of 70 real-world trials across unseen complex indoor and dynamic outdoor environments, and validated our method on both the Unitree Go2 quadruped robot and the Unitree G1 humanoid robot. Inference ran locally on an NVIDIA Jetson Orin without a remote server, with a latency of approximately 1.2 s per step.

Quantitative results of real-world experiments.

SceneSuccessFailureSRLatency/ Step (s)
Indoor30585.7%1.2
Ourdoor28780.0%1.2

In indoor environments, the navigation distances ranged from 10 to 30 m, with a success rate of 85.7% (30 successful trials and 5 failures). The failure cases were primarily associated with highly reflective materials, such as glass and ceramic tiles. These materials can introduce unstable visual observations, increasing the risk of incorrect scene interpretation and navigation hallucinations.In outdoor environments, the maximum successful navigation distance reached approximately 60 meters, with a success rate of 80% (28 successful trials and 7 failures). Compared with indoor environments, outdoor navigation involves more complex environmental variations, among which illumination conditions are a major factor affecting navigation stability. For example, under strong sunlight, significant glare may appear in camera images, resulting in the loss of critical visual evidence and consequently affecting navigation decisions.

We believe that these failure cases reveal some of the remaining limitations of current sim-to-real generalization approaches. Specifically, training data collected in simulated environments cannot fully reproduce the complex illumination variations and material reflectance encountered in real-world environments.

W2. The long-trajectory dataset lacks basic statistics, such as dataset size and trajectory lengths. The robot experiments also lack quantitative results on success rates and inference speed.

The long-trajectory evaluation set contains 536 trajectories, with an average path length of 23 meters and a maximum path length of 37 meters. For concatenated trajectories, we first use the navigation mesh in Habitat-Sim to verify the traversability of the connecting segments. We then conduct manual verification to ensure that the merged instructions preserve the key landmarks, turning sequences, and goal constraints from both original trajectory segments. We also verify that the descriptions of the connecting segments are consistent with the corresponding reference paths.

Regarding the quantitative evaluation of real-world performance, we have provided detailed results on navigation success rates and inference latency in our response to W1.

W3. IGRM improves SR by 8.5 points, but the instruction-based gate only outperforms the best fixed gate by 0.9 points, without any uncertainty estimates. Also, comparing memory usage across models of different sizes makes the efficiency gains less convincing.

Thank you for pointing out that the performance gain of the instruction-conditioned gating mechanism over fixed gating strategies is relatively modest. We agree that it is important to distinguish between the overall contribution of the IGRM module and the incremental contribution of instruction information to the gating mechanism. To further isolate the contribution of instruction information to memory updates, we have conducted additional ablation experiments with different values of α. Specifically, when α = 0, the gating mechanism relies exclusively on the content representations of visual tokens. When α = 1, an instruction-attention term with a fixed weight is incorporated. The full RAG variant, in contrast, employs relative attention mass to achieve adaptive gating.

Ablation results of different gating configurations.

IGRMParamNE↓OS↑SR↑
×\times-5.6547.241.6
✓\checkmarkα=0\alpha=05.0354.749.1
✓\checkmarkα=1\alpha=15.2653.147.5
✓\checkmarkRAG4.9355.850.1

The results suggest that simply introducing a fixed-weight instruction-attention term does not consistently improve memory gating. In contrast, the full RAG mechanism achieves the best performance, indicating that adaptively incorporating instruction-related information through relative attention mass is more effective than using a fixed-weight formulation. Moreover, the overall 8.5-percentage-point improvement brought by IGRM cannot be attributed solely to the instruction-attention term. Instead, the primary benefit comes from the long-term recurrent memory and selective memory update mechanisms, while instruction attention provides additional task-relevance constraints that enable the model to selectively preserve historical visual evidence essential for navigation.

Regarding the efficiency comparison, the backbone models used in the existing comparisons are all based on the Qwen architecture, which helps mitigate the influence of architectural differences. Furthermore, we have added a comparison with StreamVLN-2B using Qwen3.5-2B as its backbone. Both methods are deployed on the same edge computing platform (Jetson Orin) and evaluated using FP16 inference precision. The results demonstrate that, with the same backbone model, InfiniteVLN consistently outperforms StreamVLN-2B in both navigation performance and inference efficiency. In contrast, StreamVLN-7B requires a remote server for inference and therefore cannot be deployed directly on the edge device.

Comparison of memory consumption and inference latency on the RxR dataset.

MethodNE↓OS↑SR↑Mem↓Latency/ Sample (s)↓
StreamVLN-7B6.2252.946.019G×\times
StreamVLN-2B7.5940.532.36.2G35.2
InfiniteVLN-2B4.3267.855.65.5G25.8

Q1. The long-trajectory evaluation lacks some important details. Could the authors provide more information to support the reliability of these results?

As detailed in our response to W2, the long-trajectory evaluation set contains 536 trajectories, with a mean path length of 23 m and a maximum of 37 m. We used Habitat-Sim’s navigation mesh to verify that the connecting segments were traversable and manually checked that the merged instructions preserved key landmarks, turn sequences, and goal constraints from both original segments and were consistent with the reference paths.

Q2. Although instruction-based gating improves performance, the gain over simpler methods is quite small. How much does instruction information actually contribute to the improvement?

We further isolate the actual contribution of instruction information through the newly added α ablation experiments. The results indicate that instruction information provides additional task-relevance constraints for long-term memory updates, thereby improving the selective retention of historical evidence. Although the incremental performance gain is relatively modest, it demonstrates the additional benefits of instruction-conditioned gating over strategies that rely solely on content representations or fixed memory-writing ratios. Meanwhile, we would like to emphasize that the primary contribution of IGRM comes from the long-term recurrent memory mechanism itself, which improves SR by 8.5 percentage points compared with the ACC-only baseline. Instruction information further enhances this memory update process by providing task-specific guidance for retaining navigation-relevant historical evidence.

Q3. More quantitative results from real-world robot experiments would make the evaluation more convincing. Is there any additional evidence on navigation success rates and inference efficiency?

As detailed in our response to W1, we conducted 70 real-world trials using the Unitree Go2 and G1. InfiniteVLN succeeded in 30 of 35 indoor trials (85.7%) and 28 of 35 outdoor trials (80.0%). Inference ran entirely onboard an NVIDIA Jetson Orin, with a latency of approximately 1.2 s per step and no reliance on a remote server.

Reviewer_ckPdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InfiniteVLN separates observation updates from action generation and combines a persistent instruction cache, sliding visual context, and instruction-gated recurrent memory. The framework targets bounded-memory navigation and is evaluated on standard VLN benchmarks, constructed long trajectories, and physical robots.

Critical strengths

Attractive reported accuracy and memory usage. The 2B model achieves SRs of 65.1 on R2R-CE and 67.8 on RxR-CE while reporting approximately 5.5 GB of peak GPU memory. This is a promising operating point for resource-constrained navigation.

The main components have useful ablation support. In Table 4, ACC reduces memory from 10 GB to 5.5 GB, while adding IGRM raises long-trajectory SR from 41.6 to 50.1 without increasing the reported memory footprint.

Bounded storage is supported by the architecture. A fixed local window and fixed-capacity recurrent state make memory independent of trajectory length for a fixed instruction and model configuration.

Physical deployment demonstrates practical feasibility. Experiments include both Unitree Go2 and G1 platforms, with reported observation updates at 2 FPS and approximately one-second action generation on an edge platform.

Critical weaknesses

The long-horizon evaluation is insufficiently characterized. Appendix E explains how two trajectories are concatenated, but does not report the resulting dataset size, length distribution, or instruction-validation procedure. Table 3 also omits path-fidelity metrics such as nDTW, making it difficult to determine whether the complete concatenated instruction is followed rather than only its final destination.

The specific value of instruction-derived gating is not isolated. Table 6 compares adaptive gating with constant gates, but not with the learned content-only gate obtained by setting the attention contribution to zero. The improvement over the constant 0.5 gate is only 0.9 SR points, without uncertainty estimates. This does not yet establish that instruction-attention mass is the decisive signal.

Efficiency comparisons mix architectural and backbone differences. Much of the headline memory advantage compares a 2B model with 7B–8B systems. Table 4 provides a useful internal control, but matched hardware, precision, input settings, and measured latency comparisons are still needed. Figure 4’s time axis also lacks physical units or observation counts.

Baseline cache configurations need clarification. Section 2.1 describes StreamVLN as using sliding-window mechanisms, whereas Appendix A and Figure 4 characterize it as retaining a complete, continuously growing cache. The actual evaluated configuration should be specified before attributing the memory gap to unbounded history retention.

Reproduction and real-world assessment remain incomplete. Streaming fine-tuning omits details such as segment length, state initialization between segments, and LoRA configuration. Physical experiments provide demonstrations and timing figures but no trial counts, success rates, or systematic failure analysis.

Questions for authors

What are the size, trajectory-length distribution, and instruction-quality checks for the long-horizon evaluation set? Can you report nDTW or intermediate-goal completion alongside final-goal SR?

Does the instruction-attention term outperform a learned content-only gate under identical training, and are the differences reliable across seeds? Were all ablation variants independently trained using the same schedule?

What cache policies, hardware, and numerical precision were used for the efficiency comparisons? Please provide latency and memory measurements against a matched Qwen3.5-2B baseline, together with quantitative real-world trial results.

4 comments

We thank the reviewer for the constructive feedback and for recognizing InfiniteVLN’s navigation accuracy with low, bounded memory use, the supporting ablations, and the feasibility of onboard deployment. We address each concern below.

W1. Appendix E lacks dataset statistics and instruction-validation procedures, while Table 3 omits path-fidelity metrics (e.g., nDTW) needed to assess complete instruction following.

The long-horizon validation set comprises 536 trajectories, with an average path length of 23 meters and a maximum length of 37 meters. For concatenated trajectories, we used the navigation mesh (NavMesh) in Habitat-Sim to verify the traversability of the connecting segments. We also conducted manual inspections to ensure that the merged instructions preserved the key landmarks, turning sequences, and goal constraints from both original segments, and that the descriptions of the connecting segments were consistent with the reference paths.

To assess trajectory fidelity beyond final-goal success, we additionally report normalized Dynamic Time Warping (nDTW):

Performance on long-horizon navigation.

MethodNE↓OS ↑SR ↑nDTW ↑Mem↓
DecoVLN6.2842.536.947.319G
InternVLA-N15.2953.645.355.721G
Ours4.9355.850.160.85.5G

These results provide further evidence of improved trajectory fidelity. We will include the dataset statistics, validation procedures, and nDTW results in the revised manuscript.

W2. Table 6 lacks a learned content-only gating baseline. The 0.9-point SR gain over constant gating, without uncertainty estimates, does not establish the value of instruction-derived attention.

We thank the reviewer for this valuable suggestion. We have conducted an experiment in which the contribution of the attention term is fixed to zero (α = 0) from the beginning of training. The results are presented in the table below. Under this setting, memory updates are governed solely by the content branch, making the model more likely to overlook critical landmarks. Furthermore, redundant memories may interfere with subsequent navigation decisions.

Ablation results of different gating configurations.

IGRMConfigurationNE↓OS↑SR↑
×\times-5.6547.241.6
✓\checkmarkα=0\alpha=05.0354.749.1
✓\checkmarkα=1\alpha=15.2653.147.5
✓\checkmarkRAG4.9355.850.1

The instruction-attention term substantially improves navigation performance, particularly in scenarios where critical evidence is sparse and subsequent observations contain considerable distractor information. These results demonstrate that incorporating instruction attention is more effective than using a content-only gating mechanism.

Furthermore, we also evaluated InfiniteVLN using different random seeds and observed an across-seed standard deviation of approximately 0.1 percentage points in SR, indicating that the method is stable across runs.

W3. Memory comparisons involve different model sizes (2B vs. 7B–8B), potentially confounding efficiency claims. Matched latency measurements are missing, and Figure 4 lacks explicit time units.

The backbone models used by the existing methods in our efficiency comparisons are all based on Qwen, which helps mitigate architectural differences to some extent. We have further added a comparison with StreamVLN-2B, which uses Qwen3.5-2B as its backbone. Both methods were evaluated on a Jetson Orin platform with FP16 inference precision.

Comparison of memory consumption and inference latency on the RxR dataset.

MethodNE↓OS↑SR↑Mem↓Latency/ Sample(s)↓
StreamVLN-7B6.2252.946.019G×\times
StreamVLN-2B7.5940.532.36.2G35.2
InfiniteVLN-2B4.3267.855.65.5G25.8

The experimental results demonstrate that, with the same backbone, InfiniteVLN consistently outperforms StreamVLN-2B in both navigation performance and inference efficiency. In contrast, StreamVLN-7B requires a remote server for inference and therefore cannot be deployed on edge devices. To provide a clearer comparison, we have also revised the x-axis of Figure 4 to explicitly represent the number of observations.

W4.StreamVLN is described as using sliding-window caching in Section 2.1 but an unbounded cache in Appendix A and Figure 4. This inconsistency raises concerns about the fairness of memory comparisons.

We thank the reviewer for pointing this out. Although StreamVLN employs a sliding-window mechanism, it still retains all historical frames and samples from them at each inference step. Consequently, its sliding-window mechanism keeps the number of input tokens fixed during inference. However, since all historical frames must be stored, its memory consumption grows linearly over time.

To ensure a fair comparison, all experiments were conducted under identical hardware and software environments, with each method using the default cache configurations specified in their respective papers. We will clarify this implementation detail in the revised manuscript.

W5. Streaming fine-tuning omits segment length, state initialization, and LoRA configurations. Real-world experiments lack trial counts, success rates, and systematic failure analysis.

We set the LoRA rank to 8, the scaling factor to 16, and the dropout rate to 0, while keeping the pretrained backbone weights frozen. Training stagen is employed a learning rate of 1 × 10⁻⁴, a batch size of 128, and 3 training epochs, with a warmup ratio of 0.1 and BF16 precision. We have added these implementation details to the revised manuscript.

Each trajectory is chronologically divided into contiguous segments, with each segment containing at most 224 frames. At the beginning of each trajectory, the instruction cache, local KV cache, recurrent state, and convolutional state are initialized. Consecutive segments from the same trajectory inherit the numerical values of these states, while gradients are truncated only at segment boundaries. Within each segment, the full computational graph is retained.

During training, observation steps update the persistent states in a differentiable manner, whereas decision steps employ teacher forcing to compute the cross-entropy loss over the expert action tokens. We have supplemented the revised manuscript with these details to clarify the training procedure and state initialization strategy.

Q1. What are the size, trajectory-length distribution, and instruction-quality checks for the long-horizon evaluation set? Can you report nDTW or intermediate-goal completion alongside final-goal SR?

As detailed in our response to W1, the long-horizon evaluation set contains 536 trajectories with a mean path length of 23 m and a maximum of 37 m. We verified the traversability of the connecting segments using Habitat-Sim’s navigation mesh and manually checked that the merged instructions preserved key route details from both original segments and aligned with the reference paths. To assess path fidelity beyond final-goal success, we report an nDTW of 60.8 alongside an SR of 50.1%.

Q2. Does the instruction-attention term outperform a learned content-only gate under identical training, and are the differences reliable across seeds? Were all ablation variants independently trained using the same schedule?

As detailed in our response to W2, the full RAG variant achieves an SR of 50.1%, compared with 49.1% for the learned content-only gate ((\alpha=0)), a gain of 1.0 percentage point. Across the tested seeds, the full model’s SR standard deviation was approximately 0.1 percentage points, indicating stable performance across runs.

Q3. What cache policies, hardware, and numerical precision were used for the efficiency comparisons? Please provide latency and memory measurements against a matched Qwen3.5-2B baseline, together with quantitative real-world trial results.

We thank the reviewer for this careful question. For efficiency comparisons, many existing VLN methods adopt a global historical-frame caching strategy and sample a fixed set of frames at each inference step. For example, StreamVLN combines uniformly spaced sampling with a sliding-window strategy. Consequently, although the number of input tokens remains fixed at each inference step, GPU memory consumption grows linearly with the navigation trajectory length because all historical frames must be retained.

In contrast, InfiniteVLN adopts a streaming cache strategy, allowing its memory footprint to remain constant over time. Since 7B models cannot be deployed on edge devices, the efficiency comparisons involving these models were conducted on an NVIDIA RTX 4090 GPU, using identical software versions and FP16 inference precision. As detailed in our response to W3, we have also provided latency and GPU memory comparisons using Qwen3.5-2B as the backbone, further demonstrating the efficiency advantages of InfiniteVLN.

In real-world experiments, we conducted a total of 70 real-world trials across unseen complex indoor and dynamic outdoor environments, and validated our method on both the Unitree Go2 quadruped robot and the Unitree G1 humanoid robot. Inference ran locally on an NVIDIA Jetson Orin without a remote server, with a latency of approximately 1.2 s per step.

Quantitative results of real-world experiments.

SceneSuccessFailureSRLatency/ Step(s)
Indoor30585.7%1.2
Ourdoor28780.0%1.2

In indoor environments, the navigation distances ranged from 10 to 30 m, with a success rate of 85.7% (30 successful trials and 5 failures). The failure cases were primarily associated with highly reflective materials, such as glass and ceramic tiles. These materials can introduce unstable visual observations, increasing the risk of incorrect scene interpretation and navigation hallucinations.In outdoor environments, the maximum successful navigation distance reached approximately 60 meters, with a success rate of 80% (28 successful trials and 7 failures). Compared with indoor environments, outdoor navigation involves more complex environmental variations, among which illumination conditions are a major factor affecting navigation stability. For example, under strong sunlight, significant glare may appear in camera images, resulting in the loss of critical visual evidence and consequently affecting navigation decisions.

Dear Reviewer ckPd,

We sincerely thank you for your time in reviewing our paper and your constructive comments. As the discussion period draws to a close, we welcome any further questions or comments you may have. If any points remain unclear or require additional clarification, please let us know. We would be glad to address them and appreciate your consideration.

Best regards, Authors

Reviewer_tDjPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a streaming vision-and-language navigation framework that maintains bounded memory. It separates observation-driven memory updates from action generation and combines two key techniques, including Asymmetric Context Caching (ACC), which preserves the instruction alongside a sliding window of recent observations, and Instruction-Gated Recurrent Memory (IGRM), which uses instruction-attention signals to regulate the consolidation of visual information into a fixed-size recurrent state. Experiments on standard and extended-trajectory navigation benchmarks report competitive or improved performance with reduced memory usage, complemented by real-world robotic demonstrations.

Critical strengths
  • Combining a local observation window with recurrent memory provides a sensible way to preserve detailed recent context while retaining compressed historical evidence.
  • The idea of compressing long-term streaming visual information into bounded memory representations is compelling in long-horizon navigation.
  • The instruction-gated memory update mechanism is intuitive, providing a task-conditioned approach to retaining potentially relevant historical information.
Critical weaknesses
  • The motivation places undue emphasis on temporal asymmetry. The distinction between an instruction that remains fixed throughout an episode and visual observations that evolve over time is an inherent and straightforward characteristic of VLN. The paper repeatedly emphasizes this distinction as a central insight, but its significance for the proposed method is overstated. The motivation would be more compelling if it focused on the challenges of identifying and retaining task-relevant historical evidence under a bounded memory budget, rather than extensively framing the method around this observation.
  • The presentation is difficult to follow, partly because some terminology is unclear. For example, “instruction sink” appears to refer to the KV representations of a single instruction retained throughout the navigation episode. It is unclear whether “sink” denotes any mechanism beyond a persistent instruction cache or a retained instruction prefix. The authors should clarify this distinction and use more direct terminology where no additional mechanism is intended.
  • The methodological novelty appears limited. The persistence of an instruction throughout a navigation episode is an inherent task characteristic, and retaining its representations while maintaining a sliding observation window is a relatively straightforward design choice. Although ACC also incorporates positional reconstruction, the paper should more clearly distinguish its technical contribution from existing caching mechanisms and establish the benefits of its individual components.
  • The computational efficiency requires more detailed evaluation. The adaptive recurrent gate involves per-token computation, but its incremental overhead is not quantified. The reported observation-update rate of 2 FPS and approximately one second to generate an action sequence also raise questions about responsiveness in real-world navigation. These measurements alone do not establish which components dominate runtime. A stage-wise latency breakdown would help substantiate the efficiency claims.
Questions for authors
  • Is ACC necessary when IGRM is available? Table 4 evaluates ACC alone and ACC combined with IGRM, but does not establish whether a historical visual KV window is necessary alongside recurrent memory. How would a variant using only compact recurrent memory for visual history perform, while retaining instruction conditioning and access to the current observation? Since the proposed gate uses attention statistics from ACC layers, please clarify how instruction conditioning could be preserved in such a comparison.
  • Relevant to the previous question: how are the sliding-window size and frame-sampling policy determined? Appendix C specifies a window size of 8192 for first-stage training, but the settings used during streaming training and evaluation remain unclear. Please specify the unit of window capacity, eviction stride, observation-sampling frequency, and whether these settings differ across benchmarks and real-world deployment. How sensitive is performance to window size and sampling frequency?
  • How is the initial recurrent memory state (S) obtained before processing the first visual token? Is it zero-initialized, learned, or initialized by encoding the instruction? Equations (9)–(10) define subsequent updates but do not explicitly specify this initial condition.
4 comments

We thank the reviewer for the constructive feedback and for recognizing the complementary roles of the local observation window and instruction-gated recurrent memory in bounded-memory navigation. We address each concern below.

W1. The paper overemphasizes temporal asymmetry between static instructions and evolving observations, which is an inherent characteristic of VLN. The motivation should instead focus on preserving task-relevant historical evidence under a bounded memory budget.

Thank you for this valuable suggestion. We will revise the manuscript to explicitly formulate the central challenge as follows: under a bounded memroy buget, what historical evidence should be preserved, in what form should it be stored, and how can this evidence be protected from being overwritten by subsequent observations and transient reasoning.

InfiniteVLN addresses these challenges through two complementary mechanisms: (1) Unlike conventional VLM-based methods, which update their caches only during autoregressive generation, InfiniteVLN decouples observation and decision processes. The observation step updates only the persistent global state, whereas the decision step generates actions based on the current state. Meanwhile, the ACC module prevents reasoning tokens generated during decision-making from being written into the global state as new environmental evidence. (2) IGRM explicitly incorporates instruction relevance into the recurrent-state update process, increasing the likelihood that task-relevant evidence is retained while mitigating persistent interference from redundant information in the finite-capacity state.

We will clarify that temporal asymmetry serves as a design motivation, while the primary contribution lies in instruction-conditioned evidence consolidation under bounded memory.

W2. The term “instruction sink” is insufficiently defined and appears equivalent to a persistent instruction KV cache. The authors should clarify whether it involves any additional mechanism and simplify the terminology accordingly.

Thank you for this helpful suggestions. The instruction sink essentially refers to the persistent KV representation of the navigation instruction within the attention layers. The original terminology was inspired by the attention-sink phenomenon observed in existing vlm studies[1], highlighting the fact that the instruction cache receives a relatively large amount of attention during memory updates. To make it more clear, we have replaced instruction sink with persistent instruction KV cache throughout the revised manuscript.

We have also carefully refined the manuscript to identify potentially confusing terminology and revised or supplemented the corresponding explanations to improve clarity.

[1] Efficient Streaming Language Models with Attention Sinks, ICLR2024

W3. Retaining instruction representations alongside a sliding observation window is a relatively straightforward caching strategy. The authors should better differentiate ACC from existing methods and quantify the contributions of its individual components.

The technical contribution of ACC lies in adapting bounded KV caching to streaming multimodal navigation while preserving spatiotemporal consistency and supporting instruction-conditioned memory updates.

Specifically, ACC serves three complementary purposes: (1) Persistent task conditioning: Instruction KV representations remain accessible throughout navigation, while recent visual KV entries preserve fine-grained local evidence. (2) Position-reconstructable caching: ACC stores pre-rotary keys and dynamically reconstructs Partial-MRoPE when the window shifts, preserving intra-frame spatial coordinates while re-indexing temporal positions. This enables consistent KV reuse without re-encoding historical images. (3) Integration with recurrent memory: ACC provides local contextual evidence and instruction-attention signals for IGRM, which selectively consolidates historical information beyond the sliding window into a fixed-capacity recurrent state.

In Table 5 of the paper, ACC improves success rate from 31.2% to 41.6% over standard sliding-window attention, with both variants reporting 5.5 GB of peak GPU memory. In Table 4, adding IGRM to ACC further raises success rate to 50.1% (+8.5%) at 5.5 GB, compared with 43.8% success and 10 GB for full attention.

Unlike StreamVLN, which employs a recent dialogue window together with historical token pruning based on 3D information, InfiniteVLN integrates historical evidence outside the sliding window into a fixed-capacity recurrent state while explicitly decoupling the observation updates from the generation of transient decision states. The key technical contribution lies in the complementary collaboration of ACC and IGRM. ACC provides persistent task conditioning, recent fine-grained context, and a protected persistent state. Building upon this interface, IGRM utilizes instruction-relevance signals to regulate observation-induced recurrent-state transitions.

We will revise the manuscript to clarify these technical distinctions and more explicitly distinguish the individual roles of ACC and IGRM.

W4. The overhead of the adaptive recurrent gate remains unquantified, while the reported 2 FPS observation-update rate and approximately one-second action generation raise concerns about real-time responsiveness. A stage-wise latency breakdown is needed to substantiate the efficiency claims.

Thank you for this valuable suggestion. We compared the decision-step inference latency of InfiniteVLN and its backbone model, Qwen3.5, under identical hardware and environmental conditions.

Inference latency comparison between InfiniteVLN and its backbone model.

ModelPrefill Runtime(s)/StepDecode Runtime(s)/StepTotal stepsTotal latency(s)
Qwen3.50.60.53235.2
InfiniteVLN(Obs)0.60.02112.6
InfiniteVLN(Dec)0.70.51113.2

The adaptive recurrent gate can leverage the parallel computation strategy of GDN to achieve low-latency inference. Compared with the original backbone, InfiniteVLN introduces only approximately 0.1s of additional latency during the prefill stage. The observation step, which updates only the global state, takes approximately 0.5s. The combined latency of the prefill and decoding stages during the decision step is approximately 1.2s. Due to operator fusion and CUDA-based parallel acceleration, it is not possible to precisely isolate the inference latency of individual layers within the model.

Nevertheless, it is worth noting that, owing to the decoupling of observation and decision processes, InfiniteVLN achieves a total inference latency of only approximately 25.8s under identical input conditions, which is roughly two-thirds of that of the original Qwen3.5 backbone (35.2s). Furthermore, the latency of the observation step can be further overlapped with the robot's physical movement, resulting in an approximately 1.5× improvement in execution efficiency during real-world deployment compared with the original model. A more detailed explanations are provided in Appendix A. The supplementary materials also include deployment videos in real world, demonstrating the system on multiple robotic platforms across various environments.

We will include this latency breakdown and additional implementation details in the revised manuscript.

Q1. How would a variant using only recurrent memory for visual history perform while retaining instruction conditioning and the current observation? Given the gate's reliance on ACC attention statistics, how would instruction conditioning be maintained in this variant?

Thank you for raising this important question. In our framework, ACC not only provides a local visual cache, but also facilitates the decoupling of observation and decision processes by isolating the persistent global observation state from the transient decision state. This state isolation is a necessary prerequisite for the IGRM, as its recurrent state is updated based on the tokens processed by the corresponding layer. If the observation and decision processes shared the same writable persistent state, reasoning tokens generated during each decision step would also modify the long-term memory.

As the number of decision steps increases, the persistent state would gradually accumulate a mixture of environmental evidence and repeated language prompts, progressively consuming its limited capacity with information unrelated to new environmental observations. Such interference could compromise the retention of critical visual evidence acquired earlier in the navigation process. Therefore, the state isolation supported by ACC is essential in preventing decision-induced memory contamination and ensuring that IGRM functions as intended.

The visual window is retained to provide direct access to recent fine-grained visual evidence, while the recurrent state consolidates longer-term historical information. This design reduces the burden on the fixed-capacity recurrent state to preserve fine-grained visual details.

Q2.How are the sliding-window size and frame-sampling policy determined?

Since InfiniteVLN targets on resource-constrained edge platforms, such as NVIDIA Jetson Orin, the sliding-window capacity and observation frequency selected to balance navigation performance, inference latency, and computational resource constraints. An observation frequency of 2 Hz enables continuous perception for both quadrupedal and humanoid robots. We use the same configuration in both benchmark evaluations and real-world deployments.

The sliding window adopts a chronological First-In, First-Out (FIFO) eviction strategy. When sufficient window capacity is available, new observations are appended directly. Otherwise, only the oldest tokens necessary to accommodate the incoming observations within the predefined capacity (W). Thus, the stride in Eq. (5) of main paper represents the actual number of tokens evicted at each update, rather than a fixed window-shift constant.

We further evaluate the sensitivity to window size in the Table:

Ablation results of different window size.

Window Size (W)NE↓OS ↑SR↑Mem↓Latency ↓
40965.3653.746.55.3G1.0s
81924.9355.850.15.5G1.2s
122884.9356.050.25.7G1.5s

Our experiments show that increasing the window size can further improve the success rate, although the additional gains are relatively limited. Reducing the window size to 4096, however, decreases the amount of recent visual evidence directly accessible to the model, resulting in a lower success rate.

Q3. How is the initial recurrent memory state SS initialized?

Thank you for this careful question. At the beginning of each episode, the recurrent state of each GDN layer is initialized to zero before processing any input tokens:

Sempty(ℓ)=0.S_{\mathrm{empty}}^{(\ell)} = \mathbf{0}.

This initial state is neither learnable nor generated by an additional network. The navigation instruction undergoes a standard prefill forward pass. This process simultaneously generates the persistent instruction KV cache and obtains the instruction-conditioned recurrent state according to the backbone's recurrent update rule:

Finst(ℓ)(I;Sempty(ℓ)).F_{\mathrm{inst}}^{(\ell)}\bigl(\mathcal{I}; S_{\mathrm{empty}}^{(\ell)}\bigr).

Consequently, the state used to process the first visual token by the first visualn is: S1,0(ℓ)=S0(ℓ).S_{1,0}^{(\ell)} = S_{0}^{(\ell)}.

Thus, zero initialization refers to the original recurrent state before processing any input, whereas instruction-conditioned initialization refers to the state obtained after the instruction prefill. The instruction prefill is performed only once at the beginning of each episode. All subsequent observation updates inherit and continue updating the existing recurrent state.

We have added these implementation details to the revised manuscript to clarify the initialization procedure.

Dear Reviewer tDjP,

We sincerely thank you for your time in reviewing our paper and your constructive comments. As the discussion period draws to a close, we welcome any further questions or comments you may have. If any points remain unclear or require additional clarification, please let us know. We would be glad to address them and appreciate your consideration.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.