acceptodds
Withdrawn from ICLR 2027

NavTTT: Adaptive Parametric Memory via Hierarchical Test-Time Training for Vision-and-Language Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Vision-and-language navigation (VLN) demands that agents accumulate and reason over streaming visual observations throughout long navigation episodes. Existing approaches struggle to maintain long-term history, constrained by the finite context capacity of transformer architectures. To solve the problem, fast-weight test-time training (TTT) provides a mechanism for encoding streaming observations into an evolving parameter state. However, existing TTT does not distinguish informative observations from repetitive visual content, allowing redundant tokens to interfere with the retention of information needed for subsequent decisions. We propose Navigation-Conditioned Test-Time Training (NavTTT), which introduces adaptive memory formation and hierarchical memory integration for vision-and-language navigation. NavTTT leverages global navigation context to prioritize informative visual evidence and integrates language instructions with spatial information to construct spatially grounded memory representations. Episode-local nonlinear fast weights store these representations as Adaptive Parametric Memory (APM), preserving history beyond the visual context for later decisions. To reduce cross-layer coupling in memory formation, NavTTT maintains independent fast-weight memories across multiple decoder depths. This decoupled design prevents errors or biases from spreading across different memory levels. Meanwhile, it enables shared historical information to be independently represented at various levels of abstraction. Experiments demonstrate strong navigation performance on VLN-CE benchmarks and effective instruction following on a real robot.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Language Navigation, Test Time Training, Embodied AI, Mobile Robot

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EDpsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

NavTTT introduces episode-specific nonlinear fast-weight memory into vision-language navigation policies and it chooses visual patches based on the navigation context and diversity within each frame, and then it combines them with spatial and task information to update separate memory modules at 6 decoder layers. Besides, it achieves success rates of 66.7% and 63.6% on the unseen validation sets of R2R-CE and RxR-CE, and it achieves SPL scores of 59.4% and 52.7% on them too. The authors also conduct ablation studies on the memory design and update strategies under a separate LoRA setting, which I think helps clarify the role of the memory mechanism. But, the real-world results are less convincing, with only 11 successful navigation trials out of 40, suggesting that there is still considerable room for improvement in practical deployment.

Critical strengths

In my opinion, I think this paper addresses an important problem in vision-language navigation. I think using fast-weight memory to preserve useful visual information is an interesting idea, and the overall design makes sense. Besides, the experimental results are also good, with improvements on both unseen validation sets.

1.The memory design is well motivated. The paper focuses on how to preserve useful visual information from earlier observations when those frames are no longer available in the current context. I think updating memory during navigation and retrieving it later is a reasonable way to solve this problem. This makes the proposed method an effective way to handle long-term visual information in VLN.

2.The experimental results are so good. NavTTT has good performs on on both unseen validation benchmarks and the ablation results are also helpful. I think these results support the effectiveness of the proposed memory mechanism, but I think the gains still need to be verified under the main training setting.

Critical weaknesses

The main concern is whether the gains come from APM itself and whether the added hierarchy is worth its cost.

1.The main comparison is not under the same conditions. NavTTT uses a 8B backbone and additional DAgger data, while the no-memory results come from a different LoRA setting. And Table 1 appears to show that NavTTT uses panoramic input, but Sec. 3.1 describes a different input setting. I think the authors should clarify this. I would like to know how much of the performance improvement comes from APM.

2.The paper does not compare the six independent memory modules with simpler designs, such as shared memory or memory at fewer layers. It also does not report memory update latency or performance across different route lengths. Therefore, I think the benefits of the hierarchical memory design and its efficiency in long-horizon navigation have not been fully evaluated.

Questions for authors

I have a few questions about the experimental settings and ablation studies. I think the answers would help me better understand the contribution of the proposed method and could affect my overall assessment.

1.Table 1 seems to indicate that NavTTT uses both panoramic and single-RGB inputs, but Sec. 3.1 only mentions an egocentric RGB image. Could the authors clarify which input setting is actually used?

2.I want to know how much of the performance improvement comes from APM. Can authors provide to me more experimental results about the ablation experiments,such as experimental results without APM and with a static APM, while keeping all other settings the same as the main model?

3.Why does the method use six independent memory modules? I would also like to see separate ablation studies on navigation relevance scoring and within-frame diversity.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer EDps

We sincerely thank the reviewer for your careful reading and positive assessment of our motivation and results. To address your presentation feedback, we have thoroughly polished the manuscript to ensure complete clarity and consistency.

[W1 & Q1] Observation Inputs

Thank you for raising this point. NavTTT uses a single forward-facing egocentric RGB camera for all visual observations. Each decision uses the current image and up to seven earlier frames sampled from that same camera, along with episode-relative planar position and heading. In the NavTTT row of Table 1, the two checkmarks align with S.RGB (single RGB stream) and Odo. (relative pose). The Pano. entry is blank because the evaluation uses a non-panoramic view. We will make this explicit in the table caption.

[W1 & Q2] APM Under the Main Training Protocol

We trained the no-APM baseline, direct LaCT adaptation, and full NavTTT under the main joint-training protocol. Within each backbone, we matched initialization, observation inputs, training data including DAgger, and optimization budget. Static APM uses the full checkpoint with its learned initial fast state held fixed at evaluation.

On R2R-CE with Qwen3-VL-8B, full NavTTT improves SR and SPL over the no-APM baseline by 4.3 and 2.9 percentage points. Oracle success rises from 68.7% to 73.1%, indicating that more trajectories reach the goal neighborhood. With the checkpoint held fixed, online updates increase successful episodes from 1,165 to 1,226 out of 1,839, a gain of 3.32 SR points, and improve SPL by 2.6 points. Full NavTTT also improves both metrics over static APM in the other three settings.

Table A. Controlled memory ablations on Val-Unseen

BackboneVariantR2R-CE SR / SPL (%)RxR-CE SR / SPL (%)
Qwen3-VL-8BNo APM62.4 / 56.560.2 / 52.9
Qwen3-VL-8BStatic APM63.3 / 56.861.3 / 52.5
Qwen3-VL-8BDirect LaCT63.3 / 56.462.3 / 53.7
Qwen3-VL-8BFull NavTTT66.7 / 59.463.6 / 52.7
Qwen2.5-VL-7B-InstructNo APM53.6 / 48.650.7 / 43.0
Qwen2.5-VL-7B-InstructStatic APM56.4 / 50.352.1 / 43.3
Qwen2.5-VL-7B-InstructDirect LaCT55.7 / 49.751.6 / 42.1
Qwen2.5-VL-7B-InstructFull NavTTT57.3 / 51.053.4 / 44.8

[W2 & Q3] Memory Layout and Token Selection

We place APM branches at the ends of equal segments of the 36-layer decoder. The one-, three-, and six-branch layouts use zero-based indices {35}, {11, 23, 35}, and {5, 11, 17, 23, 29, 35}, respectively. The shared layout ties corresponding fast states across the same six layers. All layouts use matched training and writing settings with online updates enabled.

At equal fast-state storage, six shared branches improve SR and SPL by 0.6 percentage points each over the last-layer branch. We chose six independent branches for their highest observed SR (66.7%), with comparable SPL and 2.19 ms higher latency per decision than six shared branches.

Table B. Memory configurations on R2R-CE Val-Unseen

ConfigurationSR (%)SPL (%)Fast-state storage (MiB)Decision latency (ms)
Six layers, independent66.759.4576.28347.51
Last layer only64.758.996.05313.37
Three layers, independent65.159.2288.14327.81
Six layers, shared65.359.596.05345.32

At a fixed write budget, relevance and diversity alone improve SR over random selection by 1.3 and 0.7 points on R2R-CE and 2.2 and 1.3 points on RxR-CE, respectively. Combining them yields the highest SR and SPL on both benchmarks.

Table C. Visual token selection on Val-Unseen

SelectionR2R-CE SR / SPL (%)RxR-CE SR / SPL (%)
Random62.8 / 56.860.5 / 49.6
Relevance only64.1 / 58.362.7 / 51.8
Diversity only63.5 / 57.361.8 / 50.7
Combined66.7 / 59.463.6 / 52.7

[W2] Inference Cost

We profiled all variants on one RTX 5090 D v2 using the same 128 R2R-CE Val-Unseen inputs, batch size 1, and four generated tokens per decision. Writing takes 14.28 ms per decision. Full NavTTT’s total decision latency is 13.13 ms (4.12%) higher than that of the profiled static configuration.

Table E. Inference cost on fixed R2R-CE inputs

VariantDecision (ms)Writing (ms)Peak allocated GPU memory (GiB)
No APM288.69—16.82
Static APM318.82—17.62
Full NavTTT331.9514.2818.17
AuthorsOpenReview ↗

Reply to Reviewer EDps

[W2] Performance by Route Length

We group episodes by tertiles of cumulative reference trajectory length, using the same groups for all variants within each benchmark. The cutoffs are about 7.8 and 10.0 m on R2R-CE and 9.5 and 17.7 m on RxR-CE.

Table F. Navigation by reference route length

DatasetGroupEpisodesFull NavTTTNo APMStatic APMDirect LaCT
R2R-CEShort61574.96 / 65.8971.54 / 63.6974.47 / 64.3973.66 / 63.15
R2R-CEMedium61271.08 / 63.3267.65 / 60.1471.57 / 62.0868.63 / 61.49
R2R-CELong61253.92 / 48.8447.88 / 45.7443.95 / 43.8947.55 / 44.53
RxR-CEShort1,22479.33 / 70.3477.78 / 70.5378.43 / 70.8178.84 / 69.44
RxR-CEMedium1,22464.95 / 51.2563.89 / 55.6061.52 / 52.5862.09 / 53.71
RxR-CELong1,22146.44 / 36.4738.82 / 32.4343.98 / 34.2146.03 / 37.91

*Entries are SR / SPL (%) for Qwen3-VL-8B.

Full NavTTT’s SR gain over no APM is largest on long routes in both benchmarks. On long R2R-CE routes, it improves SR and SPL over static APM by 9.97 and 4.95 points, respectively. This group accounts for all 61 net additional successes over static APM. On RxR-CE, the SR gains over static APM are 3.43 points on medium routes and 2.46 points on long routes.

NavTTT brings instruction-conditioned writing, spatial encoding, and retrieval at multiple decoder depths to fast-weight memory for navigation. It stores relevant, diverse visual cues with their spatial and task context, giving the decoder access to accumulated history during later decisions. Under matched training, NavTTT achieves higher SR and lower NE than both the no-APM baseline and direct LaCT on both benchmarks with each tested backbone. With Qwen3-VL-8B, its SR gains over no APM are largest on long routes in both benchmarks.

AuthorsOpenReview ↗

Reply to Reviewer EDps

Dear Reviewer EDps,

Thank you for your careful review. Your comments helped us clarify the input setting and test APM under the main training protocol. In our response, we explain the single-camera RGB and relative-pose inputs, report matched comparisons with no APM and static APM, and examine how memory layout, token selection, and online updates affect navigation across route lengths.

If you have had a chance to review our response, could you let us know whether any concerns remain? A brief reply before the discussion closes would give us time to address any remaining questions promptly.

Best regards,
The authors

Reviewer_HyiuOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents NavTTT, a test-time training framework that maintains navigation history through hierarchical parametric memory. It selects relevant visual tokens, encodes them with spatial and navigation context, and updates fast weights to support subsequent retrieval. Experiments on R2R-CE and RxR-CE demonstrate improved navigation performance, with additional ablations and real-world evaluations.

Critical strengths
  • The paper proposes a structured parametric memory framework for VLN, combining context-aware visual token selection, fast-weight memory updates, and hierarchical retrieval.
  • Experiments on R2R-CE and RxR-CE demonstrate consistent navigation improvements, supported by ablations of key design components and real-world robot evaluations.
Critical weaknesses
  • The rationale for the selected memory layers and their independent parameterization is insufficiently established.
  • The paper also lacks direct evaluations of historical information retrieval accuracy and detailed measurements of test-time computational overhead.
Questions for authors
  1. NavTTT builds on LaCT's nonlinear fast-weight memory mechanism and combines it with navigation-conditioned visual token selection, spatial-context encoding, and hierarchical retrieval. What are the key technical innovations beyond adapting existing parametric memory mechanisms to VLN? The paper should clarify which limitations of prior memory methods are addressed by each design and what fundamentally new capabilities these additions provide.

  2. NavTTT places independent APM branches at six Transformer layers (layers 5, 11, 17, 23, 29, and 35), arguing that they capture complementary historical information at different depths. However, the rationale for selecting these specific layers remains unclear. Were these positions empirically determined or heuristically chosen? Moreover, the current ablations do not compare single-layer memory, shared memory across layers, and independent layer-specific memories. How do the authors justify these layer choices and establish that the hierarchical design provides benefits beyond using a simpler memory architecture?

  3. The visual token selector combines navigation-conditioned relevance scoring with redundancy suppression. Although Table 2 shows that the complete selection strategy outperforms random selection and dense writing, it does not isolate the contribution of either component. How much does each component contribute independently, and is the learned relevance score effective at identifying navigation-critical landmarks and spatial cues rather than merely visually salient features?

  4. NavTTT updates fast weights by maximizing the inner product between the memory network's output and the corresponding value. However, this objective does not directly enforce value reconstruction or guarantee reliable retrieval after repeated updates. While Table 2 demonstrates the benefit of online memory updates through downstream navigation performance, the paper does not directly evaluate retrieval accuracy or memory interference over long trajectories. How do the authors verify that the updated fast weights reliably preserve and retrieve useful historical visual and spatial information?

  5. NavTTT performs gradient-based fast-weight updates after each navigation decision, introducing additional test-time computation. Although the paper reports substantial navigation gains and the overall training cost, it does not sufficiently quantify the latency of online updates or their additional computational overhead. What is the per-decision latency of the memory update, and how does its computational cost compare with the navigation gains and a variant without online updates?

I would be open to revising my score upward if the authors can adequately address these concerns and provide convincing evidence to clarify the novelty, design choices, and practical benefits of the proposed method.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer Hyiu

We sincerely thank the reviewer for your thorough, constructive, and highly detailed review. We deeply appreciate the opportunity to address your valuable questions and concerns, which touch upon the core technical innovations, layer selection rationale, component-wise contributions, retrieval reliability, and test-time computational overhead of our method.


[Q1] Navigation-Specific Contributions Beyond LaCT:

NavTTT builds on LaCT’s nonlinear fast-weight operator and inner-product association objective. Its key contribution lies in organizing how visual evidence is selectively captured, spatially grounded, and hierarchically retrieved during navigation:

  • Navigation-conditioned selection: Repeated visual patches can quickly saturate a limited write budget. The instruction, agent pose, and current decision context jointly guide which tokens are written, while within-frame diversity ensures room is made for distinct, informative cues.
  • Spatial and navigation context encoding: Similar-looking visual cues frequently recur at different points along an extended route. We encode visual appearance alongside patch location, relative pose, and navigation context, ensuring that stored evidence preserves where it was observed and its specific task context.
  • Retrieval at multiple decoder depths: Because earlier frames eventually leave the active visual context, layer-specific queries allow multiple decoder stages to access the accumulated fast state during subsequent action prediction.

Under matched training conditions, full NavTTT improves success rate (SR) by 1.3–3.4 percentage points and reduces navigation error (NE) by 0.14–0.99 m over direct LaCT across the four backbone and benchmark settings in Table A. This comparison evaluates our overall design framework, while Tables B and C further analyze the layer layout and token selector components.

We will incorporate these explanations and the matched LaCT comparison into the revised manuscript.


Table A: Controlled Memory Ablations on Val-Unseen (R2R: R2R-CE; RxR: RxR-CE)

  • Backbone 1: Qwen3-VL-8B
IDVariantR2R NE↓R2R OS↑R2R SR↑R2R SPL↑RxR NE↓RxR SR↑RxR SPL↑RxR nDTW↑
A0Full NavTTT3.8873.166.759.44.2563.652.771.0
A1No APM4.5568.762.456.56.2360.252.962.3
A2Static APM4.4669.763.356.85.6261.352.566.9
A3Direct LaCT4.3970.263.356.45.2462.353.768.3
  • Backbone 2: Qwen2.5-VL-7B-Instruct
IDVariantR2R NE↓R2R OS↑R2R SR↑R2R SPL↑RxR NE↓RxR SR↑RxR SPL↑RxR nDTW↑
B0Full NavTTT4.8367.357.351.06.7153.444.863.1
B1No APM5.5362.453.648.67.6650.743.061.4
B2Static APM4.9163.656.450.37.2052.143.362.5
B3Direct LaCT4.9764.255.749.77.4651.642.160.4

[W1 & Q2] Layer Placement and Independent States

We choose layer locations using a uniform placement heuristic. We divide the 36-layer Qwen3-VL-8B decoder into equal segments and place a branch at the end of each segment. With zero-based indexing, the one-, three-, and six-layer configurations use layer 35, layers 11, 23, and 35, and layers 5, 11, 17, 23, 29, and 35, respectively. This retains the deepest access point as earlier ones are added. The six-layer shared variant ties the fast states of corresponding heads at the same six locations.

The variants match training, the selector, write features, the number of heads, and per-head state dimensions, with online updates enabled throughout. At equal persistent state storage, six shared branches improve SR and SPL by 0.6 points each over the last-layer branch. With all six access locations held fixed, independent states give 1.4 points higher SR, with about six times the state storage. Shared memory attains 0.1 point higher reported SPL at similar latency. We selected six independent APM branches for NavTTT after considering navigation success, path efficiency, and inference latency. This configuration achieves the highest SR among the tested layouts (66.7%) and an SPL of 59.4%, with a mean decision latency of 347.51 ms, just 2.19 ms above the six-layer shared configuration.

Table B: Memory Configurations on R2R-CE Val-Unseen

IDConfigurationSR (%)SPL (%)State (MiB)Decision (ms)
H0Six layers, independent66.759.4576.28347.51
H1Last layer only64.758.996.05313.37
H2Three layers, independent65.159.2288.14327.81
H3Six layers, shared65.359.596.05345.32
AuthorsOpenReview ↗

Reply to Reviewer Hyiu

[Q3] Relevance and Within-Frame Diversity

We compare four selection rules in closed-loop navigation with the same write budget. Relative to random selection, relevance alone improves SR and SPL by 1.3 and 1.5 percentage points on R2R-CE and by 2.2 points each on RxR-CE. Diversity alone improves them by 0.7 and 0.5 points on R2R-CE and by 1.3 and 1.1 points on RxR-CE. Combining the two yields the highest SR and SPL on both benchmarks.

Table C. Visual token selection on Val-Unseen

Selection ruleR2R-CE SR / SPL (%)RxR-CE SR / SPL (%)
Random62.8 / 56.860.5 / 49.6
Relevance only64.1 / 58.362.7 / 51.8
Diversity only63.5 / 57.361.8 / 50.7
Combined66.7 / 59.463.6 / 52.7

In screened examples with matched observations and decision contexts, relevance selects tokens near the dining table, white carpet, and kitchen passage specified by the instructions. The combined rule retains evidence near these cues while reducing similarity among the selected features. These examples provide qualitative evidence of instruction-related selection. We will include the visual comparisons in Figure 4 and the closed-loop results in Table 3 of Appendix C (Visual Token Selection Analysis) in the revised manuscript.

[W2 & Q4] Historical Readout and Action Prediction

We agree that a higher inner product alone leaves open how well individual associations survive repeated updates. In reference replays of 199 long RxR-CE routes across eight scenes, we keep the learned parameters fixed and update the fast weights online. We query the same head with the original write-time key immediately after writing and after further updates. At both times we rank the same 32 values from the source write by cosine similarity to the readout. Delayed probes use source frames that have left the active visual context.

The target ranks first in 6.22%, 6.05%, and 5.23% of probes after 8, 16, and 32 updates. For the same associations at 32 updates, the hit rate changes by +0.14 percentage points from the immediate readout (95% scene-bootstrap CI [−0.57, +0.73]). The absolute hit rate indicates limited discrimination among values written together. The paired interval is compatible with small decreases and increases after subsequent updates.

We keep the decision input fixed and score the same reference-action tokens under teacher forcing. We compare three fast states: the state accumulated along the current route, the learned initial state (reset), and a state from another route with the same number of updates (wrong history). Table D reports each control’s NLL minus the NLL with correct history. On both R2R-CE and long RxR-CE routes, correct history lowers NLL relative to both controls, with all four 95% scene-bootstrap confidence intervals above zero. The reset comparison shows that online updates improve reference-action likelihood. The wrong-history comparison shows that this benefit depends on the history matching the current route, even when the update count is fixed.

Table D: Reference-Action NLL Differences Under History Controls

DatasetControl stateΔ NLL95% CI
R2R-CEReset+0.008204[0.007885, 0.011118]
R2R-CEWrong route+0.005426[0.004127, 0.011335]
RxR-CE (long)Reset+0.006827[0.004219, 0.007660]
RxR-CE (long)Wrong route+0.004970[0.003809, 0.010436]

Differences are in nats per action token. Positive values favor correct history.

[W2 & Q5] Inference Cost and Navigation Gains

We profiled all variants on one RTX 5090 D v2 using the same 128 R2R-CE Val-Unseen inputs, with batch size 1 and four generated tokens per decision. We averaged latencies over three runs after warm-up. Memory writing takes 14.28 ms per decision in full NavTTT, including token selection, key–value preparation, and fast-weight optimization. Compared with the profiled static APM configuration, full NavTTT takes 13.13 ms (4.12%) longer per decision and uses 0.55 GiB more peak GPU memory.

Table E. Inference cost on fixed R2R-CE inputs

VariantMean decision latency (ms)Writing (ms)Peak allocated GPU memory (GiB)
No APM288.69—16.82
Static APM318.82—17.62
Full NavTTT331.9514.2818.17

Total decision latency includes memory writing.

In the closed-loop R2R-CE ablation, enabling online updates on the same trained checkpoint yields 61 net additional successes across 1,839 episodes over static APM, increasing SR by 3.32 percentage points and SPL by 2.6 points. We will report the main findings in Section 4.4 and provide the profiling protocol and full measurements in Appendix F of the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Our Responses and Additional Experiments

Dear Reviewer Hyiu,

Thank you for your questions about what NavTTT adds beyond LaCT and how its memory design affects navigation. In our response, we explain the choice of memory layers and report additional experiments on state sharing, token selection, and historical readout, along with measurements of online update cost.

If you have had a chance to review our response, could you let us know whether any questions remain? A brief reply before the discussion closes would give us time to respond promptly and clarify any points you would like to discuss.

Best regards,
The authors

Reviewer_fJYTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NavTTT builds on LaCT-style fast-weight mechanisms to provide online parametric memory for vision-and-language navigation through navigation-conditioned token selection, contextual encoding, and hierarchical memory retrieval. The method is evaluated on R2R-CE, RxR-CE, and a real robot.

Critical strengths
  1. The method has a clear motivation. Selectively writing memory to address redundancy and task relevance in navigation observations while incorporating spatial and instruction information.
  2. The method achieves strong results on two navigation benchmarks, and controlled LoRA ablations show improvements of 2.45 percentage points in both SR and SPL over the memory-free variant.
Critical weaknesses
  1. The gains attributable to the proposed memory design are insufficiently isolated. The main comparisons involve differences in backbones, inputs, and training configurations, while the LoRA ablations do not cover the main training configuration and lack a direct LaCT comparison using the same backbone, leaving the contribution beyond existing TTT mechanisms insufficiently established.
  2. Applicability across backbones remains unverified. The mechanism is jointly trained and evaluated only with Qwen3-VL-8B, leaving it unclear whether its benefits depend on this particular backbone and training configuration.
Questions for authors
  1. Under the main training configuration, how do a memory-free model, a direct LaCT adaptation, and full NavTTT compare when the backbone and training data are held fixed?
  2. Does the proposed memory design provide consistent gains when the same controlled comparison is conducted with another backbone?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Thank you for the time and care you put into reviewing NavTTT. Your questions helped us clarify the contribution of our memory design and identify the comparisons needed to evaluate it more directly.

NavTTT’s contribution is a navigation-conditioned design for writing and reading fast-weight memory. Building on LaCT’s nonlinear fast weights, NavTTT uses the instruction, agent pose, and decoder decision context to select visual evidence. A within-frame diversity term discourages writing similar features under a fixed token budget. The selected features are encoded together with spatial and task context, and the resulting memory update becomes available at the next decision. This allows queries at several decoder depths to draw on accumulated experience alongside the recent RGB frames.

We revised the manuscript in response to your feedback:

  • We clarified the observation and training settings. NavTTT uses one forward-facing RGB camera, with the current image and up to seven earlier frames from the same camera, together with episode-relative position and heading. We replaced the original LoRA ablations with comparisons against a memory-free model, static APM, and a direct LaCT adaptation under the main joint-training protocol. Across both tested backbones on R2R-CE and RxR-CE, full NavTTT improves SR over direct LaCT by 1.3–3.4 percentage points and reduces NE by 0.14–0.99 m.
  • We tested the main design choices more directly. We explain how the memory layers were placed and compare one-, three-, and six-layer layouts, including independent and shared states at the same six locations. We also evaluate relevance and diversity separately at the same write budget. Combining the two gives the highest SR and SPL among the four tested selection rules on both benchmarks.
  • We added analyses of historical memory, route length, and inference cost. We probe stored associations after repeated updates and compare reference-action scores using the correct-route, reset, and wrong-route states. We also report closed-loop performance by route length and measure the cost of online writing, which takes 14.28 ms per decision in our fixed-input profile.

As the discussion period draws to a close, we would appreciate a brief follow-up on whether the revisions and our point-by-point responses address your main concerns. We would be glad to clarify any remaining points. If your assessment remains unchanged, a short acknowledgment that you have considered our responses would also be appreciated.

AuthorsOpenReview ↗

Reply to Reviewer fJYT

We sincerely thank the reviewer for your careful follow-up and constructive feedback. We fully agree that isolating the impact of individual design choices is essential to verifying our methodological contribution beyond aggregate system performance.

NavTTT contributes a navigation-specific design for forming and accessing fast-weight memory. Consecutive observations often contain similar visual evidence, and a landmark may become useful again after its source frame leaves the visual context. Building on LaCT’s nonlinear fast-weight operator, NavTTT allocates a fixed write budget to evidence relevant to the instruction and the agent’s progress, associates that evidence with spatial and decision context, and makes the accumulated state accessible at multiple decoder depths. This gives the policy a way to use earlier observations while keeping its visual context bounded.

Token selection at a fixed write budget

Selection ruleR2R-CE SR / SPL (%)RxR-CE SR / SPL (%)
Random62.8 / 56.860.5 / 49.6
Relevance only64.1 / 58.362.7 / 51.8
Diversity only63.5 / 57.361.8 / 50.7
Relevance + diversity66.7 / 59.463.6 / 52.7

Both criteria improve SR and SPL when used individually. Combining them raises SR over random selection by 3.9 points on R2R-CE and 3.1 points on RxR-CE. These gains show that the choice of evidence matters even when the number of writes is fixed.

Memory access across decoder layers on R2R-CE

ConfigurationSR (%)SPL (%)Fast-state storage (MiB)
Last layer only64.758.996.05
Three layers, independent65.159.2288.14
Six layers, shared65.359.596.05
Six layers, independent66.759.4576.28

At the same 96.05 MiB of persistent fast-state storage, the six-layer shared layout improves SR and SPL by 0.6 points each over the last-layer layout. This supports the value of giving several decoder layers access to the episode state under the same storage budget.

Explicit context conditioning in APM on R2R-CE

VariantSR (%) ↑SPL (%) ↑OS (%) ↑NE (m) ↓
Full APM66.759.473.13.88
w/o global decision context65.258.771.603.93
w/o explicit instruction conditioning65.158.371.774.04
w/o explicit pose conditioning66.159.573.43.91

Removing global decision context lowers SR and SPL by 1.5 and 0.7 percentage points, respectively. Removing explicit instruction conditioning lowers them by 1.6 and 1.1 points. Removing explicit pose conditioning has a smaller, metric-dependent effect, lowering SR by 0.6 points and increasing SPL by 0.1 points. Decision context and instruction conditioning provide the clearest gains across both metrics in this evaluation.

The matched LaCT comparison measures the joint effect of these design choices. Across both tested backbones and both benchmarks, full NavTTT improves SR by 1.3–3.4 percentage points and reduces NE by 0.14–0.99 m. Ultimately, these component studies bridge the gap between aggregate performance gains and concrete algorithmic choices, demonstrating a fundamental shift: moving fast-weight memories from passive, generic sequence buffers to active, task-conditioned, and hierarchically grounded spatial memory architectures that unlock robust long-horizon reasoning for embodied agents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.