acceptodds
Under review as a conference paper at ICLR 2027

LociNav: Pose-Indexed Visual Caching and Retrieval for Iterative Object-Goal Navigation

Abstract

Vision-language model (VLM) driven navigation agents process streaming egocentric video to find objects in unseen environments, yet they discard the visual features computed at each step, thus losing access to useful observations as the context budget fills. When the agent pursues consecutive goals in the same scene, this per-step forgetting wastes both the information and the computation that earlier exploration has produced. We propose LociNav, which maintains a pose-indexed store of computed visual features that grows as the agent explores. The store is organized by spatial landmarks and viewing direction, keeping it compact without discarding distinct viewpoints. When a new object goal is issued, a scoring model selects the most relevant stored observations and assembles a compact, goal-tailored memory block that is prepended to the model's input. Since the stored features need not be re-encoded and the memory block is arranged to maximize key-value (KV) prefix reuse across navigation steps, the agent consults its memory at reduced computational cost. We further construct an iterative object-goal navigation dataset and benchmark by organizing standard HM3D ObjectNav and HM3D-OVON episodes into tours of consecutive goals and incorporating the natively iterative GOAT-Bench. LociNav achieves state-of-the-art performance across the evaluated benchmarks, with its advantage growing significantly as tours deepen, while achieving over 3 Hz on-device inference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied navigation, object-goal navigation, vision-language model, visual memory, pose-indexed feature store, KV cache reuse, iterative navigation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HhDsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to retain and efficiently use past observations when an agent pursues consecutive object goals within the same scene. LociNav maintains a pose-indexed visual feature store organized by spatial landmarks and viewing-direction sectors. It selects candidates based on goal relevance and graph distance, and prepends a memory block consisting of candidate, path, and live frames to the VLM's input. A spatial pose bias communicates relative pose information, while a ViT cache and KV prefix reuse reduce inference cost. The authors organize existing HM3D ObjectNav and HM3D-OVON episodes into tours and train and evaluate the model together with GOAT-Bench. Results show improved SR on multiple splits and substantially lower inference latency than some memory-augmented baselines, although SPL does not consistently outperform the strongest baselines.

Critical strengths
  1. The paper presents a coherent design addressing a concrete bottleneck in iterative navigation. The persistent pose-indexed visual feature store, goal-directed retrieval, memory block with different update frequencies, and compute reuse jointly address information retention and inference cost, rather than simply increasing the context length. On the fixed OVON-Tour subset used in Table 4, the full memory block improves SR/SPL from 41.2/22.0 to 61.6/31.6. In Table 6, additive attention bias improves SR from 44.4 without pose injection to 61.6. These results support the value of spatial organization and route guidance.

  2. The efficiency evaluation is concrete and complements results across multiple benchmarks and hardware platforms. In Table 5, the two reuse mechanisms reduce per-step latency from 0.41 s to 0.25 s, with Table 8 in the appendix providing a breakdown into ViT encoding, KV prefilling, and decoding. Evaluation on workstation GPUs and Jetson Orin, together with the storage–accuracy–latency trade-offs in Table 9, provides evidence of practical utility beyond success rate alone. This is an important consideration supporting acceptance.

Critical weaknesses
  1. The controlled comparison of memory block designs could be strengthened. Tables 4 and 6 support the effectiveness of the candidate/path/live structure and spatial pose bias. However, the configurations in Table 4 also differ in the number of memory frames provided, making it difficult to fully distinguish the benefit of organizing frames by role from that of supplying more historical visual information. A simple goal-relevant top-k retrieval baseline with the same number of memory frames or the same token budget would more clearly establish the additional value of route previews and local guidance.

  2. The efficiency ablation could report navigation metrics more explicitly. Table 5 reports RtL/E2EL and discusses improved SPL relative to the non-memory baseline, but does not report SR/SPL for its configurations. Although Table 4 supports the navigation benefit of the memory block, reporting SR/SPL alongside the latency results would help verify that the reuse mechanisms preserve navigation performance and clarify the interpretation of the E2EL gains.

Questions for authors
  1. Contribution of the memory block structure. Could the candidate/path/live structure be compared with goal-relevant top-k retrieval using the same number of memory frames or the same token budget, to distinguish the benefit of memory organization from that of additional input information?

  2. Metrics supporting the efficiency ablation. Could you report SR/SPL for the configurations in Table 5, including the non-memory baseline, and confirm whether navigation performance remains unchanged when ViT cache and KV prefix reuse are enabled?

  3. Spatial pose bias and KV prefix reuse. In Eq. (2), is the egocentric frame tied to each frame's observation pose or to the agent's current pose? As the agent moves, how is attention computation within the cached prefix kept unchanged?

1 comment
AuthorsOpenReview ↗

Kindly requesting for feedback on our rebuttal

Dear Reviewer,

Thank you again for your detailed review. We have posted a response addressing each weakness and question, including a same-budget comparison with top-k retrieval baselines that isolates the benefit of the candidate/path/live structure from the number of frames (W1+Q1), SR and SPL results added to the efficiency ablation confirming that the reuse mechanisms preserve navigation performance (W2+Q2), and a clarification of the pose bias reference frame and the layer-wise KV reuse argument with empirical verification (Q3). New experiments conducted for this rebuttal include the same-budget retrieval baselines trained from scratch. We have also uploaded a revised paper that addresses these points with changes highlighted in red.

We would be grateful to know whether these address your concerns, in particular the controlled comparison of memory block designs and the metrics supporting the efficiency ablation. If anything remains unclear, we would welcome further discussion.

With a few days remaining in the discussion period, we would appreciate any additional comments at your earliest convenience.

Thank you for your time and consideration.

Reviewer_jbWiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LociNav for iterative object-goal navigation in the same environment. The method stores visual features indexed by position and viewing direction, then retrieves candidate, path, and live frames for each goal. A pose-aware attention bias provides spatial information, while visual caching and KV-prefix reuse reduce inference cost. Experiments on ObjectNav-Tour, OVON-Tour, and GOAT-Bench show improved navigation performance with modest additional latency.

Critical strengths
  1. The candidate/path/live structure provides both target evidence and navigation guidance. The ablations support this decomposition.
  2. Ordering memory by update frequency supports efficient prefix reuse. The latency breakdown demonstrates the benefits of visual caching and KV reuse.
Critical weaknesses
  1. The main technical contribution is still unclear. • Most of the framework combines familiar ideas: spatial indexing, semantic retrieval, graph planning, and caching. The combination is useful, but I am not sure it adds enough method-level novelty beyond existing memory-based navigation approaches. • The memory slots and spatial bias seem to be the main new parts. The paper should make clearer what these designs let the agent do that earlier methods could not. At present, the contribution reads more like system integration.
  2. The spatial bias and exact KV reuse are not clearly described. • It is unclear which reference frame Equation (2) uses. If historical displacements are expressed in the current agent frame, turning could change the bias even when the stored frames stay the same. • The bias can change the K/V in later layers, even if it is only added to attention logits. The paper needs to explain why reusing the old K/V gives the same result as computing them again. • As written, x and y share frequencies and their contributions are summed. Swapping Δx and Δy therefore gives the same bias. Sine terms help distinguish some positive and negative displacements, but they do not resolve this axis ambiguity.
Questions for authors
  1. What is the main new mechanism beyond existing visual-memory and caching methods? Which previously unresolved problem is addressed by the three memory slots and their update rules?
  2. The paper lacks visual trajectory comparisons. Could the authors provide representative comparisons showing how the method improves navigation? Such comparisons are important for supporting claims of superior navigation performance.
  3. Do x and y share frequencies without independent weights? If so, how is the invariance to exchanging Δx and Δy addressed? If the implementation differs, please provide the exact formula.
  4. Does Equation (2) use the current agent frame or a fixed frame associated with each historical query? Does the bias apply within the memory prefix? Please explain why the prefix K/V at every layer remains equivalent to full recomputation after the agent turns.
3 comments
AuthorsOpenReview ↗

Kindly requesting for feedback on our rebuttal

Dear Reviewer,

Thank you again for your detailed review. We have posted a response addressing each weakness and question, including the main technical contribution and how the designs constrain one another (W1+Q1), a clarification of the reference frame and a formal argument for exact KV reuse at every layer with empirical verification on ~12.5k steps (W2+Q4), the corrected per-channel gate formulation that resolves the x/y symmetry (W2+Q3), and trajectory visualizations comparing all three methods (Q2). New experiments conducted for this rebuttal include prefix reuse vs. full recomputation equivalence verification, and same-budget top-k retrieval baselines trained from scratch. We have also uploaded a revised paper that addresses these points with changes highlighted in red.

We would be grateful to know whether these address your concerns, in particular the novelty of the mechanism beyond system integration and the correctness of KV prefix reuse. If anything remains unclear, we would welcome further discussion.

With a few days remaining in the discussion period, we would appreciate any additional comments at your earliest convenience.

Thank you for your time and consideration.

Reviewer_jbWiOpenReview ↗

The authors have addressed my concerns.

The authors have addressed my concerns and I would like to suggest accepting this paper.

AuthorsOpenReview ↗

Thank you for the updated assessment

Dear Reviewer,

Thank you for reconsidering our rebuttal and for raising the score. Your detailed and constructive feedback helped us significantly improve the clarity and rigor of the paper. We are grateful for the time and effort you invested in the review.

Best regards,
The Authors

Reviewer_Re2ROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LociNav, a memory-augmented vision-language navigation system for iterative object-goal navigation. It maintains a persistent store of visual features indexed by spatial landmarks and viewing directions. For each new goal, the system retrieves relevant observations and constructs a compact memory block containing candidate, path, and local guidance frames. A pose-aware attention bias conveys spatial relationships, while cached visual features and a stable KV prefix reduce repeated computation. The authors also construct ObjectNav-Tour and OVON-Tour by linking existing episodes and evaluate on the natively iterative GOAT-Bench. The reported results show competitive success rates, increasing benefits as tours deepen, and inference latency close to the memoryless baseline.

Critical strengths
  1. Efficient persistent memory. Visual feature caching and KV prefix reuse reduce per-step latency from 0.41 s to 0.25 s, only approximately 40 ms above the memoryless baseline (Table 5).

  2. Effective memory organization. Combining candidate, path, and live frames improves SR from 41.2 to 61.6 on the ablation subset (Table 4), supporting the importance of spatial guidance alongside retrieved observations.

  3. Useful accuracy–resource trade-off. Pooling cached features to four tokens reduces storage from 1.85 GB to 0.15 GB per tour and latency from 0.70 s to 0.25 s, with SR decreasing from 65.0 to 61.6 (Table 9).

  4. Evaluation addresses accumulated experience. Experiments cover ObjectNav-Tour, OVON-Tour, and GOAT-Bench, while the tour-depth analysis directly examines whether additional experience improves subsequent navigation.

Critical weaknesses
  1. Evaluation comparability needs clarification. Oracle-controlled connecting segments supply additional observations to persistent memory. It is unclear whether all memory baselines receive equivalent observations or how much these segments contribute to the reported gains.

  2. KV reuse correctness needs validation. The coordinate frame used for the pose-dependent attention bias is ambiguous. If biases within the memory prefix change as the agent turns, cached states may become stale. Consistency with full recomputation should be verified.

  3. Complete latency accounting is unclear. Table 8 reports only ViT encoding, prefilling, and decoding. The paper should clarify whether reported timings also include SigLIP 2 encoding, retrieval, graph operations, and memory updates.

  4. Robustness to pose errors is untested. All experiments use simulator-provided poses, although pose estimates directly affect memory indexing, retrieval, and spatial attention. Without noise or drift experiments, applicability to practical localization systems remains uncertain.

  5. Gains over the strongest GOAT-Bench baseline are modest. Compared with AstraNav-Memory, SR improves by 1.6 and 1.9 percentage points on Seen and Unseen, but decreases on Synonyms, while SPL is lower across all three splits. Without uncertainty estimates, the reliability of these small SR improvements is difficult to assess.

Questions for authors

The following questions concern clarification and validation of the current claims.

  1. Evaluation protocol: Were AstraNav-Memory and Uni-NaVid-Mem rerun using the same tour ordering and connecting-segment protocol, with access to connecting-segment observations? When those observations are excluded from persistent memory, how do LociNav’s relative SR, SPL, and performance gains with increasing tour depth change?

  2. Cache correctness: Which token and observation time define the egocentric coordinate frame in Equation (2)? Does the bias apply to attention within the memory prefix? Please explain which quantities remain invariant as the agent moves and turns, and provide a comparison of output differences or action agreement between full recomputation and KV reuse for identical inputs.

  3. Complete system cost: Do the timings in Tables 3, 5, and 8 include SigLIP 2 image encoding, candidate scoring, graph-path queries, and memory updates? Table 8 accounts for total latency using only ViT encoding, prefilling, and decoding, leaving the treatment of these additional operations unclear. Does E2EL include connecting segments, and how are failed episodes handled when computing its average?

1 comment
AuthorsOpenReview ↗

Kindly requesting for feedback on our rebuttal

Dear Reviewer,

Thank you again for your detailed review. We have posted a response addressing each weakness and question, including a clarification that all memory baselines were rerun with the same connecting-segment protocol (W1+Q1), the reference frame and exact KV reuse argument with a full recomputation comparison table (W2+Q2), a breakdown of which operations are included in RtL and E2EL (W3+Q3), a pose noise robustness analysis under both bounded noise and uncorrected drift (W4), and a discussion of the GOAT-Bench margins in terms of input modality, latency, and tour length (W5). New experiments conducted for this rebuttal include the reuse equivalence verification and the pose noise sensitivity analysis. We have also uploaded a revised paper that addresses these points with changes highlighted in red.

We would be grateful to know whether these address your concerns, in particular the evaluation protocol fairness and the robustness to pose errors. If anything remains unclear, we would welcome further discussion.

With a few days remaining in the discussion period, we would appreciate any additional comments at your earliest convenience.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.