- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents ReMaP, a retrieval-augmented memory and planning framework for long-horizon robot navigation. Instead of compressing prior experience into a predefined metric map, topological graph, or scene graph, ReMaP stores relatively raw observations and retrieves task-relevant context from a dense memory when a new navigation task is given. The core retrieval model uses separate semantic and geometric embeddings. Semantic embeddings are intended to capture task- and landmark-level relevance, while geometric embeddings capture spatial consistency and reachability. Retrieval is recursive: previously retrieved observations are incorporated into subsequent queries, with the goal of producing an ordered and spatially coherent context sequence. The retrieved context is then given to a VLM planner, which generates image and/or language subgoals for a low-level navigation policy. The paper evaluates ReMaP on GOAT-Bench and on eight real-world navigation tasks. On GOAT-Bench it reports 41.6% SR and 27.6% SPL, outperforming the reported RGB baselines. In the real-world evaluation, ReMaP obtains 29.2% success over 24 trials compared with 8.3% for the strongest reported memory baseline. The paper also reports substantially faster retrieval than LM-Nav and RAVEN.
Critical strengths
-
ReMaP provides a simple and flexible alternative to explicitly structured robot memories. Storing observations and learning how to retrieve task-relevant information avoids committing to a fixed scene-graph ontology or a particular set of spatial relationships. This is attractive for long-horizon navigation where the relevant information can depend strongly on the task. The separate semantic and geometric representation is also a sensible design for navigation, since retrieving something that looks semantically relevant is not sufficient if the observations do not form a navigable sequence.
-
The paper evaluates a complete system in the real world and explicitly studies retrieval efficiency. Unlike many memory/planning papers that stop at simulation, ReMaP is integrated with OmniVLA and evaluated on real navigation trajectories of 20–80 m. The efficiency result is also practically relevant: the reported full-context retrieval time is 1.49 s for ReMaP versus 40.73 s for RAVEN and 4.41 s for LM-Nav.
Critical weaknesses
- The paper does not adequately isolate or validate the components that constitute its main technical contribution. The paper explicitly motivates two navigation-specific mechanisms: recursive retrieval and separate semantic/geometric embeddings. However, I could not find controlled comparisons such as:
- recursive retrieval vs. one-shot/top- retrieval;
- split semantic/geometric embeddings vs. a single joint embedding;
- learned embeddings vs. frozen/off-the-shelf embeddings under the same retrieval procedure;
- semantic-only vs. geometric-only retrieval.
There is also no direct retrieval benchmark reporting quantities such as Recall@K, ordering accuracy, spatial consistency, or another measure of whether the correct memory observations are actually retrieved. This is important because the end-to-end system contains several strong components: Gemini-based image labeling, a VLM planner, OmniVLA, and a large learned retrieval model. Without these ablations, it is difficult to tell which part of ReMaP is responsible for the reported improvement.
- The main evaluations do not provide a sufficiently clean or statistically strong comparison. The GOAT-Bench result uses oracle pre-exploration to ensure full scene coverage, converts image goals to language using Gemini-2.5-Pro, and then uses a fast-marching planner. The paper itself acknowledges that its exploration protocol differs from those of the reported baselines. Therefore, I do not think the 41.6% vs. 38.5% SR comparison can by itself support a strong claim of outperforming prior RGB navigation systems. The real-world experiment is more relevant, but it contains only eight tasks with three trials each, i.e. 24 trials per method. The tasks are explicitly designed to benefit from memory, and a manual tour is collected beforehand to ensure that each task has sufficient memory coverage.
ReMaP succeeds in 29.2% of these trials versus 8.3% for RAVEN, corresponding to only 7 versus 2 successful trials. No uncertainty estimate is reported for this comparison.
There are also some baseline-specific issues. LM-Nav reaches essentially the same average progress as ReMaP (67.3% vs. 67.5%) but receives 0% success because it never issues the required stop command. This makes the headline success comparison sensitive to differences in the termination interface rather than only navigation quality.
Questions for authors
- Can the authors provide ablations for the main retrieval design choices?
In particular, I would like to see recursive vs. non-recursive retrieval, joint vs. split semantic/geometric embeddings, and learned vs. frozen/off-the-shelf retrieval under the same downstream planner and policy. These comparisons would materially affect my assessment of the technical contribution.
- Can the authors provide a matched-protocol GOAT-Bench comparison?
Since ReMaP uses oracle pre-exploration while the comparison methods use different exploration protocols, can the authors evaluate a relevant baseline with exactly the same pre-collected memory / coverage assumption, or otherwise isolate retrieval/planning quality from exploration quality?
- How robust is the real-world improvement statistically?
The current experiment contains only 24 trials per method. Please report confidence intervals or another appropriate uncertainty estimate for success, and ideally evaluate more trials / environments. It would also be useful to report an endpoint-reaching metric independent of issuing Stop, given that LM-Nav reaches 67.3% average progress but obtains 0% success under the current stopping criterion.
- Some related works are missing:
P. Chen, D. Ji, K. Lin, R. Zeng, T. H. Li, M. Tan, and C. Gan. Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation. NeurIPS, 2022
S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev. Think Global, Act Local: Dual-Scale Graph Transformer for Vision-and-Language Navigation. CVPR, 2022.
D. An, Y. Qi, Y. Li, Y. Huang, L. Wang, T. Tan, and J. Shao. BEVBert: Multimodal Map Pre-training for Language-guided Navigation. ICCV, 2023.
5 comments
“...split semantic/geometric embeddings vs. a single joint embedding …“...semantic-only vs. geometric-only retrieval…”
We evaluate the individual embeddings on GOAT-bench on a subset where only episode 0 of each environment is run.
| Retrieval | SR (%) | SPL (%) | Object SR | Description SR | Image SR |
|---|---|---|---|---|---|
| Geometric only | 60.4 [51.6, 69.6] | 31.2 [24.3, 38.9] | 67.7 | 53.8 | 59.1 |
| Semantic only | 73.0 [64.5, 81.0] | 46.5 [39.5, 53.4] | 74.7 | 68.1 | 76.1 |
| Geometric + semantic (ReMaP) | 74.8 [67.5, 81.6] | 46.7 [40.3, 53.2] | 80.8 | 67.0 | 76.1 |
Note: these results were edited from the original response, which was run with an older version of the embedding model that did not include a dispersal term in the loss. However, the original results showed similar relative performance.
On GOAT-bench, geometric and semantic components are both important, but the semantic component contributes most to performance. Adding the geometric embedding to semantic-only retrieval increased overall SR from 73.0% to 74.8% and object-goal SR from 74.7% to 80.8%. Semantic-only retrieval outperforms combined retrieval by 1.1 percentage points. However, we also recognize that GOAT-bench can sometimes require only the end-goal to be retrieved, such as description-goal tasks, therefore making semantic embeddings more important than geometric embeddings in these cases.
Furthermore, we provide training curves to justify the decision to use split semantic and geometric embeddings rather than a single learned embedding. We found that when using a single embedding, the model would learn to completely ignore either the geometric or the semantic information, resulting in worse performance overall.
| Epoch | Single embedding | Dual embedding (ours) |
|---|---|---|
| 0 | 0.220 | 0.209 |
| 4 | 0.217 | 0.208 |
| 8 | 0.202 | 0.186 |
| 12 | 0.184 | 0.177 |
| 16 | 0.176 | 0.167 |
| 20 | 0.172 | 0.165 |
| Best | 0.171 (epoch 17) | 0.162 (epoch 17) |
The validation curve also shows that we steadily achieve lower validation loss with split embeddings compared to joint embeddings.
“...learned embeddings vs. frozen/off-the-shelf embeddings under the same retrieval procedure…”
We compare ReMaP with off-the-shelf embeddings on GOAT-bench, including SigLIP, CLIP, and Qwen3-VL. We use a subset of the GOAT-bench benchmark where only episode 0 of each environment is run. Note that this comparison is run with a new regularization term in the loss that encourages dispersion of the embeddings and improves the performance of ReMaP. We will describe this term in the updated manuscript with additional discussion.
| Embedding Model | Parameter count | SR (%) [95% CI] | SPL( %) [95% CI] |
|---|---|---|---|
| SigLIP 2 | 375 M | 70.5 [61.7, 78.6] | 41.8 [34.3, 49.3] |
| CLIP | 428 M | 74.5 [67.0, 81.3] | 49.8 [42.5, 56.9] |
| Qwen3-VL | 2.1 B | 76.3 [68.6, 83.1] | 48.3 [42.1, 54.5] |
| ReMaP | 465 M | 74.8 [67.5, 81.6] | 46.7 [40.3, 53.2] |
Qwen3-VL achieved the highest success rate (76.3%), and CLIP achieved the highest SPL (49.8%). However, Qwen3-VL has 4.5x the number of parameters, and CLIP has a slightly worse success rate. While GOAT-bench provides large-scale planning evaluation, we found GOAT-bench often only requires retrieval of the end goal to succeed. Therefore, we also perform a more thorough evaluation of ReMaP and Qwen3-VL on real-world tasks and provide qualitative results. Overall, we found that the plans generated with Qwen3-VL retrieval contained the end goal of most tasks just as often as the ReMaP plans for those same tasks. However, we found that ReMaP was much more effective at producing plans that were more continuous in nature, meaning that more correctly ordered intermediate observations were present between objectives that were specified in the task. This means that ReMaP’s plans are likely easier for a VLA to follow step-by-step because they are specified more exactly than the plans produced by Qwen3-VL.
[Weakness 3] “...The GOAT-Bench result uses oracle pre-exploration to ensure full scene coverage, converts image goals to language using Gemini-2.5-Pro, and then uses a fast-marching planner…” [Question 2] “Can the authors provide a matched-protocol GOAT-Bench comparison?”
Since submission, we adapted our method to use online exploration rather than oracle pre-exploration, which matches the protocol of the GOAT-Bench benchmark. We found that with this change, we were able to dramatically increase the performance of ReMaP, as now purposeful progress can be made through the environment. We now also outperform most RGB-D-based methods we introduced previously, except for 3D-Mem.
| Input | Method | SR (%) ↑ | SPL (%) ↑ |
|---|---|---|---|
| RGB-D | MTU3D | 40.8 | 12.1 |
| RGB-D | MSGNav | 52.0 | 29.6 |
| RGB-D | 3D-Mem | 69.1 | 48.9 |
| RGB | Modular GOAT | 24.9 | 17.2 |
| RGB | VLMNav | 16.3 | 6.6 |
| RGB | DyNaVLM | 25.5 | 10.2 |
| RGB | TANGO | 35.5 | 19.5 |
| RGB | Uni-NaVid | 38.5 | 19.8 |
| RGB | ReMaP (Ours) | 65.2 [60.9, 69.8] | 36.6 [33.2, 40.6] |
Note: these numbers were updated to use the latest model with a dispersal loss term. Previously, they were 65.2 and 33.3.
[Weakness 4] “...The real-world experiment is more relevant, but it contains only eight tasks with three trials each, i.e. 24 trials per method…the tasks are explicitly designed to benefit from memory, and a manual tour is collected beforehand to ensure that each task has sufficient memory coverage…” [Question 3] “How robust is the real-world improvement statistically?... It would also be useful to report an endpoint-reaching metric independent of issuing Stop”
It is correct that we adjust the kinds of tasks we use to evaluate ReMaP in the real world, as we are targeting the setting where memory must be leveraged to inform how to complete a task. This setting is specifically called “Iterative vision-language navigation (as introduced in Krantz et al., “Iterative Vision-and-Language Navigation”, 2022). We will clarify that this is the target domain of the method in the final manuscript.
We recognize that the real-world evaluation is more limited than the GOAT-bench evaluation. These results are much harder to scale, especially with the slower baselines and longer tasks, as a single trial can take several minutes. Therefore, these results are more qualitative and for demonstrating the capabilities of ReMaP rather than hard empirical evidence.
However, we do extend the real-world evaluation of ReMaP, introducing 5 new tasks and increasing the number of trials to 39. We report the standard error across tasks.
| Method | Progress: Short | Progress: Medium | Progress: Long | Progress: Overall | Success: Short | Success: Medium | Success: Long | Success: Overall |
|---|---|---|---|---|---|---|---|---|
| LM-Nav | 71.7 ± 7.5 | 75.3 ± 10.5 | 42.7 ± 6.4 | 63.9 ± 5.2 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 |
| RAVEN | 51.5 ± 7.3 | 53.4 ± 8.5 | 34.9 ± 6.7 | 47.0 ± 4.4 | 13.3 ± 9.1 | 0.0 ± 0.0 | 0.0 ± 0.0 | 5.1 ± 3.6 |
| ReMaP | 84.5 ± 5.6 | 60.3 ± 9.6 | 52.7 ± 1.8 | 67.3 ± 4.3 | 46.7 ± 13.3 | 8.3 ± 8.3 | 0.0 ± 0.0 | 20.5 ± 6.6 |
We find that ReMaP outperforms all the baselines in success and progress along the task trajectory. ReMaP improves progress along the task trajectory over RAVEN by 20.3 percentage points and over LM-Nav by 3.4 percentage points. In success rate, ReMaP outperforms RAVEN by 15.4 points and LM-Nav by 20.5 points.
Qualitatively, we found that ReMaP and RAVEN had similar failure modes, especially on tasks where abrupt turns had to be made, resulting in failure because the observation did not make the turn visible early enough for the planner to respond. The remaining gap between RAVEN and ReMaP is accounted for by planning performance, as RAVEN often does not spread plan coverage across the task, therefore resulting in poor guidance for the low-level policy. LM-Nav has strong collision avoidance behavior but most often fails at decision points, such as hallway junctions, as the plans mostly draw in geometrically incoherent goals.
We believe that progress does provide a “stop” independent metric to measure if the method reaches the end goal. If the method reaches the end goal, 100% progress will be recorded, regardless of task completion. However, we’d like to emphasize the importance of the “success” metric, as this indicates the method is able to recognize its progress through the task and when it has reached the end goal.
[Question 4] “Some related works are missing”
Thank you for bringing these works to our attention. We will discuss these in the related work section in our final manuscript.
We thank the reviewer for their insightful comments and suggestions. In this response, we will provide new experimental results and clarifications of our method.
[Weakness 1] “...does not adequately isolate or validate the components that constitute its main technical contribution….recursive retrieval vs. one-shot/top-(K) retrieval …” [Question 1] “Can the authors provide ablations for the main retrieval design choices?” [Weakness 2] “...no direct retrieval benchmark reporting quantities such as Recall@K, ordering accuracy, spatial consistency, or another measure of whether the correct memory observations are actually retrieved…”
Thank you for your detailed suggestions for ablations of our method. To evaluate the retrieval capabilities of ReMaP as well as ablate several design choices, we construct a retrieval benchmark by using the same labeling techniques that we used to generate the training data for ReMaP. We label two new and unseen datasets, NavWareSet (Brayan et al., 2026) and OpenLORIS (Shi et al., 2019). Each benchmark example consists of a task, a start image, and a corresponding ground-truth plan that is an ordered list of observations, sampled every 2m or every time a turn greater than 45 degrees is made. To avoid overfitting to camera artifacts and to better represent how retrieval models would function in the real world, the observations belonging to the ground-truth route that is being assessed are held out during evaluation. Instead, we use a geometric matcher to confirm that the retrieved observations correspond to observations in the ground-truth plan. We discard tasks that are less than 1m in length. In total, the benchmark contains 2,064 non-overlapping tasks (1,847 NavWareSet, 217 OpenLORIS). The path lengths range from 1m to 79m, with 1,225 tasks less than 5m in length, 613 tasks between 5m and 15m in length, 191 tasks 15m to 40m in length, and 35 tasks 40m+ in length.
We measure the recall, precision, if the goal is in the retrieved context, off-route rate (how often are retrievals not on the ground truth route), and Kendall τ, which represents the correctness of the plan ordering, with -1 being a fully reversed plan and 1 being a plan whose ordering is fully correct. To calculate each score,for each pair of matched observations in the retrieved plan, we first assign a score of 1 if they appear in the same order in the generated plan as in the ground truth, and -1 if the order is reversed. The total score is calculated by averaging the score across all pairs. Backtrack rate is the fraction of consecutive steps between a plan's correctly retrieved waypoints that move backward along the ground-truth route.
First, we compare recursive retrieval to top-K retrieval. We compare these two approaches on a retrieval benchmark constructed from ground truth plans and their associated tasks. We find that at k=10, top-k has slightly higher recall and precision while sacrificing ordering quality. While recursive retrieval sacrifices some recall and precision, it does not find the goal significantly less than the top-K retrieval and preserves ordering significantly better, and we find that this characteristic is favorable for generating coherent plans that a VLA can follow.
We find that at k=10, top-k has slightly higher recall and precision while sacrificing ordering quality. While recursive retrieval sacrifices some recall and precision, it does not find the goal significantly less than the top-K retrieval and preserves ordering significantly better, and we find that this characteristic is favorable for generating coherent plans that a VLA can follow.