acceptodds
Under review as a conference paper at ICLR 2027

ReMaP: Retrieval-Augmented Multimodal Planning with Dense Robot Memory

Abstract

General-purpose robotic systems must not only perform low-level skills, but also accomplish long-horizon goals. In partially observable settings, like navigation, this requires maintaining a memory or “mental map” that supports efficient, flexible retrieval of task-relevant prior experience. To this end, we present \MethodName, or Retrieval-Augmented Multimodal Planning, a navigation memory and planning framework inspired by retrieval-augmented generation (RAG). \MethodName uses a learned embedding model that captures both semantic and geometric information to retrieve context from a dense memory of prior experience. This context is used to ground a plan of sub-goals, generated by a VLM, for a low-level navigation policy to execute. To assess \MethodName's capabilities for long-horizon navigation, we evaluate \MethodName as part of a full robotic system on real-world robot tasks, beating other navigation baselines by over 20% in success rate and 27 in speed, and on GOAT-bench, where it outperforms prior RGB-based methods by 3% in success rate and 7.8% in SPL.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot navigation, robot memory, planning, retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W1eNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ReMaP, a retrieval-augmented memory and planning framework for long-horizon robot navigation. Instead of compressing prior experience into a predefined metric map, topological graph, or scene graph, ReMaP stores relatively raw observations and retrieves task-relevant context from a dense memory when a new navigation task is given. The core retrieval model uses separate semantic and geometric embeddings. Semantic embeddings are intended to capture task- and landmark-level relevance, while geometric embeddings capture spatial consistency and reachability. Retrieval is recursive: previously retrieved observations are incorporated into subsequent queries, with the goal of producing an ordered and spatially coherent context sequence. The retrieved context is then given to a VLM planner, which generates image and/or language subgoals for a low-level navigation policy. The paper evaluates ReMaP on GOAT-Bench and on eight real-world navigation tasks. On GOAT-Bench it reports 41.6% SR and 27.6% SPL, outperforming the reported RGB baselines. In the real-world evaluation, ReMaP obtains 29.2% success over 24 trials compared with 8.3% for the strongest reported memory baseline. The paper also reports substantially faster retrieval than LM-Nav and RAVEN.

Critical strengths
  1. ReMaP provides a simple and flexible alternative to explicitly structured robot memories. Storing observations and learning how to retrieve task-relevant information avoids committing to a fixed scene-graph ontology or a particular set of spatial relationships. This is attractive for long-horizon navigation where the relevant information can depend strongly on the task. The separate semantic and geometric representation is also a sensible design for navigation, since retrieving something that looks semantically relevant is not sufficient if the observations do not form a navigable sequence.

  2. The paper evaluates a complete system in the real world and explicitly studies retrieval efficiency. Unlike many memory/planning papers that stop at simulation, ReMaP is integrated with OmniVLA and evaluated on real navigation trajectories of 20–80 m. The efficiency result is also practically relevant: the reported full-context retrieval time is 1.49 s for ReMaP versus 40.73 s for RAVEN and 4.41 s for LM-Nav.

Critical weaknesses
  1. The paper does not adequately isolate or validate the components that constitute its main technical contribution. The paper explicitly motivates two navigation-specific mechanisms: recursive retrieval and separate semantic/geometric embeddings. However, I could not find controlled comparisons such as:
  • recursive retrieval vs. one-shot/top-KK retrieval;
  • split semantic/geometric embeddings vs. a single joint embedding;
  • learned embeddings vs. frozen/off-the-shelf embeddings under the same retrieval procedure;
  • semantic-only vs. geometric-only retrieval.

There is also no direct retrieval benchmark reporting quantities such as Recall@K, ordering accuracy, spatial consistency, or another measure of whether the correct memory observations are actually retrieved. This is important because the end-to-end system contains several strong components: Gemini-based image labeling, a VLM planner, OmniVLA, and a large learned retrieval model. Without these ablations, it is difficult to tell which part of ReMaP is responsible for the reported improvement.

  1. The main evaluations do not provide a sufficiently clean or statistically strong comparison. The GOAT-Bench result uses oracle pre-exploration to ensure full scene coverage, converts image goals to language using Gemini-2.5-Pro, and then uses a fast-marching planner. The paper itself acknowledges that its exploration protocol differs from those of the reported baselines. Therefore, I do not think the 41.6% vs. 38.5% SR comparison can by itself support a strong claim of outperforming prior RGB navigation systems. The real-world experiment is more relevant, but it contains only eight tasks with three trials each, i.e. 24 trials per method. The tasks are explicitly designed to benefit from memory, and a manual tour is collected beforehand to ensure that each task has sufficient memory coverage.

ReMaP succeeds in 29.2% of these trials versus 8.3% for RAVEN, corresponding to only 7 versus 2 successful trials. No uncertainty estimate is reported for this comparison.

There are also some baseline-specific issues. LM-Nav reaches essentially the same average progress as ReMaP (67.3% vs. 67.5%) but receives 0% success because it never issues the required stop command. This makes the headline success comparison sensitive to differences in the termination interface rather than only navigation quality.

Questions for authors
  1. Can the authors provide ablations for the main retrieval design choices?

In particular, I would like to see recursive vs. non-recursive retrieval, joint vs. split semantic/geometric embeddings, and learned vs. frozen/off-the-shelf retrieval under the same downstream planner and policy. These comparisons would materially affect my assessment of the technical contribution.

  1. Can the authors provide a matched-protocol GOAT-Bench comparison?

Since ReMaP uses oracle pre-exploration while the comparison methods use different exploration protocols, can the authors evaluate a relevant baseline with exactly the same pre-collected memory / coverage assumption, or otherwise isolate retrieval/planning quality from exploration quality?

  1. How robust is the real-world improvement statistically?

The current experiment contains only 24 trials per method. Please report confidence intervals or another appropriate uncertainty estimate for success, and ideally evaluate more trials / environments. It would also be useful to report an endpoint-reaching metric independent of issuing Stop, given that LM-Nav reaches 67.3% average progress but obtains 0% success under the current stopping criterion.

  1. Some related works are missing:

P. Chen, D. Ji, K. Lin, R. Zeng, T. H. Li, M. Tan, and C. Gan. Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation. NeurIPS, 2022

S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev. Think Global, Act Local: Dual-Scale Graph Transformer for Vision-and-Language Navigation. CVPR, 2022.

D. An, Y. Qi, Y. Li, Y. Huang, L. Wang, T. Tan, and J. Shao. BEVBert: Multimodal Map Pre-training for Language-guided Navigation. ICCV, 2023.

5 comments

We thank the reviewer for their insightful comments and suggestions. In this response, we will provide new experimental results and clarifications of our method.

[Weakness 1] “...does not adequately isolate or validate the components that constitute its main technical contribution….recursive retrieval vs. one-shot/top-(K) retrieval …” [Question 1] “Can the authors provide ablations for the main retrieval design choices?” [Weakness 2] “...no direct retrieval benchmark reporting quantities such as Recall@K, ordering accuracy, spatial consistency, or another measure of whether the correct memory observations are actually retrieved…”

Thank you for your detailed suggestions for ablations of our method. To evaluate the retrieval capabilities of ReMaP as well as ablate several design choices, we construct a retrieval benchmark by using the same labeling techniques that we used to generate the training data for ReMaP. We label two new and unseen datasets, NavWareSet (Brayan et al., 2026) and OpenLORIS (Shi et al., 2019). Each benchmark example consists of a task, a start image, and a corresponding ground-truth plan that is an ordered list of observations, sampled every 2m or every time a turn greater than 45 degrees is made. To avoid overfitting to camera artifacts and to better represent how retrieval models would function in the real world, the observations belonging to the ground-truth route that is being assessed are held out during evaluation. Instead, we use a geometric matcher to confirm that the retrieved observations correspond to observations in the ground-truth plan. We discard tasks that are less than 1m in length. In total, the benchmark contains 2,064 non-overlapping tasks (1,847 NavWareSet, 217 OpenLORIS). The path lengths range from 1m to 79m, with 1,225 tasks less than 5m in length, 613 tasks between 5m and 15m in length, 191 tasks 15m to 40m in length, and 35 tasks 40m+ in length.

We measure the recall, precision, if the goal is in the retrieved context, off-route rate (how often are retrievals not on the ground truth route), and Kendall τ\tau, which represents the correctness of the plan ordering, with -1 being a fully reversed plan and 1 being a plan whose ordering is fully correct. To calculate each score,for each pair of matched observations in the retrieved plan, we first assign a score of 1 if they appear in the same order in the generated plan as in the ground truth, and -1 if the order is reversed. The total score is calculated by averaging the score across all pairs. Backtrack rate is the fraction of consecutive steps between a plan's correctly retrieved waypoints that move backward along the ground-truth route.

First, we compare recursive retrieval to top-K retrieval. We compare these two approaches on a retrieval benchmark constructed from ground truth plans and their associated tasks. We find that at k=10, top-k has slightly higher recall and precision while sacrificing ordering quality. While recursive retrieval sacrifices some recall and precision, it does not find the goal significantly less than the top-K retrieval and preserves ordering significantly better, and we find that this characteristic is favorable for generating coherent plans that a VLA can follow.

RetrievalRecallPrecisionGoal in contextKendall τ\tauBacktrack rate ↓\downarrow
One-shot top-K0.3700.2390.3660.1860.415
Recursive (ReMaP)0.3030.1780.3420.3210.341

We find that at k=10, top-k has slightly higher recall and precision while sacrificing ordering quality. While recursive retrieval sacrifices some recall and precision, it does not find the goal significantly less than the top-K retrieval and preserves ordering significantly better, and we find that this characteristic is favorable for generating coherent plans that a VLA can follow.

“...split semantic/geometric embeddings vs. a single joint embedding …“...semantic-only vs. geometric-only retrieval…”

We evaluate the individual embeddings on GOAT-bench on a subset where only episode 0 of each environment is run.

RetrievalSR (%)SPL (%)Object SRDescription SRImage SR
Geometric only60.4 [51.6, 69.6]31.2 [24.3, 38.9]67.753.859.1
Semantic only73.0 [64.5, 81.0]46.5 [39.5, 53.4]74.768.176.1
Geometric + semantic (ReMaP)74.8 [67.5, 81.6]46.7 [40.3, 53.2]80.867.076.1

Note: these results were edited from the original response, which was run with an older version of the embedding model that did not include a dispersal term in the loss. However, the original results showed similar relative performance.

On GOAT-bench, geometric and semantic components are both important, but the semantic component contributes most to performance. Adding the geometric embedding to semantic-only retrieval increased overall SR from 73.0% to 74.8% and object-goal SR from 74.7% to 80.8%. Semantic-only retrieval outperforms combined retrieval by 1.1 percentage points. However, we also recognize that GOAT-bench can sometimes require only the end-goal to be retrieved, such as description-goal tasks, therefore making semantic embeddings more important than geometric embeddings in these cases.

Furthermore, we provide training curves to justify the decision to use split semantic and geometric embeddings rather than a single learned embedding. We found that when using a single embedding, the model would learn to completely ignore either the geometric or the semantic information, resulting in worse performance overall.

EpochSingle embeddingDual embedding (ours)
00.2200.209
40.2170.208
80.2020.186
120.1840.177
160.1760.167
200.1720.165
Best0.171 (epoch 17)0.162 (epoch 17)

The validation curve also shows that we steadily achieve lower validation loss with split embeddings compared to joint embeddings.

“...learned embeddings vs. frozen/off-the-shelf embeddings under the same retrieval procedure…”

We compare ReMaP with off-the-shelf embeddings on GOAT-bench, including SigLIP, CLIP, and Qwen3-VL. We use a subset of the GOAT-bench benchmark where only episode 0 of each environment is run. Note that this comparison is run with a new regularization term in the loss that encourages dispersion of the embeddings and improves the performance of ReMaP. We will describe this term in the updated manuscript with additional discussion.

Embedding ModelParameter countSR (%) [95% CI]SPL( %) [95% CI]
SigLIP 2375 M70.5 [61.7, 78.6]41.8 [34.3, 49.3]
CLIP428 M74.5 [67.0, 81.3]49.8 [42.5, 56.9]
Qwen3-VL2.1 B76.3 [68.6, 83.1]48.3 [42.1, 54.5]
ReMaP465 M74.8 [67.5, 81.6]46.7 [40.3, 53.2]

Qwen3-VL achieved the highest success rate (76.3%), and CLIP achieved the highest SPL (49.8%). However, Qwen3-VL has 4.5x the number of parameters, and CLIP has a slightly worse success rate. While GOAT-bench provides large-scale planning evaluation, we found GOAT-bench often only requires retrieval of the end goal to succeed. Therefore, we also perform a more thorough evaluation of ReMaP and Qwen3-VL on real-world tasks and provide qualitative results. Overall, we found that the plans generated with Qwen3-VL retrieval contained the end goal of most tasks just as often as the ReMaP plans for those same tasks. However, we found that ReMaP was much more effective at producing plans that were more continuous in nature, meaning that more correctly ordered intermediate observations were present between objectives that were specified in the task. This means that ReMaP’s plans are likely easier for a VLA to follow step-by-step because they are specified more exactly than the plans produced by Qwen3-VL.

[Weakness 3] “...The GOAT-Bench result uses oracle pre-exploration to ensure full scene coverage, converts image goals to language using Gemini-2.5-Pro, and then uses a fast-marching planner…” [Question 2] “Can the authors provide a matched-protocol GOAT-Bench comparison?”

Since submission, we adapted our method to use online exploration rather than oracle pre-exploration, which matches the protocol of the GOAT-Bench benchmark. We found that with this change, we were able to dramatically increase the performance of ReMaP, as now purposeful progress can be made through the environment. We now also outperform most RGB-D-based methods we introduced previously, except for 3D-Mem.

InputMethodSR (%) ↑SPL (%) ↑
RGB-DMTU3D40.812.1
RGB-DMSGNav52.029.6
RGB-D3D-Mem69.148.9
RGBModular GOAT24.917.2
RGBVLMNav16.36.6
RGBDyNaVLM25.510.2
RGBTANGO35.519.5
RGBUni-NaVid38.519.8
RGBReMaP (Ours)65.2 [60.9, 69.8]36.6 [33.2, 40.6]

Note: these numbers were updated to use the latest model with a dispersal loss term. Previously, they were 65.2 and 33.3.

[Weakness 4] “...The real-world experiment is more relevant, but it contains only eight tasks with three trials each, i.e. 24 trials per method…the tasks are explicitly designed to benefit from memory, and a manual tour is collected beforehand to ensure that each task has sufficient memory coverage…” [Question 3] “How robust is the real-world improvement statistically?... It would also be useful to report an endpoint-reaching metric independent of issuing Stop”

It is correct that we adjust the kinds of tasks we use to evaluate ReMaP in the real world, as we are targeting the setting where memory must be leveraged to inform how to complete a task. This setting is specifically called “Iterative vision-language navigation (as introduced in Krantz et al., “Iterative Vision-and-Language Navigation”, 2022). We will clarify that this is the target domain of the method in the final manuscript.

We recognize that the real-world evaluation is more limited than the GOAT-bench evaluation. These results are much harder to scale, especially with the slower baselines and longer tasks, as a single trial can take several minutes. Therefore, these results are more qualitative and for demonstrating the capabilities of ReMaP rather than hard empirical evidence.

However, we do extend the real-world evaluation of ReMaP, introducing 5 new tasks and increasing the number of trials to 39. We report the standard error across tasks.

MethodProgress: ShortProgress: MediumProgress: LongProgress: OverallSuccess: ShortSuccess: MediumSuccess: LongSuccess: Overall
LM-Nav71.7 ± 7.575.3 ± 10.542.7 ± 6.463.9 ± 5.20.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.0
RAVEN51.5 ± 7.353.4 ± 8.534.9 ± 6.747.0 ± 4.413.3 ± 9.10.0 ± 0.00.0 ± 0.05.1 ± 3.6
ReMaP84.5 ± 5.660.3 ± 9.652.7 ± 1.867.3 ± 4.346.7 ± 13.38.3 ± 8.30.0 ± 0.020.5 ± 6.6

We find that ReMaP outperforms all the baselines in success and progress along the task trajectory. ReMaP improves progress along the task trajectory over RAVEN by 20.3 percentage points and over LM-Nav by 3.4 percentage points. In success rate, ReMaP outperforms RAVEN by 15.4 points and LM-Nav by 20.5 points.

Qualitatively, we found that ReMaP and RAVEN had similar failure modes, especially on tasks where abrupt turns had to be made, resulting in failure because the observation did not make the turn visible early enough for the planner to respond. The remaining gap between RAVEN and ReMaP is accounted for by planning performance, as RAVEN often does not spread plan coverage across the task, therefore resulting in poor guidance for the low-level policy. LM-Nav has strong collision avoidance behavior but most often fails at decision points, such as hallway junctions, as the plans mostly draw in geometrically incoherent goals.

We believe that progress does provide a “stop” independent metric to measure if the method reaches the end goal. If the method reaches the end goal, 100% progress will be recorded, regardless of task completion. However, we’d like to emphasize the importance of the “success” metric, as this indicates the method is able to recognize its progress through the task and when it has reached the end goal.

[Question 4] “Some related works are missing”

Thank you for bringing these works to our attention. We will discuss these in the related work section in our final manuscript.

Reviewer_M6ZgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a RAG system for navigation. Specifically they train an embedding model to support that reg instead of using off-the-shelf models for matching. Then they use an existing VLM for planning on the retrieved observations.

Critical strengths

Overall this paper is quite well written and well motivated. The figures and descriptions are clear, and they develop a novel model for training a navigation specific RAG encoder.

Critical weaknesses

This biggest weakness of this paper is some missing baselines and ablations. Specifically as a baselines RANa [1] and Remembr [2] are other RAG based navigation systems which seem as if they could be a drop in replacement in the same setting. This should be discussed or compared against.

However, the biggest contribution of this paper is the embedding model that they train. Yet this model is not properly ablated in the experiments. I would expect to see experiments where they run the same high-level planner, but swap out the system used build the context. I.e. replace their trained model with CLIP, SigLIP, or Qwen3-VL encoder, for image/language embeddings. This would justify all this extra machinery to train the embedder.

If these issues could be addressed I would gladly raise my rating.

[1] Monaci, Gianluca, et al. "RANa: Retrieval-Augmented Navigation." arXiv, 4 Apr. 2025, https://doi.org/10.48550/arXiv.2504.03524. [2] Anwar, Abrar, et al. "Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation." 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025.

Questions for authors

Some peculiar choices with how the images are encoded, such as breaking it into four pieces and encoding the patches along with the whole image. Why were these choices made and could an ablation be introduced to justify their existence? At 301 L_geo and L_sem have not been defined? It’s not quite clear how the training data is constructed. For example in the random walk training data how is the text for the semantic component of the goal determined? For the semantic training, the next step is given label 1 and other steps label 0, Is it possible that some of the previous steps still have the necessary landmark where it's visible in the image just slightly further away? Labeling those as zero seems like a contradictory training signal that would be an issue given this is already a very data-imbalanced problem. Is this correct? How is the Gemini based planner outputting its waypoints exactly? Is it giving coordinates on a top-down map or something highly local to the robot frame.

4 comments

We thank the reviewer for their valuable comments and suggestions. In this response, we will provide new experimental results and answer questions about the method.

[Weakness 1] “...This biggest weakness of this paper is some missing baselines and ablations. … RANa [1] and Remembr [2] are other RAG-based navigation systems …”

We thank the reviewer for bringing these works to our attention. Unfortunately, RANa does not offer open-source code, but we will include this work in our related work discussion.

Remembr is directly compared to in the RAVEN paper and uses the same embedding model, but instead uses captions and text embeddings to retrieve relevant context with a VLM. As Remembr is very similar to RAVEN already and RAVEN demonstrates that it outperforms Remembr across several benchmarks, we believe it is reasonable to use RAVEN as a representative method for other retrieval-based navigation methods.

[Weakness 2] “... model is not properly ablated … run the same high-level planner, but swap out the system used to build the context. I.e. replace their trained model with CLIP, SigLIP, or Qwen3-VL encoder, for image/language embeddings…”

Thank you for your suggestions regarding ablations. We compare ReMaP’s embedding model with CLIP, SigLIP, and Qwen3-VL on the GOAT-bench benchmark.

Note that since submission, we have modified ReMaP to use its own exploration method rather than the pre-exploration method described in the original submission. We will update the manuscript to describe this part of our approach.

Embedding ModelParameter countSR (%) [95% CI]SPL( %) [95% CI]
SigLIP 2375 M70.5 [61.7, 78.6]41.8 [34.3, 49.3]
CLIP428 M74.5 [67.0, 81.3]49.8 [42.5, 56.9]
Qwen3-VL2.1 B76.3 [68.6, 83.1]48.3 [42.1, 54.5]
ReMaP465 M74.8 [67.5, 81.6]46.7 [40.3, 53.2]

Only Qwen3-VL outperforms ReMaP on the GOAT-bench benchmark, albeit with 4 times the number of parameters. While GOAT-bench provides large-scale planning evaluation, we found GOAT-bench often only requires retrieval of the end goal to succeed. Therefore, we also explore the qualitative performance of ReMaP compared to Qwen3-VL on real-world tasks. Overall, we found that the plans that Qwen3-VL produced contained the end goal of most tasks just as often as the ReMaP plans for those same tasks. However, we found that ReMaP was much more effective at producing plans that were more continuous in nature, meaning that more correctly ordered intermediate observations were present between objectives that were specified in the task. This means that ReMaP’s plans are likely easier for a VLA to follow step-by-step because they are specified more exactly than the plans produced by Qwen3-VL.

[Question 1] “...Some peculiar choices with how the images are encoded, such as breaking it into four pieces and encoding the patches along with the whole image…”

Thank you for noting that this aspect needs some clarification. This decision was made in the process of training the embedding model. We include the training curves that were used to make this decision.

Here is the validation loss for the single image and the 2x2 split + full image variant. The training loss for both methods is the same.

EpochSingle image2×2 split + full image
40.5670.454
140.4830.364
240.3990.338
490.2640.215
690.2760.209
940.2120.178
Best0.212 (epoch 79)0.166 (epoch 84)

By providing the split and combined image, we were able to get better attention to the different parts of the image while retaining the overall structure. This kind of tiling approach has been used in foundational works for vision-language model training, including Idefics3 (Laurencon et al., 2024) and Molmo (Deitke et al., 2025). Additionally, we qualitatively found that this helped the model distinguish left and right patches of the image, thereby learning relative positions of landmarks in the observations.

[Question 2] “At 301 L_geo and L_sem have not been defined?”

Thank you for noting this error. We will define Lgeo\mathcal{L}_{\mathrm{geo}} and Lsem\mathcal{L}_{\mathrm{sem}} in the final manuscript.

Let NN be the number of queries and MM the number of targets in a batch. For each head h∈geo,semh \in \\{\mathrm{geo}, \mathrm{sem}\\}, let qihq_i^h denote the embedding of query ii and tjht_j^h the embedding of target jj. The predicted distribution fihf_i^h is the softmax over the in-batch targets, with entries

\frac{\exp\left(\cos(q_i^h, t_j^h)/\tau\right)} {\sum_{r=1}^{M} \exp\left(\cos(q_i^h, t_r^h)/\tau\right)}, \qquad \tau = 0.07.$$ Let $G_i$ and $S_i$ denote the sets of in-batch targets labeled positive for query $i$ under the geometric and semantic heads, respectively. Query–target pairs absent from the dataset are treated as negatives. To form each target distribution, we clamp the labels at zero and normalize across targets. If a query has no in-batch positive for a given head, we assign it a uniform target distribution and include it in that head's average loss. For the geometric head, let $k_{ij} \ge 1$ denote the number of trajectory steps from query $i$ to a positive target $j$. The immediate next step receives label $1$, and each additional step multiplies the label by $\gamma = 0.7$. The resulting target distribution is $$p_{ij}^{\mathrm{geo}} = \begin{cases} \dfrac{\gamma^{k_{ij}-1}}{\sum_{r \in G_i} \gamma^{k_{ir}-1}} & \text{if } j \in G_i, \\\\ 0 & \text{if } G_i \neq \varnothing \text{ and } j \notin G_i, \\\\ 1/M & \text{if } G_i = \varnothing. \end{cases}$$ For the semantic head, a positive target contains the landmarks of the next sub-goal. We set every positive label to $1$, giving all positive targets equal mass after normalization. The resulting target distribution is $$p_{ij}^{\mathrm{sem}} = \begin{cases} \dfrac{1}{\lvert S_i \rvert} & \text{if } j \in S_i, \\\\ 0 & \text{if } S_i \neq \varnothing \text{ and } j \notin S_i, \\\\ 1/M & \text{if } S_i = \varnothing. \end{cases}$$ For each head, we compute the forward KL divergence per query and average over the batch. The training objective sums the two losses. $$\mathcal{L}_h = \frac{1}{N} \sum_{i=1}^{N} D_{\mathrm{KL}}(p_i^h \parallel f_i^h)$$ $$\mathcal{L} = \mathcal{L}_{\mathrm{geo}} + \mathcal{L}_{\mathrm{sem}}$$ (It doesn't seem to want to render, hopefully it renders for you!) **[Question 3] “It’s not quite clear how the training data is constructed.”** We provide more detail about dataset construction in Appendix E.2, as mentioned on line 307 of the manuscript. We use two kinds of data to train our model. First, we use ground-truth trajectories from simulation and the real world, which we label with tasks post hoc using a VLM (Gemini 2.5 Pro). Second, we build graphs from scene datasets using ViNT and traverse them with A* to produce plans. A VLM (Gemini 3.1 Pro Preview) provides the A* heuristic by assessing whether any neighbor of the current node fits into the current plan. This mix lets us learn connectivity from the trajectories and object-reaching behavior from a small but sample-efficient set of graph-based plans. Appendix E.2 provides the prompts we used and a breakdown of the training data. We will add more detail to the main text and direct readers to the appendix in the final manuscript. **[Question 4] “... in the random walk training data how is the text for the semantic component of the goal determined?...Is it possible that some of the previous steps still have the necessary landmark where it's visible in the image just slightly further away? Labeling those as zero seems like a contradictory training signal…”** For the random walk data, as well as the ground truth trajectory data, the semantic landmarks are determined by generating descriptions of the image observations with a VLM. These descriptions densely capture the landmarks in the scene, but can miss objects if they are partially visible or distant, as these are failure modes of the underlying VLM. Therefore, even landmarks that are more distant but still captured in the generated description will be given a label of 1. Only objects that are in the image but are not captured in the language label would be given a false negative label. We hope this has clarified this part of the method and will update our manuscript to make this more apparent. **[Question 5] “How is the Gemini-based planner outputting its waypoints exactly?”** The planner does not directly output waypoints. As described in line 323, the planner produces a plan consisting of image sub-goals and language instructions that is then executed by a low-level policy. The image sub-goals and language instructions are provided to the low-level policy step-by-step. For real-world experiments, this low-level policy is OmniVLA, and for GOAT-bench experiments, we use a fast-marching planner.

Thank you again for taking the time to review our work and provide valuable comments and suggestions. As the rebuttal period comes to a close, we would sincerely appreciate you taking the time to review our response and let us know if you have any remaining concerns; we would be happy to address them.

As the rebuttal period is a few hours from ending, we would like to thank you again for your thoughtful feedback, which has helped us strengthen the paper. We would still appreciate hearing whether our responses and additional experiments have addressed your concerns and whether you have any remaining questions. If you feel that we have resolved the main issues you raised, we would be grateful if you would consider updating your assessment. Again, thank you for your time and consideration!

Reviewer_RwAjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReMaP, a retrieval-augmented multimodal planning framework for long-horizon robot navigation using unstructured visual memory. The key idea is to decouple memory retrieval from navigation planning, enabling a robot to efficiently retrieve task-relevant observations from previously collected experience. Experiments on GOAT-Bench and real-world navigation tasks demonstrate promising performance and retrieval efficiency.

Critical strengths

1.The paper presents an interesting approach to long-horizon robot navigation by integrating learned multimodal retrieval with VLM-based planning. The combination of semantic and geometric information provides a promising way to retrieve task-relevant observations from unstructured robot memory.

2.The paper provides encouraging experimental results on both GOAT-Bench and real-world robot navigation tasks. The improvements in retrieval efficiency and the real-world tasks demonstrate the practical potential of the proposed framework.

Critical weaknesses

1.Section 4.1 explicitly identifies recursive retrieval and split geometric/semantic embeddings as two key mechanisms. However, the paper lacks ablations isolating their individual contributions. Additional experiments would help better understand the effectiveness of these designs.

2.Although long-horizon navigation is one of the central motivations of the paper, ReMaP achieves no successful completion across all six trials of the two long-horizon real-world tasks (Appendix A, Table 4). Further analysis would help clarify the limitations of the proposed framework on challenging long-horizon tasks.

3.More representative examples, especially failure cases, would help explain the behavior of the proposed framework.

Questions for authors

1.Could the authors provide ablation studies on recursive retrieval and the separate geometric and semantic embedding components to better understand their individual contributions?

2.What are the main reasons for the failures on the real-world long-horizon tasks? Are they primarily caused by retrieval errors, planning errors, or low-level navigation failures?

3.While the reported retrieval efficiency is impressive, could the authors also provide an end-to-end runtime breakdown, including retrieval, VLM plan generation, and task monitoring? This would offer a more complete understanding of the computational efficiency of the overall framework.

4 comments

We thank the reviewer for their insightful comments and suggestions. In this response, we will provide new experimental results, clarifications on our method, and a qualitative discussion on ReMaP failure cases.

[Weakness 1] “...The paper lacks ablations isolating their individual contributions [recursive retrieval / split embeddings]...”

We agree that ablations of our design choices are necessary for justifying their effectiveness. To evaluate the retrieval capabilities of ReMaP as well as ablate several design choices, we construct a retrieval benchmark by using the same labeling techniques that we used to generate the training data for ReMaP. We label two new and unseen datasets, NavWareSet (Brayan et al., 2026) and OpenLORIS (Shi et al., 2019). Each benchmark example consists of a task, a start image, and a corresponding ground-truth plan that is an ordered list of observations, sampled every 2m or every time a turn greater than 45 degrees is made. To avoid overfitting to camera artifacts and to better represent how retrieval models would function in the real world, the observations belonging to the ground-truth route that is being assessed are held out during evaluation. Instead, we use a geometric matcher to confirm that the retrieved observations correspond to observations in the ground-truth plan. We discard tasks that are less than 1m in length. In total, the benchmark contains 2,064 non-overlapping tasks (1,847 NavWareSet, 217 OpenLORIS). The path lengths range from 1m to 79m, with 1,225 tasks less than 5m in length, 613 tasks between 5m and 15m in length, 191 tasks 15m to 40m in length, and 35 tasks 40m+ in length.

We measure recall, precision, whether the goal is in the retrieved context, off-route rate (how often retrievals are not on the ground truth route), and Kendall τ\tau, which represents the correctness of the plan ordering, with -1 being a fully reversed plan and 1 being a plan whose ordering is fully correct. To calculate each score, for each pair of matched observations in the retrieved plan, we first assign a score of 1 if they appear in the same order in the generated plan as in the ground truth, and -1 if the order is reversed. The total score is calculated by averaging the score across all pairs. Backtrack rate is the fraction of consecutive steps between a plan's correctly retrieved waypoints that move backward along the ground-truth route.

First, we compare recursive retrieval to top-K retrieval. We compare these two approaches on a retrieval benchmark constructed from ground-truth plans and their associated tasks. We find that at k=10, top-K has slightly higher recall and precision while sacrificing ordering quality. While recursive retrieval sacrifices some recall and precision, it does not find the goal significantly less than the top-K retrieval and preserves ordering significantly better, and we find that this characteristic is favorable for generating coherent plans that a VLA can follow.

RetrievalRecallPrecisionGoal in contextKendall τ\tauBacktrack rate ↓\downarrow
One-shot top-K0.3700.2390.3660.1860.415
Recursive (ReMaP)0.3030.1780.3420.3210.341

We evaluate the individual embeddings on GOAT-bench on a subset where only episode 0 of each environment is run. We also report the success on the different kinds of tasks in the GOAT-bench benchmark. Note that this comparison was run without a new regularization term that improved the performance of ReMaP. We are currently running these experiments, and episode 0 results are available in the learned vs. frozen embedding comparison.

RetrievalSR (%)SPL (%)Object SRDescription SRImage SR
Geometric only56.1 [46.6, 65.7]25.4 [17.9, 34.3]67.744.055.7
Semantic only67.3 [58.7, 75.3]33.9 [26.9, 41.3]79.858.262.5
Geometric + semantic (ReMaP)68.0 [60.2, 75.2]36.0 [29.3, 43.1]73.757.172.7

On GOAT-bench, geometric and semantic components are both important, but the semantic component contributes most to performance. However, we also recognize that GOAT-bench can sometimes require only the end-goal to be retrieved, therefore making semantic embeddings more important than geometric embeddings.

Furthermore, we provide training curves to justify the decision to use split semantic and geometric embeddings rather than a single learned embedding. We found that when using a single embedding, the model would learn to ignore either the geometric or the semantic information, resulting in worse performance overall.

EpochSingle embeddingDual embedding (ours)
00.2200.209
40.2170.208
80.2020.186
120.1840.177
160.1760.167
200.1720.165
Best0.171 (epoch 17)0.162 (epoch 17)

The validation curve also shows that we steadily achieve lower validation loss with split embeddings compared to joint embeddings.

[Weakness 2] “...ReMaP achieves no successful completion across all six trials of the two long-horizon real-world task…”

We believe this concern is due to a miscommunication in the nature of the tasks we evaluate on. All of the tasks we evaluate ReMaP and the baselines on in the real world are long-horizon tasks. They all require multiple subgoals, leverage information from memory, and do not have the goals immediately observable from the starting position. We use “short”, “medium”, and “long” to denote the relative lengths and complexity of the tasks. We will clarify this in the updated manuscript as the length and estimated number of subgoals to complete the task.

The “long” long-horizon tasks are complex and require several subgoals to correctly complete the task. Therefore, these are very challenging to complete, which is why we also include plan coverage and progress as metrics to glean richer information about how each of the methods performed.

[Weakness 3] “...More representative examples, especially failure cases, would help explain the behavior of the proposed framework…”

We thank the reviewers for noting that we should include this discussion.

One of the main failure modes we observe is the retrieval of semantically similar but geometrically more distant contexts. For example, for the task “Go down the hallway and stop at the blue recycling bins on the left”, many recycling bins are captured in the initial tour data. As a result, context retrieval will sometimes pull the wrong bin into context. While this does not necessarily result in failure, it provides incorrect subgoals to the low-level policy.

Another mode that can occur is failure to update to the next subgoal, even when it is complete. To help mitigate this problem, we replan at regular intervals, which allows the model to leverage new context to improve its plan and get unstuck from such cases. However, this failure mode can still affect the overall performance of the method and presents an opportunity to improve progress detection with ReMaP in future work.

Finally, many failures can be attributed to the low-level policy (OmniVLA for ReMaP, RAVEN, and LM-Nav) not being able to successfully follow the subgoals provided in the plan. For instance, the policy may incorrectly follow a command such as “turn left at the fork” by going right instead. This results in the downstream task completion being incorrect.

We will certainly integrate this discussion into our updated manuscript.

[Question 3] “... could the authors also provide an end-to-end runtime breakdown, including retrieval, VLM plan generation, and task monitoring?...”

We provide a full breakdown of the inference time for each component of the deployed system, which we host on an RTX 4090 GPU with context sizes of 10, 20, and 30, testing across 15 different tasks.

ComponentTime (s)
Embedding (per element in memory)0.016
Retrieval (per element in context)0.064
VLM Planning (context size 10)57.8
VLM Planning (context size 20)69.5
VLM Planning (context size 30)71.2
Low-level policy (OmniVLA)0.112
Task completion check7

Note that between the initial task embedding and subsequent task embeddings, where the most recent elements of context are integrated, the inference time remains ~0.064 seconds. The longest component is plan generation, which will only improve as the speed of API-based VLMs improves.

Reviewer_RwAjOpenReview ↗

Post-rebuttal assessment

Thank you for the detailed response and additional experiments. The new retrieval benchmark and ablation studies substantially address my main concern regarding recursive retrieval and the split geometric and semantic embeddings. The discussion of failure cases and the runtime breakdown are also helpful.

I still have reservations about the lack of successful completion on the most challenging long-horizon tasks and the substantial latency associated with VLM planning and task monitoring. Nevertheless, the response provides meaningful additional evidence supporting the core design choices. I therefore intend to raise my rating from Weak Reject to Weak Accept.

Thank you again for taking the time to review our work and for reconsidering your rating. We are glad our additional experiments addressed some of your main concerns, and we agree that the lack of successful completion on challenging long-horizon tasks and latency remain important limitations. We will ensure these are made explicit in the final manuscript.

We sincerely appreciate your thoughtful engagement during the review process and for allowing us to strengthen our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We sincerely thank all the reviewers for their detailed comments and suggestions. We believe they have helped us greatly strengthen our paper. We have done our best to address all the concerns in the individual responses. We summarize the main changes here and provide the updated manuscript, which is color-coded for each of the reviewers: RwAj → green, M6Zg → blue, and W1eN → red.

New experiments:

  • The embedding model comparisons have been added to Section 5.5
  • The analysis of the geometric and semantic embeddings independently has been added to Section 5.5
  • We discuss other design decisions (recursive retrieval, dual vs. split embeddings, and image tiling) in Appendix E and reference them in the main text
  • We update our GOAT-bench results to use the latest version of our embedding model and our frontier-based exploration method (described in Section 4.2) in Section 5.2
  • We update our real-world experiments to expand from 8 tasks to 13 and update our discussion accordingly in Section 5.3 and provide the expanded results in Appendix A.2

Details and clarifications:

  • We update our related work to include work brought forward by the reviewers (Section 2)
  • We include details about the training objective for the embedding model (Section 4.1)
  • We describe our frontier-based exploration method for GOAT-bench in Section 4.2
  • We include a breakdown of runtime speed of each component in Appendix B.1 and reference this section in the main text
  • We discuss ReMaP’s failure modes in Appendix A.3 and reference this section in the main text

As the rebuttal period comes to a close, please let us know if you have any remaining questions. If you believe we have sufficiently addressed your concerns, please consider increasing your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.