acceptodds
Withdrawn from ICLR 2027

Remembering Is Not Navigating: Goal Recurrence and Closed-Loop Success in City-Scale Drone Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language navigation agents are increasingly fine-tuned on the benchmarks that rank them, and a higher closed-loop success rate is read as evidence that an agent has learned to navigate. Success can also come from memory: if the places an agent is sent to also appear in its training data, recalling where a description led is enough. EmbodiedNav-Bench, a city-scale drone benchmark, is exposed to this risk: it ships without a train/test split, so each fine-tuning study draws its own, and nothing checks what the held-out goals share with training. We measure how much success memory alone can buy there. Holding out every tenth episode, we find that held-out goals are rarely new: for 60 of them, one training episode has both a near-duplicate instruction and a goal inside the success radius. On 101 held-out episodes flown in the benchmark's simulator under our harness, a controller that flies from the start pose to the remembered endpoint of the matching training instruction succeeds on 83 of the episodes it flies from memory and on 58 overall, with a learned policy acting on the rest, against 22 for an imitation-learned policy and 32 for always flying forward; without a near-duplicate instruction or a nearby training goal, it does no better than flying forward. Gains from training fixes need the same scrutiny: an offline exposure-bias correction with geometry-verified preference optimization reduces the harm of the agent's own action history in offline next-action accuracy, yet same-budget controls without the preference signal give reductions we cannot distinguish from it, and its closed-loop effect is untested. We release goal-disjoint episode lists and recommend reporting success on them beside a retrieval baseline.

Primary area:
datasets and benchmarks
Keywords:
vision-language navigation, train-test overlap, aerial navigation, exposure bias, preference optimization, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AUNwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies EmbodiedNav-Bench, a benchmark for drone navigation in a single simulated city. The benchmark has no official training split, so the authors hold out every 10th episode as a test set and train on the rest. They find that most test goals already appear in training, with a near-identical instruction and a goal in almost the same place. To show what this is worth, they build a simple controller that ignores the images, looks up the most similar training instruction, and flies to where that episode ended. It succeeds more often than a trained imitation policy and an always-flying-forward policy. On goals that are truly new, it does no better than flying forward. The authors then examine an offline training method that stops the policy from copying its own past actions. The method improves offline accuracy, but simpler controls with the same training budget do just as well. The authors conclude by recommending reporting results on test episodes with new goals, next to a lookup baseline, and they release lists of such episodes.

Critical strengths
  1. The core question that the authors investigate is meaningful: whether a high success rate on a navigation benchmark means that an agent has learned to navigate, or it just remembers where goals were. The paper is also very honest about its limitations. The authors state clearly what they do not claim. For instance, that they have not shown a better navigator, and that the offline method was never tested in the simulator.
  2. The idea behind designing the probe is good. It ignores the images entirely and only looks up where similar instructions led in training. This makes the benefit of memory easy to see.
  3. Throughout the paper, the reported statistics are thorough. Nearly every result has an uncertainty interval. Example episodes in the figures are picked by fixed rules, not “cherry-picked”. Evaluations that vary between runs are repeated. The authors also do a good job at verifying the claims they make.
Critical weaknesses
  1. The authors state that their entry to the navigation track of the UrbanCup 2026 challenge ranked second, which breaks anonymity.
  2. The novelty aspect of the paper is not great. The authors use known methods (unimodal/lookup baselines, train-test overlap) to analyze a single benchmark in a closed-loop setting. Do the findings extend beyond EmbodiedNav-Bench? Measuring recurrence jointly in the instruction and the goal position + running the lookup as a closed-loop controller in the simulator are interesting but rather incremental in the grand scheme of things. The analysis itself is rigorous, thorough, and honest, but the novelty does not match ICLR standards.
  3. The probe can rely on information the other learned agents don’t have. Section 4.2 + the limitations section state that it knows its exact starting position from the episode file. So it can compute exactly where a remembered goal lies and fly there (this is possible because every episode uses the same map coordinates). Meanwhile, the learned policies are never told their position. The authors do say in the limitations section that the shared coordinate frame is “the property measured”, but this doesn’t really address the problem. A fine-tuned model would first have to figure out its location from the images before it would be able to use a goal that it has remembered.
  4. The benchmark that the authors use was released as a single evaluation set with no “official” training split. If you hold out every tenth episode from a single city where many episodes share the same instruction, it is no wonder that most test goals will appear in training almost by default. I am therefore not convinced that this is a relevant problem to begin with. The paper says that “each fine-tuning study draws its own” split, but I didn’t really find any examples of such studies. Which published papers fine-tune on EmbodiedNav-Bench, and could their gains perhaps come from repeated goals? The UrbanCup challenge would be the most relevant case to check.
  5. The learned baselines are weak, and the test setup is easy. An episode counts as a success as soon as the drone comes within 15 m of the goal, even if it would have flown past. The drone can also fly through buildings, and crashes don’t end the episode. It is no wonder that simply flying forward succeeds 31.7% of the time, more often than the imitation policy (21.8%). Beating these policies with memory is therefore not very informative. The authors could compare against any of the agents tested in the original benchmark paper. Finally, the ORPO policy used as the fallback turns left on 43% of its steps and never turns right. That looks like a policy that doesn’t work in closed loop, not really a fair example of a “learned navigator”.
  6. The conclusions about new goals rely on very few episodes. The paper says the probe does no better than flying forward when the goal is new. But this is based on only 5, 10 and 16 episodes, depending on how “new” is defined. With so few episodes, the uncertainty is very large. For the 5 fully new goals, the 95% interval runs from 1% to 72%. The closed-loop results also come from a single run. In addition, the authors admit they no longer have the training record for the fallback policy, so they cannot confirm that it was never trained on the test episodes. They also chose the final probe settings after an earlier run on the same test episodes, which may have tuned the probe to them.

Lesser points

  1. The contributions listed in the introduction should stand on their own without relying on references to the rest of the paper. The reader should get the crux of the paper’s contributions before reading the succeeding sections.
  2. The paper reads like two separate papers. The first half shows that test goals repeat training goals. The second half (Section 5) tests an offline training method meant to stop the policy from copying its own past actions. Its main finding is negative. The gain cannot be credited to the method’s key idea, and the method is never tested in the simulator. This has little to do with goal recurrence.
  3. The paper is hard to read. Almost every sentence contains several numbers, which makes the main points easy to miss.
Questions for authors
  1. Do the findings extend beyond EmbodiedNav-Bench?
  2. Which published papers fine-tune on EmbodiedNav-Bench?
Reviewer_4meJOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits goal recurrence in EmbodiedNav-Bench and tests an instruction-to-endpoint retrieval probe under a custom closed-loop harness. It shows that remembering training goals can account for substantial success and proposes goal-disjoint evaluation subsets. A separate offline study examines exposure-bias correction: dependence on self-generated action history is reduced, but matched controls do not support attributing the improvement specifically to preference optimization. The contribution is a benchmark audit and diagnostic study rather than a new navigation policy.

Critical strengths
  1. The recurrence analysis is backed by a closed-loop retrieval probe. This connects overlap in instructions and goals to an actual source of benchmark success, rather than treating overlap alone as sufficient evidence.
  2. The offline study includes continued-training controls, seed analysis, and explicit limits on attribution. These controls are useful for checking whether the reported improvement requires the proposed preference signal.
Critical weaknesses
  1. The paper includes imitation, always-forward, and offline training controls, but its closed-loop comparison combines endpoint memory, a learned fallback, and unequal localization access. Retrieval with a nonlearned fallback and stronger navigators with and without training-goal memory should be evaluated on the same episodes with matched pose access. Why is a relevant memory-based aerial-navigation method such as CityNavAgent not included or adapted? If task or action interfaces prevent a direct comparison, the paper should explain the incompatibility and provide a matched alternative. The recurrence measurements stand independently, but the numerical advantage of memory needs these controls.

  2. Only five grid episodes and 21 episodes in the full held-out pool are goal-disjoint. The 1/5 result has a 95% interval of [0.5%, 71.6%]; matching always-forward on five episodes says little about whether the methods generalize equally well. A reliable evaluation requires a larger split defined by goal clusters or spatial regions before training, with sensitivity to recurrence thresholds and paired outcomes. A recurrence audit on another aerial VLN dataset would also test how specific the finding is to this single-city setting.

  3. The continued-SFT and negative-sampling controls are useful, and the authors do not overattribute their shared improvement to ORPO. The unresolved issue is whether reduced history harm on logged expert images helps when the policy visits its own states. I would like to see imitation, continued-SFT, and ORPO models evaluated on the same recurrence-stratified closed-loop episodes, with matched observations and execution settings. Without that experiment, the offline results support a separate diagnostic finding, with limited evidence about navigation performance.

Questions for authors

Please see the weaknesses.

Reviewer_iNYtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether goal recurrence in EmbodiedNav-Bench allows navigation success to be achieved through memorization rather than visual grounding. The author analyze instruction and endpoint overlap, construct a retrieval-based controller, and propose goal-disjoint evaluation subsets. They also examine an offline exposure-bias correction using ORPO, showing that improvements in self-history action accuracy cannot be clearly attributed to the preference signal.

Critical strengths
  1. This paper provides valuable insights. It addresses an important evaluation issue by demonstrating substantial instruction and goal-location recurrence between training and held-out episodes.
  2. The retrieval-based closed-loop experiments provide concrete evidence that memorized goal localizations can yield high success rates without visual observations.
  3. The authors present careful attribution analyses, including confidence intervals, multiple training seeds, continued SFT controls, and alternative preference construction strategies.
Critical weaknesses
  1. The goal-disjoint evaluation is statistically insufficient, only 5 episodes in the 101-episode closed-loop subset satisfy the strict goal-disjoint criterion.
  2. The analysis does not directly establish that learned navigation policies exploit the identified memorization shortcut. Although explicit retrieval performs well, the experiments do not demonstrate whether standard fine-tuned VLM navigators achieves their success through similar instruction-endpoint memorization.
  3. The offline exposure-bias analysis is not directly connected to closed-loop navigation performance. Self-generated action histories are evaluated along expert trajectories, and no paired closed-loop comparison establishes whether the observed reduction in history-related errors improves actual navigation success.
Questions for authors
  1. Can the authors evaluate the retrieval controller and learned navigation policies on the same episodes under more closely matched input and control conditions? In particular, how much of the retrieval advantage remains when exact pose information and collision-permissive movement are restricted?
  2. Can a larger goal-disjoint evaluation split be constructed through goal clustering or spatially separated sampling? The current five-episode closed-loop subset does not provide a reliable estimate of generalization to unseen goals.
  3. Can the authors provide paired per-episode comparisons between retrieval-based navigation and the imitation policy, particularly on episodes with recurring and non-recurring goals? This would help distinguish the effect of memorization from differences in episode difficulty.
  4. How sensitive are the recurrence statistics and retrieval results to the instruction-similarity threshold, spatial-distance threshold, and train/test partition?
  5. Can the authors clarify whether the learned fallback policy was trained exclusively on the designated training pool?
  6. Can the authors provide a paired closed-loop evaluation of imitation and ORPO-trained policies on identical episodes? The current offline analysis does not establish whether reduced history-related errors translate into improved navigation.
  7. What is the intended connection between the goal-recurrence analysis and the ORPO attribution study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.