- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies EmbodiedNav-Bench, a benchmark for drone navigation in a single simulated city. The benchmark has no official training split, so the authors hold out every 10th episode as a test set and train on the rest. They find that most test goals already appear in training, with a near-identical instruction and a goal in almost the same place. To show what this is worth, they build a simple controller that ignores the images, looks up the most similar training instruction, and flies to where that episode ended. It succeeds more often than a trained imitation policy and an always-flying-forward policy. On goals that are truly new, it does no better than flying forward. The authors then examine an offline training method that stops the policy from copying its own past actions. The method improves offline accuracy, but simpler controls with the same training budget do just as well. The authors conclude by recommending reporting results on test episodes with new goals, next to a lookup baseline, and they release lists of such episodes.
Critical strengths
- The core question that the authors investigate is meaningful: whether a high success rate on a navigation benchmark means that an agent has learned to navigate, or it just remembers where goals were. The paper is also very honest about its limitations. The authors state clearly what they do not claim. For instance, that they have not shown a better navigator, and that the offline method was never tested in the simulator.
- The idea behind designing the probe is good. It ignores the images entirely and only looks up where similar instructions led in training. This makes the benefit of memory easy to see.
- Throughout the paper, the reported statistics are thorough. Nearly every result has an uncertainty interval. Example episodes in the figures are picked by fixed rules, not “cherry-picked”. Evaluations that vary between runs are repeated. The authors also do a good job at verifying the claims they make.
Critical weaknesses
- The authors state that their entry to the navigation track of the UrbanCup 2026 challenge ranked second, which breaks anonymity.
- The novelty aspect of the paper is not great. The authors use known methods (unimodal/lookup baselines, train-test overlap) to analyze a single benchmark in a closed-loop setting. Do the findings extend beyond EmbodiedNav-Bench? Measuring recurrence jointly in the instruction and the goal position + running the lookup as a closed-loop controller in the simulator are interesting but rather incremental in the grand scheme of things. The analysis itself is rigorous, thorough, and honest, but the novelty does not match ICLR standards.
- The probe can rely on information the other learned agents don’t have. Section 4.2 + the limitations section state that it knows its exact starting position from the episode file. So it can compute exactly where a remembered goal lies and fly there (this is possible because every episode uses the same map coordinates). Meanwhile, the learned policies are never told their position. The authors do say in the limitations section that the shared coordinate frame is “the property measured”, but this doesn’t really address the problem. A fine-tuned model would first have to figure out its location from the images before it would be able to use a goal that it has remembered.
- The benchmark that the authors use was released as a single evaluation set with no “official” training split. If you hold out every tenth episode from a single city where many episodes share the same instruction, it is no wonder that most test goals will appear in training almost by default. I am therefore not convinced that this is a relevant problem to begin with. The paper says that “each fine-tuning study draws its own” split, but I didn’t really find any examples of such studies. Which published papers fine-tune on EmbodiedNav-Bench, and could their gains perhaps come from repeated goals? The UrbanCup challenge would be the most relevant case to check.
- The learned baselines are weak, and the test setup is easy. An episode counts as a success as soon as the drone comes within 15 m of the goal, even if it would have flown past. The drone can also fly through buildings, and crashes don’t end the episode. It is no wonder that simply flying forward succeeds 31.7% of the time, more often than the imitation policy (21.8%). Beating these policies with memory is therefore not very informative. The authors could compare against any of the agents tested in the original benchmark paper. Finally, the ORPO policy used as the fallback turns left on 43% of its steps and never turns right. That looks like a policy that doesn’t work in closed loop, not really a fair example of a “learned navigator”.
- The conclusions about new goals rely on very few episodes. The paper says the probe does no better than flying forward when the goal is new. But this is based on only 5, 10 and 16 episodes, depending on how “new” is defined. With so few episodes, the uncertainty is very large. For the 5 fully new goals, the 95% interval runs from 1% to 72%. The closed-loop results also come from a single run. In addition, the authors admit they no longer have the training record for the fallback policy, so they cannot confirm that it was never trained on the test episodes. They also chose the final probe settings after an earlier run on the same test episodes, which may have tuned the probe to them.
Lesser points
- The contributions listed in the introduction should stand on their own without relying on references to the rest of the paper. The reader should get the crux of the paper’s contributions before reading the succeeding sections.
- The paper reads like two separate papers. The first half shows that test goals repeat training goals. The second half (Section 5) tests an offline training method meant to stop the policy from copying its own past actions. Its main finding is negative. The gain cannot be credited to the method’s key idea, and the method is never tested in the simulator. This has little to do with goal recurrence.
- The paper is hard to read. Almost every sentence contains several numbers, which makes the main points easy to miss.
Questions for authors
- Do the findings extend beyond EmbodiedNav-Bench?
- Which published papers fine-tune on EmbodiedNav-Bench?