- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces I2O2I-Nav, a benchmark to evaluate long-horizon navigation from indoor to outdoor environments and back indoors, aiming to assess how heterogeneous navigation capabilities compose under continuous execution. The benchmark comprises 20 reconstructed real-world scenes and 1,374 evaluation episodes, with corresponding 3DGS reconstructions and aligned collision geometry. Evaluation is done on four tasks: exiting a building, outdoor navigation, entering a building, and open-vocabulary object navigation. These are evaluated both independently and in continuous sequences, where each task starts from the position and orientation reached by the previous one. The authors also introduce MODULAR-Agent, which coordinates frozen navigation policies through subtask planning, completion verification, and recovery. Results show improvements over the underlying policies on individual tasks and over a fixed policy chain on composed missions, although success on complete missions remains low.
Critical strengths
-
Scale and quality of the benchmark. The construction of 20 real-world reconstructed scenes, covering approximately 115,603 m² and supporting 1,374 evaluation episodes, represents a substantial data collection and curation effort. The combination of 3DGS reconstructions, aligned collision geometry, and semantic annotations provides a valuable resource for evaluating navigation across indoor and outdoor environments. I consider the continuous evaluation across stage transitions particularly useful, since it exposes failures that evaluating each task independently would miss.
-
Practical framework for coordinating existing policies. MODULAR-Agent coordinates a frozen set of navigation policies through subtask planning, completion verification, and recovery. I find this a useful direction because it allows existing navigation capabilities to be composed without retraining the underlying policies. The improvements over a fixed policy chain support the value of this approach for longer missions and make it relevant to the future deployment of navigation systems.
Critical weaknesses
-
Evaluations could be more complete. Recent works have introduced integrated navigation systems operating in indoor and outdoor environments, such as NaVILA [1], which combines general language-guided navigation with low-level robot locomotion, and OmniVLA [2], which supports navigation conditioned on language, images, and goal poses. Evaluating these systems under the same task and information conditions would provide a more complete comparison with MODULAR-Agent, whose full-mission performance is currently compared only against a fixed chain of the same underlying policies. Furthermore, the results would benefit from confidence intervals, particularly for the full-mission evaluation, where the reported improvement corresponds to seven successful episodes compared with two for the fixed chain.
-
MODULAR-Agent should be explained more clearly. The main paper introduces its components, but the actual execution procedure remains relatively abstract. A concise algorithm or worked example would help explain what information the VLM receives, how subtasks are generated, when completion is verified, and how failed execution triggers recovery. This would make the framework easier to understand.
-
Lack of real-world validation. The benchmark is motivated by practical service and delivery scenarios, and the proposed framework coordinates existing policies towards completing such missions. However, all reported experiments use a collision-constrained kinematic agent in simulation. Real-world experiments involving transitions between domains, such as indoor to outdoor, would help demonstrate the benchmark’s relevance to developing deployable policies that handle domain shifts. Even a limited physical robot experiment covering these transitions would substantially strengthen the work.
Minor weaknesses
- The related work section is brief and would benefit from a more detailed comparison with the closest benchmarks, particularly NavVerse and OmniNavBench, to clarify the additional challenges introduced by the complete indoor-to-outdoor-to-indoor mission.
- At L368 and in Table 4, “Omninav + Socialnav” should be written consistently as “OmniNav + SocialNav.”
- At L337, the reported MODULAR-Agent success rate for OVON is 33.00%, whereas Table 2 reports 40.50%.
- Overall, the paper could benefit from tighter writing. Some passages repeat the same points, particularly the literature overview (L078–086, L103–136), state continuity (L156–158, L221–223, L297–298), and headline results (L357–358, L433–438; L375–376, L441–443). Condensing these passages would leave more space for explaining the agent’s execution procedure.
[1] Cheng et al. NaVILA: Legged Robot Vision-Language-Action Model for Navigation. RSS, 2025. https://www.roboticsproceedings.org/rss21/p018.html
[2] Hirose et al. OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation. ICRA, 2026. https://arxiv.org/abs/2509.19480
Questions for authors
- Could the authors provide confidence intervals for the reported results, particularly the full-mission improvement corresponding to seven successful episodes compared with two for the fixed chain?
- Could the authors explain MODULAR-Agent’s execution procedure more concretely, including the VLM’s inputs and outputs, subtask generation, and how verification triggers recovery? An algorithm or worked example would be helpful.
- Does the episode’s wall-clock timer continue running while MODULAR-Agent reasons, verifies completion, or waits for navigation-policy responses, or is it paused during these calls? Please clarify whether inference runs asynchronously with simulator execution and whether the reported timeout includes all planning and inference latency.
I lean towards weak acceptance. I consider the benchmark a valuable contribution, particularly its reconstructed environments and continuous evaluation across navigation stages. Although the evaluation could be broader and the agent’s execution procedure needs clarification, I believe the benchmark’s usefulness and the effort invested in its construction support acceptance. Addressing the questions above would strengthen my assessment.
3 comments
Part 2 of 2
C4. Wall-Clock Timeout, Inference Latency, and Asynchronous Execution (Q3)
The episode clock runs continuously. Planning, verification, policy inference, and waiting for responses all count toward the timeout. The simulator continues executing actions while inference runs asynchronously (Section 4.1 and Appendix F.4).
C5. Comparison with NavVerse and OmniNavBench (Minor W1)
Section 2.2 now compares the three benchmarks directly. NavVerse follows a robot through an exit and along an outdoor route. OmniNavBench tests composite instructions across six navigation task families and multiple robot embodiments. In I2O2I-Nav, the outdoor route ends near a second building. From the pose it actually reaches, the robot must discover the entrance, cross the threshold, and find the requested object inside. The three evaluation levels let us examine the individual skills, a single handoff, and the complete journey.
C6. Naming Consistency, OVON SR, and Writing Concision (Minor W2–W4)
Thank you for catching the naming and numerical inconsistencies. Section 4.1 now defines the abbreviation “O + S” used in Table 4 as “OmniNav + SocialNav.” We corrected the OVON result in Section 4.2 to match Table 2. In the revised two-run evaluation, the OmniNav-based agent achieves 56.00% OVON SR. The 40.50% entry is for the Uni-NaVid-based agent.
We shortened the literature overview, consolidated the explanation of state continuity, and removed repeated summaries of the main results. Section 3.6 uses the space to show how the agent verifies entry and recovers from an early stop.
We appreciate your support.
Dear Reviewer LntP,
Thank you again for your detailed review. As the discussion period ends in a few hours, we would like to kindly remind you of our responses above, which include new experiments addressing each of your comments. We would greatly appreciate it if you could let us know whether they address your concerns, and, if so, consider updating your score.
Best regards, The Authors
Part 1 of 2
Dear Reviewer LntP,
Thank you for recognizing the value of evaluating stage transitions.
C1. Evaluation Coverage and Statistical Reliability (W1, Q1)
The revised L3 evaluation replaces the original seven-versus-two comparison. We ran MODULAR-Agent and its fixed chain twice on the same 183 missions. With OmniNav + SocialNav, MODULAR-Agent completed 9 missions in the first run and 11 in the second. The fixed chain completed none in either run (Section 4.1). The table gives the mean full-mission success rate (FSR) for each configuration, with Wilson 95% confidence intervals calculated from the pooled counts (Table 5 and Appendix H, Table 19).
Appendix H reports intervals for L1 SR and L2/L3 FSR across the main and ablation experiments.
We evaluated OmniNav across all task levels because its indoor and outdoor policies match our task interfaces. OmniNav handles Exit, Entrance, and OVON, and OmniNav-fast follows the outdoor route to its goal (Tables 2–5). With both executor pairs shown above, MODULAR-Agent achieved higher mean FSR than the corresponding fixed chain.
NaVILA achieved 10.00%, 6.50%, and 29.50% SR on Exit, Entrance, and OVON, respectively (Table 2). We limited its evaluation to these tasks because its outdoor setting uses language instructions, whereas OutdoorNav supplies a route and a goal position.
For OmniVLA, we could not identify an episode termination rule or success radius in the paper. The released inference code returns linear and angular velocities without an explicit STOP signal, which our Exit, Entrance, and OVON tasks require (Appendix B.2). All episodes in our small pilot reached the step limit without a valid STOP, so we excluded OmniVLA from the full comparison.
C2. Agent Execution Procedure and Worked Example (W2, Q2)
Section 3.6 now walks through the Entrance stage of an L3 mission. The VLM receives the stage instruction, RGB views, and execution history. Judge uses these inputs to assess the robot’s state. Planning then gives the frozen policy a sequence of subtasks such as “approach the entrance” and “enter the building.”
Suppose the policy proposes STOP at the doorway. The verifier checks the current views for evidence that the robot has crossed the threshold. If the robot is still outside, the verifier can reject completion, cite the visual evidence, and recommend crossing the doorway. The planner then revises the instruction, and the policy resumes from the same pose. Only when the independent evaluator confirms entry does the mission advance to OVON. The verifier also checks periodically and when progress stalls. Each stage allows up to three replans shared across its subtasks (Appendix F).
With the same frozen OmniNav executor, removing verification reduced Entrance success from 45.00% to 24.00% (Appendix G, Table 15).
C3. Real-World Validation (W3)
We evaluated MODULAR-Agent (OmniNav + OmniNav-fast) on a Unitree Go2W in Scene 8 (Office). We ran 10 L2 trials in each direction, using start and goal positions from the corresponding benchmark tasks and Amap walking routes outdoors. The robot completed 2/10 I2O trials and 2/10 O2I trials. The full-task success rate was 20.00% in each direction (Wilson 95% CI: 5.67%–50.98%, Appendix I, Table 22).
Figure 14 shows one success and one failure in each direction. In the successful I2O example, the verifier rejected an early STOP indoors and the robot continued outside to reach the goal. In the successful O2I example, the agent turned toward the open entrance visible in the right camera and verified entry. The two illustrated failures occurred when the outdoor policy went back into the building or the entrance policy exhausted its step budget near a closed door.