acceptodds
Under review as a conference paper at ICLR 2027

I2O2I-Nav: A Benchmark with a Modular Agent for Long-Horizon Indoor-to-Outdoor-to-Indoor Navigation

Abstract

Service and delivery robots often need to complete long-horizon navigation missions spanning indoor and outdoor environments. Existing embodied navigation benchmarks primarily evaluate isolated stages or partial cross-domain journeys, providing limited support for assessing how heterogeneous navigation capabilities compose under continuous execution. We introduce I2O2I-Nav, a physics-enabled benchmark of 20 real-world composite scenes spanning over 115,000 m² across multiple buildings and floors, with aligned 3DGS rendering and collision geometry. Its three-level evaluation progresses from independent primitives to two-stage compositions and complete four-stage I2O2I missions. Each stage starts from its predecessor’s actual endpoint, while stage-conditional metrics track completion and localize failures along the mission. We further propose the MODULAR-Agent, which coordinates frozen navigation policies through completion verification, stage handoffs, and bounded recovery. Evaluation across 1,374 episodes highlights the difficulty of continuous composition. The MODULAR-Agent achieves 40.5–76% success on individual primitives, but full I2O2I success remains low. Compared with a fixed policy chain using the same underlying policies, it more than doubles success on both Indoor-to-Outdoor (I2O) and Outdoor-to-Indoor (O2I), while improving full-mission success from 1.15% to 4.02%. Stage-wise analysis shows that the MODULAR-Agent enables more missions to reach the final object-navigation stage, where conditional success remains low, revealing challenges in both long-horizon stage execution and completion decisions. The anonymous project repository is available at https://anonymous.4open.science/r/i2o2i_nav-390E/README.md.

Primary area:
datasets and benchmarks
Keywords:
Embodied Navigation, Indoor-Outdoor Navigation, Long-Horizon Navigation, Navigation Benchmarks, Modular Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LntPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces I2O2I-Nav, a benchmark to evaluate long-horizon navigation from indoor to outdoor environments and back indoors, aiming to assess how heterogeneous navigation capabilities compose under continuous execution. The benchmark comprises 20 reconstructed real-world scenes and 1,374 evaluation episodes, with corresponding 3DGS reconstructions and aligned collision geometry. Evaluation is done on four tasks: exiting a building, outdoor navigation, entering a building, and open-vocabulary object navigation. These are evaluated both independently and in continuous sequences, where each task starts from the position and orientation reached by the previous one. The authors also introduce MODULAR-Agent, which coordinates frozen navigation policies through subtask planning, completion verification, and recovery. Results show improvements over the underlying policies on individual tasks and over a fixed policy chain on composed missions, although success on complete missions remains low.

Critical strengths
  • Scale and quality of the benchmark. The construction of 20 real-world reconstructed scenes, covering approximately 115,603 m² and supporting 1,374 evaluation episodes, represents a substantial data collection and curation effort. The combination of 3DGS reconstructions, aligned collision geometry, and semantic annotations provides a valuable resource for evaluating navigation across indoor and outdoor environments. I consider the continuous evaluation across stage transitions particularly useful, since it exposes failures that evaluating each task independently would miss.

  • Practical framework for coordinating existing policies. MODULAR-Agent coordinates a frozen set of navigation policies through subtask planning, completion verification, and recovery. I find this a useful direction because it allows existing navigation capabilities to be composed without retraining the underlying policies. The improvements over a fixed policy chain support the value of this approach for longer missions and make it relevant to the future deployment of navigation systems.

Critical weaknesses
  • Evaluations could be more complete. Recent works have introduced integrated navigation systems operating in indoor and outdoor environments, such as NaVILA [1], which combines general language-guided navigation with low-level robot locomotion, and OmniVLA [2], which supports navigation conditioned on language, images, and goal poses. Evaluating these systems under the same task and information conditions would provide a more complete comparison with MODULAR-Agent, whose full-mission performance is currently compared only against a fixed chain of the same underlying policies. Furthermore, the results would benefit from confidence intervals, particularly for the full-mission evaluation, where the reported improvement corresponds to seven successful episodes compared with two for the fixed chain.

  • MODULAR-Agent should be explained more clearly. The main paper introduces its components, but the actual execution procedure remains relatively abstract. A concise algorithm or worked example would help explain what information the VLM receives, how subtasks are generated, when completion is verified, and how failed execution triggers recovery. This would make the framework easier to understand.

  • Lack of real-world validation. The benchmark is motivated by practical service and delivery scenarios, and the proposed framework coordinates existing policies towards completing such missions. However, all reported experiments use a collision-constrained kinematic agent in simulation. Real-world experiments involving transitions between domains, such as indoor to outdoor, would help demonstrate the benchmark’s relevance to developing deployable policies that handle domain shifts. Even a limited physical robot experiment covering these transitions would substantially strengthen the work.

Minor weaknesses

  • The related work section is brief and would benefit from a more detailed comparison with the closest benchmarks, particularly NavVerse and OmniNavBench, to clarify the additional challenges introduced by the complete indoor-to-outdoor-to-indoor mission.
  • At L368 and in Table 4, “Omninav + Socialnav” should be written consistently as “OmniNav + SocialNav.”
  • At L337, the reported MODULAR-Agent success rate for OVON is 33.00%, whereas Table 2 reports 40.50%.
  • Overall, the paper could benefit from tighter writing. Some passages repeat the same points, particularly the literature overview (L078–086, L103–136), state continuity (L156–158, L221–223, L297–298), and headline results (L357–358, L433–438; L375–376, L441–443). Condensing these passages would leave more space for explaining the agent’s execution procedure.

[1] Cheng et al. NaVILA: Legged Robot Vision-Language-Action Model for Navigation. RSS, 2025. https://www.roboticsproceedings.org/rss21/p018.html

[2] Hirose et al. OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation. ICRA, 2026. https://arxiv.org/abs/2509.19480

Questions for authors
  • Could the authors provide confidence intervals for the reported results, particularly the full-mission improvement corresponding to seven successful episodes compared with two for the fixed chain?
  • Could the authors explain MODULAR-Agent’s execution procedure more concretely, including the VLM’s inputs and outputs, subtask generation, and how verification triggers recovery? An algorithm or worked example would be helpful.
  • Does the episode’s wall-clock timer continue running while MODULAR-Agent reasons, verifies completion, or waits for navigation-policy responses, or is it paused during these calls? Please clarify whether inference runs asynchronously with simulator execution and whether the reported timeout includes all planning and inference latency.

I lean towards weak acceptance. I consider the benchmark a valuable contribution, particularly its reconstructed environments and continuous evaluation across navigation stages. Although the evaluation could be broader and the agent’s execution procedure needs clarification, I believe the benchmark’s usefulness and the effort invested in its construction support acceptance. Addressing the questions above would strengthen my assessment.

3 comments

Part 1 of 2

Dear Reviewer LntP,

Thank you for recognizing the value of evaluating stage transitions.

C1. Evaluation Coverage and Statistical Reliability (W1, Q1)

The revised L3 evaluation replaces the original seven-versus-two comparison. We ran MODULAR-Agent and its fixed chain twice on the same 183 missions. With OmniNav + SocialNav, MODULAR-Agent completed 9 missions in the first run and 11 in the second. The fixed chain completed none in either run (Section 4.1). The table gives the mean full-mission success rate (FSR) for each configuration, with Wilson 95% confidence intervals calculated from the pooled counts (Table 5 and Appendix H, Table 19).

Frozen executorsFixed chainMODULAR-Agent
OmniNav + SocialNav0.00% (0.00–1.04%)5.46% (3.56–8.29%)
OmniNav + OmniNav-fast0.00% (0.00–1.04%)3.01% (1.69–5.30%)

Appendix H reports intervals for L1 SR and L2/L3 FSR across the main and ablation experiments.

We evaluated OmniNav across all task levels because its indoor and outdoor policies match our task interfaces. OmniNav handles Exit, Entrance, and OVON, and OmniNav-fast follows the outdoor route to its goal (Tables 2–5). With both executor pairs shown above, MODULAR-Agent achieved higher mean FSR than the corresponding fixed chain.

NaVILA achieved 10.00%, 6.50%, and 29.50% SR on Exit, Entrance, and OVON, respectively (Table 2). We limited its evaluation to these tasks because its outdoor setting uses language instructions, whereas OutdoorNav supplies a route and a goal position.

For OmniVLA, we could not identify an episode termination rule or success radius in the paper. The released inference code returns linear and angular velocities without an explicit STOP signal, which our Exit, Entrance, and OVON tasks require (Appendix B.2). All episodes in our small pilot reached the step limit without a valid STOP, so we excluded OmniVLA from the full comparison.

C2. Agent Execution Procedure and Worked Example (W2, Q2)

Section 3.6 now walks through the Entrance stage of an L3 mission. The VLM receives the stage instruction, RGB views, and execution history. Judge uses these inputs to assess the robot’s state. Planning then gives the frozen policy a sequence of subtasks such as “approach the entrance” and “enter the building.”

Suppose the policy proposes STOP at the doorway. The verifier checks the current views for evidence that the robot has crossed the threshold. If the robot is still outside, the verifier can reject completion, cite the visual evidence, and recommend crossing the doorway. The planner then revises the instruction, and the policy resumes from the same pose. Only when the independent evaluator confirms entry does the mission advance to OVON. The verifier also checks periodically and when progress stalls. Each stage allows up to three replans shared across its subtasks (Appendix F).

With the same frozen OmniNav executor, removing verification reduced Entrance success from 45.00% to 24.00% (Appendix G, Table 15).

C3. Real-World Validation (W3)

We evaluated MODULAR-Agent (OmniNav + OmniNav-fast) on a Unitree Go2W in Scene 8 (Office). We ran 10 L2 trials in each direction, using start and goal positions from the corresponding benchmark tasks and Amap walking routes outdoors. The robot completed 2/10 I2O trials and 2/10 O2I trials. The full-task success rate was 20.00% in each direction (Wilson 95% CI: 5.67%–50.98%, Appendix I, Table 22).

Figure 14 shows one success and one failure in each direction. In the successful I2O example, the verifier rejected an early STOP indoors and the robot continued outside to reach the goal. In the successful O2I example, the agent turned toward the open entrance visible in the right camera and verified entry. The two illustrated failures occurred when the outdoor policy went back into the building or the entrance policy exhausted its step budget near a closed door.

Part 2 of 2

C4. Wall-Clock Timeout, Inference Latency, and Asynchronous Execution (Q3)

The episode clock runs continuously. Planning, verification, policy inference, and waiting for responses all count toward the timeout. The simulator continues executing actions while inference runs asynchronously (Section 4.1 and Appendix F.4).

C5. Comparison with NavVerse and OmniNavBench (Minor W1)

Section 2.2 now compares the three benchmarks directly. NavVerse follows a robot through an exit and along an outdoor route. OmniNavBench tests composite instructions across six navigation task families and multiple robot embodiments. In I2O2I-Nav, the outdoor route ends near a second building. From the pose it actually reaches, the robot must discover the entrance, cross the threshold, and find the requested object inside. The three evaluation levels let us examine the individual skills, a single handoff, and the complete journey.

C6. Naming Consistency, OVON SR, and Writing Concision (Minor W2–W4)

Thank you for catching the naming and numerical inconsistencies. Section 4.1 now defines the abbreviation “O + S” used in Table 4 as “OmniNav + SocialNav.” We corrected the OVON result in Section 4.2 to match Table 2. In the revised two-run evaluation, the OmniNav-based agent achieves 56.00% OVON SR. The 40.50% entry is for the Uni-NaVid-based agent.

We shortened the literature overview, consolidated the explanation of state continuity, and removed repeated summaries of the main results. Section 3.6 uses the space to show how the agent verifies entry and recovers from an early stop.


We appreciate your support.

Dear Reviewer LntP,

Thank you again for your detailed review. As the discussion period ends in a few hours, we would like to kindly remind you of our responses above, which include new experiments addressing each of your comments. We would greatly appreciate it if you could let us know whether they address your concerns, and, if so, consider updating your score.

Best regards, The Authors

Reviewer_BJtDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces I2O2I-Nav, a benchmark for evaluating embodied robots on long-horizon indoor–outdoor–indoor navigation. The authors scanned 20 real-world scenes with a 3D scanner. In Isaac Sim, 3D Gaussian Splatting provides visual rendering, and meshes handle collision detection. The benchmark has three evaluation levels of increasing difficulty: L1, L2, and L3, which cover primitive tasks, short-horizon tasks, and long-horizon tasks, respectively. The paper also proposes MODULAR-Agent, which uses a VLM for workflow planning and invokes OmniNav and SocialNav for execution. The agent is evaluated at all three levels. Compared with the baselines, it improves performance at every level, but overall results remain low. The authors conclude that continuous composition is highly challenging.

Critical strengths
  1. Rather than continuing to evaluate navigation on the isolated single tasks commonly studied by the community, the paper proposes a new indoor–outdoor–indoor navigation setting. This setting also evaluates whether handoffs between consecutive stages are carried out properly, which closely reflects the practical requirements of embodied navigation.
  2. The paper provides a staged evaluation from L1 to L3. The three levels focus on individual tasks, single handoffs, and the complete task chain, respectively. Task difficulty increases level by level, which makes it possible to pinpoint the stage at which a task fails.
  3. In addition to the multi-level evaluation, the paper provides a viable baseline for the proposed benchmark. This baseline offers a reference point for future work evaluated on the benchmark, and the overall pipeline is complete.
Critical weaknesses
  1. The experimental results are thin for a benchmark paper. Although the paper proposes a three-level evaluation, it relies heavily on results from very few methods. L2 and L3 evaluate only a single fixed chain, OmniNav+SocialNav, and the agent built on that chain. These results are insufficient to establish the difficulty and challenge of the benchmark or to show that it can distinguish between different methods. They also cannot determine whether the paper's core premise holds, namely that continuous composition introduces additional difficulty. In addition, all experiments are single runs, with no variance or confidence intervals reported. This makes the results more open to question, given that they depend on a commercial VLM API and online simulation.
  2. The agent's gains are not attributed to specific components. The paper designs MODULAR-Agent around the failure causes observed in other methods, and the agent mitigates some of the previously observed failure modes. However, the paper does not show which component mitigates each type of failure. More specifically, there is no component ablation, so it can't tell which specific VLM capability the gains come from.
  3. Visual–collision consistency of the scenes is not verified. The paper checks path reachability and edits the Gaussians and meshes together during scene editing, but it does not quantitatively check the consistency between the two. Possible inconsistencies include regions that appear traversable but are blocked by collision geometry, and visible obstacles that lack collision geometry. Such inconsistencies pose a potential risk to the validity of the test results obtained across the scenes.
Questions for authors
  1. Can the authors report results on the proposed benchmark for a more diverse set of combinations of existing methods? Can they also provide repeated runs to verify that the current conclusions hold?
  2. For the failure phenomena identified in the paper, can the authors break down the different capabilities provided by the VLM and analyze the improvement each one contributes?
  3. Can the authors provide experiments showing that the proposed scenes are free of inconsistencies between visual appearance and physical collision?
2 comments

Dear Reviewer BJtD,

Thank you for highlighting the importance of handoffs.

C1. Policy Coverage, Composition Difficulty, and Repeated Runs (W1, Q1)

  1. Policy coverage. Tables 2–3 now include NaVid and NaVILA on Exit, Entrance, and OVON and CityWalker on OutdoorNav. Tables 4–5 compare three fixed chains and two MODULAR-Agent configurations. Both agent configurations outperform their matching chains on all three composed tasks. The benchmark distinguishes the methods even on L3, where none of the fixed chains finished a mission: their mean stage-completion scores (CSR) ranged from 6.63% to 15.16%. The table gives two-run mean full-mission success rates (FSR).

    ConfigurationI2O (%)O2I (%)I2O2I (%)
    Fixed chain (OmniNav + SocialNav)14.0012.500.00
    Fixed chain (OmniNav + OmniNav-fast)19.0026.500.00
    Fixed chain (Uni-NaVid + OmniNav-fast)10.505.000.00
    MODULAR-Agent (OmniNav + SocialNav)30.5032.505.46
    MODULAR-Agent (OmniNav + OmniNav-fast)26.5039.003.01
  2. Difficulty after a handoff. SocialNav completes 74.00% of standalone OutdoorNav tasks from their prescribed starts. In the OmniNav + SocialNav chain, it must join the route from the position where OmniNav finished Exit. The aligned tasks share the outdoor goal, success rule, and 600-step OutdoorNav limit. Among missions that completed Exit, SocialNav completed OutdoorNav in 68.29% of L2 I2O missions and 57.14% of L3 missions (Tables 3–5). These observations support the paper’s core premise: continuous composition can expose navigation difficulties that standalone evaluation misses.

  3. Repeated runs. We evaluated every main and ablation configuration twice. On the same 183 L3 missions, MODULAR-Agent (OmniNav + SocialNav) finished 9 in the first run and 11 in the second. The corresponding fixed chain finished none in either run. The agent’s mean FSR was 5.46%, with a between-run sample SD of 0.77 percentage points (Appendix D.6). The pooled Wilson 95% intervals were 3.56–8.29% for the agent and 0.00–1.04% for the chain (Appendix H, Table 19). These intervals assume independent executions of the repeated tasks.

C2. Component Contributions (W2, Q2)

The failure examples in Section 4.6 include stopping outside a doorway and repeatedly moving into vegetation. These cases motivate checking boundary crossings and revising plans when progress stalls. We tested each component on the same episodes with the same frozen policies (OmniNav for Exit, Entrance, and OVON, and SocialNav for OutdoorNav).

Component removedFunctionChange from the full agent, averaged over two runs
VerificationChecks proposed completion before accepting itExit, Entrance, and OVON SR fall by 16.00, 21.00, and 16.00 percentage points
JudgeAssesses location and boundary stateEntrance SR falls from 45.00% to 34.50%
Planning and replanningDecomposes the task and revises the plan during recoveryOutdoorNav SR falls from 76.50% to 66.00%
EpisodeMemoryKeeps the execution history within a stageExit SR falls from 36.00% to 22.00%
MissionMemoryCarries context across handoffsI2O, O2I, and L3 FSR fall by 12.00, 10.00, and 3.27 percentage points

Of the tested removals, verification had the largest effect on success for the three tasks that require a valid stop. Removing planning and replanning had the largest effect on OutdoorNav. These are changes in task success. The failure examples are qualitative (Sections 4.5–4.6 and Appendix G, Tables 15–17).

C3. Visual–Collision Consistency (W3, Q3)

We audited passages and obstacles selected from rendered views across all 20 scenes. Table 7 (Appendix A.3) reports zero discrepancies in the inspected samples.

CheckSamplesDiscrepanciesRate
Visually traversable passage blocked by collision geometry1,00000%
Obstacle probe missing collision at the visible target3,00000%
Distinct obstacle with at least one probe miss1,00000%

The last row groups the same probes by obstacle. Each obstacle was probed at three contact points or approach directions.

We also replayed every benchmark reference trajectory with collision enabled, both with and without the robot model. All replays completed in both modes, and all replayed routes appeared traversable in the RGB views (Appendix C.5).

C4. Scene Acquisition and Release Permissions (Ethics Concern)

Yes. The respective site owners authorized us to collect the scans, publicly release the benchmark scene assets, and use them in research publications. The Ethics Statement records this permission.


Thank you for helping us strengthen the evaluation.

Dear Reviewer BJtD,

Thank you again for your detailed review. As the discussion period ends in a few hours, we would like to kindly remind you of our responses above, which include new experiments addressing each of your comments. We would greatly appreciate it if you could let us know whether they address your concerns, and, if so, consider updating your score.

Best regards, The Authors

Reviewer_G4yDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The research topic of I2O2I-Nav is valuable:

  • However, the current experiments and evaluation protocols are insufficient to substantiate the main conclusions.
  • Key concerns include the validity of the success criteria, the attribution of performance gains, and the stability of results across complete, multi-stage tasks.

The paper’s notable strengths include:

  • The 20 reconstructed real-world scenes hold practical value for the field.
  • By maintaining the actual end-pose from the preceding stage, the composite task setup can reveal issues that might be overlooked when evaluating individual tasks in isolation.
  • Using the same underlying policy for both the fixed strategy chain and the agent provides a sound starting point for investigating orchestration mechanisms.
Critical strengths
  • The 20 reconstructed real-world scenes hold practical value for the field.
  • By maintaining the actual end-pose from the preceding stage, the composite task setup can reveal issues that might be overlooked when evaluating individual tasks in isolation.
Critical weaknesses
  1. The object-navigation success criterion is too permissive.

    Appendix B.2 counts a STOP within 3 m of the target on the same navigation plane as success, without checking visibility or reachability. This could count a stop on the other side of a wall as successful object navigation. I cannot tell whether this occurred in the reported runs, but the criterion allows it. Since reaching the final indoor object is central to the benchmark, the authors should inspect the successful trajectories and evaluate how the results change when such cases are excluded.

  2. The experiments do not establish which agent components produce the gains.

    The agent changes task decomposition, completion verification, and recovery together. It can reject a policy’s STOP proposal and continue, whereas the fixed chain submits that proposal for evaluation. Using identical frozen policies is a useful control, but it does not separate semantic verification from additional attempts. An equal-budget retry baseline would help resolve this. Appendix D.7 discusses such a comparison without reporting results. The verifier’s false acceptance and rejection rates are also defined in F.5 but never measured.

  3. The full-mission improvement rests on very few successes.

    The increase from 1.15% to 4.02% corresponds to 2 versus 7 successes out of 174 episodes. This is encouraging, but the “3.5-fold improvement” sounds stronger than the evidence warrants without repeated runs or uncertainty estimates. The stage counts support a more specific finding: the agent brings 33 episodes to the final OVON stage, compared with 10 for the fixed chain. Conditional success at that stage remains similar, at 7/33 versus 2/10.

  4. RC is difficult to interpret in multi-floor environments.

    Appendix D.4 projects the final position onto the entire reference route in the horizontal plane, without restricting the projection by floor or completed stage. Routes that overlap across floors or pass near downstream segments can therefore produce misleading progress values. The manuscript mentions floor and connectivity labels elsewhere; these should be incorporated into the metric, or its interpretation should be narrowed to terminal geometric proximity along the reference route.

  5. Some baseline results raise questions about the implementation.

    Uni-NaVid achieves zero success on Exit and Entrance and travels only about 1–1.5 m on average. This may reflect poor transfer, but such short trajectories warrant checking the stopping and action interfaces. Appendix F.4 also says the policy client resets before each “step.” Does this mean every action or every subtask? The distinction matters for policies that rely on observation history. More precise adapter and history-handling details are needed to interpret these results.

  6. The experiments do not isolate the difficulty introduced by composition.

    L1 and L3 differ in starting states, goals, and budget consumption. Their success rates therefore cannot directly establish a handoff penalty. The conditional stage rates also involve different subsets of episodes for each method. Appendix D.7 acknowledges this issue, but a matched handoff experiment would make the analysis much stronger—for example, evaluating the same downstream task from prescribed starts and actual handoff poses while controlling the remaining budget.

  7. The benchmark’s coverage needs better characterization.

    Scene area and goal-instance counts provide only a partial picture. Route-length distributions, cross-floor episode counts, building-pair coverage, and per-scene results would help readers judge what the benchmark tests. The open-vocabulary evaluation is also limited to annotated goal names, with no synonym or novel-category analysis. Finally, because there is no development split, the authors should explain how prompts, verification thresholds, and recovery settings were selected.

  8. Several reporting inconsistencies should be corrected.

    Section 4.2 gives an OVON success rate of 33%, while Table 2 reports 40.5%. The increase from 38% to 40.5% is described twice as 2.3 rather than 2.5 percentage points. The tables also omit the DCR eligibility counts required by Appendix D.5, and the promised measurements of model calls and execution time are missing.

Questions for authors
  1. How many of the successful OVON episodes would remain successful if the stopping criterion excluded positions separated from the target by a wall or other obstacle? Have you inspected the seven successful full-mission trajectories for such cases?

  2. How does MODULAR-Agent compare with a simple retry baseline under the same execution budget? Can you separate the contributions of task decomposition, completion verification, and recovery?

  3. What are the verifier’s measured false acceptance and false rejection rates? How much additional inference time and how many model calls does it require?

  4. Are the full-mission results based on a single run per episode? Given the small success counts—2 versus 7 out of 174—how stable is the improvement across repeated runs?

  5. In Appendix F.4, does resetting the policy client before each “step” mean before each action or each subtask? How is the observation history required by the underlying policies preserved, and have you checked whether adapter behavior contributes to Uni-NaVid’s unusually short trajectories?

  6. How does RC handle reference segments that overlap across floors or pass near later stages? Would restricting projection by floor and valid stage prefix change the reported results?

  7. Were prompts, verification thresholds, and recovery rules selected using separate development environments? Can you clarify this process, given that all benchmark episodes are designated for evaluation?

3 comments

Part 1 of 2

Dear Reviewer G4yD,

Thank you for pointing out the ambiguous success rule and short Uni-NaVid trajectories. We checked the final stops, corrected the checkpoint, and ran each configuration twice.

C1. OVON Success Audit (W1, Q1)

The 3 m, same-plane rule can accept a STOP across a wall. In the original submission, none of the seven successful MODULAR-Agent missions ended across a wall from its target. We then checked every successful standalone OVON episode and L3 mission in the revised main and ablation runs. No wall or obstacle separated a stop from its target. All successes survive this exclusion (Section 4.1 and Appendix B.2).

C2. Ablations, Retries, and Verifier Cost (W2, Q2–Q3)

With the same frozen executors, removing verification lowered Exit, Entrance, and OVON SR by 16, 21, and 16 percentage points. Removing planning lowered OutdoorNav SR from 76.50% to 66.00%. The verification ablation also changes retry opportunities. The planning ablation removes decomposition and replanning together. We have no equal-budget retry-only result. Removing Judge lowered Entrance SR by 10.50 points, removing EpisodeMemory lowered Exit SR by 14 points, and removing MissionMemory lowered L3 full success from 5.46% to 2.19% (Section 4.5, Tables 15–17).

In 427 audited ACCEPT or REJECT decisions from L2 Entrance and L3 Exit, Entrance, and OVON, the verifier accepted 27/322 invalid proposals (8.39%) and rejected 32/105 valid proposals (30.48%). The 183 sampled L3 missions used 2,642 Kimi requests, averaging 14.44 calls and 110.8 s of cumulative request time per mission across all modules. These totals do not isolate verifier overhead (Appendix F.5).

C3. Statistical Reliability (W3, Q4)

Both configurations used OmniNav + SocialNav on the same 183 missions in each run (Section 4.1).

ConfigurationRun 1 successesRun 2 successesMean FSR (Wilson 95% CI)
Fixed chain0/1830/1830.00% (0.00–1.04%)
MODULAR-Agent9/18311/1835.46% (3.56–8.29%)

The agent outperformed the fixed chain in both runs, by 5.46 percentage points on average (agent sample SD: 0.77 points, Appendix D.6). Mean CSR was 26.91% versus 15.16%. Among missions that reached OVON under each method, final-stage success averaged 27.61% versus 0.00% (Table 5). The Wilson intervals pool both runs and assume independent executions despite repeated missions and shared scenes (Appendix H, Table 19).

Part 2 of 2

C4. Multi-Floor Route Completion (W4, Q6)

RC already restricts projection to the terminal floor and to completed stages or the current stage. A stop on the first floor cannot receive credit for a second-floor segment directly overhead. RC is therefore unchanged for a given trajectory (Section 3.5 and Appendix D.4).

C5. Policy Adapter and Observation History (W5, Q5)

The client resets at the start of each subtask attempt, including a retry. History persists across actions within that attempt. Resetting the client preserves simulator state and agent memory (Appendix F.4).

We found a checkpoint issue and reran Uni-NaVid with the correct frozen weights. Mean trajectory length rose from about 1–1.5 m to 14.43 m on Exit and 24.98 m on Entrance. The corresponding success rates are 9.00% and 12.00% (Table 2).

C6. Composition and Handoffs (W6)

Aligned L1 and composed OutdoorNav tasks share the scene, goal, success rule, and 600-step limit. SocialNav succeeds on 74.00% of standalone tasks from prescribed starts. Among missions that completed Exit, SocialNav starts where OmniNav stopped and succeeds in 68.29% of L2 I2O missions and 57.14% of L3 missions (Tables 3–5). This shows lower success after Exit in both compositions. Standalone SR covers all L1 tasks. Composed rates condition on reaching OutdoorNav and do not control the remaining episode budget in a task-paired comparison (Appendices D.7 and F.4).

C7. Benchmark Coverage and Configuration (W7, Q7)

We used no separate development environments. During development, we inspected baseline runs, including runs on final evaluation tasks. Policies sometimes stopped at a doorway before entering or got stuck. These observations informed the prompts and recovery rules, which we used across all scenes. The prompts describe the available skills and navigation policies. At a proposed stop, the verifier uses RGB views and execution history without access to hidden evaluator labels.

We fixed the verifier’s confidence threshold at 0.7 throughout development and evaluation. When the verifier judges a stage complete with confidence at or above this threshold, the agent submits the proposal to the independent evaluator. To limit recovery time and model calls, we allowed up to three replans per stage in every scene (Appendices C.5 and F.4).

The 20 scenes contain 336 annotated object goals and 96 cross-floor episodes. L3 building pairs cover 50 of 65 indoor buildings (Table 6). Mean L3 reference-route length ranges from 41.2 to 303.4 m across scenes and averages 175.3 m across missions (Table 9). These figures describe building coverage and scene-level route means. We report policy performance aggregated across scenes and evaluate annotated goal names, without synonym or novel-category tests (Appendix C.6).

C8. Corrections and DCR Counts (W8)

You correctly identified the original 38% to 40.5% change as a 2.5-point gain. The revised two-run OVON SR is 47.50% for OmniNav and 56.00% for MODULAR-Agent (OmniNav), an 8.50-point gain. The 40.50% value belongs to MODULAR-Agent (Uni-NaVid) (Section 4.2, Table 2). For the Uni-NaVid + OmniNav-fast fixed chain, Table 4 gives the corrected O2I CSR of 42.75%. Table 5 marks L3 final-stage conditional success as undefined for this chain because none of its missions reached OVON.

DCR eligibility. Table 10 in Appendix D.5 gives mean eligible counts across the two runs for Tables 3–5. Any episode with positive outdoor travel counts, including failures and zero-DCR episodes. Each run has 200 episodes per L1/L2 task and 183 for L3. Agent denotes MODULAR-Agent.

L1 policyOutdoorNavL2/L3 configurationI2OO2II2O2I
SocialNav200OmniNav + SocialNav4120063
OmniNav-fast200OmniNav + OmniNav-fast5220041
CityWalker200Uni-NaVid + OmniNav-fast2520030
Agent (SocialNav)200Agent (OmniNav + SocialNav)8520086
Agent (OmniNav-fast)200Agent (OmniNav + OmniNav-fast)9620091

Table 14 in Appendix F.5 gives calls and cumulative Kimi request time for six sampled task types (L3 figures in C2). These times cover model requests only.


We hope these revisions address your concerns.

Dear Reviewer G4yD,

Thank you again for your detailed review. As the discussion period ends in a few hours, we would like to kindly remind you of our responses above, which include new experiments addressing each of your comments. We would greatly appreciate it if you could let us know whether they address your concerns, and, if so, consider updating your score.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.