acceptodds
Under review as a conference paper at ICLR 2027

RATIO: A Reasoning Agent for Task-Adaptive Interaction Orchestration in Zero-Shot Object Navigation

Abstract

Zero-shot object navigation requires sequential decisions under partial observations. Existing methods increasingly use large language models (LLMs) for high-level reasoning, yet navigation capabilities are still commonly organized by predefined routines, while precise spatial computation is often mixed with semantic decision making. We introduce **RATIO** , a **R**easoning **A**gent for **T**ask-Adaptive **I**nteraction **O**rchestration for zero-shot object navigation. RATIO organizes geometric, semantic, and navigational information into an agent-facing environment representation and an executable navigation graph that the agent can query on demand, and exposes specialized navigation capabilities as tools. At each decision step, the agent selects tools based on the current task context, while dedicated modules perform geometric, retrieval, and candidate computations. By separating capability selection from execution, RATIO enables context-dependent navigation while preserving precise and reproducible spatial computation. We evaluate RATIO on HM3Dv1, HM3Dv2, MP3D, and HM3D-OVON Unseen, achieving a **53.0%** success rate and **34.0%** success weighted by path length on the open-vocabulary benchmark with GPT-5.6 Luna.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM Agents, Agentic Reasoning, Tool Use, Embodied AI, Zero-Shot Object Navigation, Vision-Language Navigation, Decision Making, Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_FLouOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RATIO for zero-shot object navigation. The system organizes geometric uncertainty, task semantics, and navigable positions into a queryable environment representation and an executable navigation graph. An LLM selects tools for candidate search, experience retrieval, exploration, and multi-view verification based on current evidence, then uses tool feedback to select navigation nodes. Dedicated modules handle spatial computation and path planning, while all model parameters remain frozen. The authors evaluate the system on HM3Dv1, HM3Dv2, MP3D, and HM3D-OVON Unseen, reporting improvements in success rate and path efficiency.

Critical strengths
  1. The paper associates task semantics, geometric uncertainty, visit history, and path information with navigable nodes, grounding the LLM's high-level choices in concrete executable positions. The interfaces between environment representation, tool queries, and navigation execution are reasonably clear and form a complete navigation pipeline.
  2. The experiments cover four navigation benchmarks, including unseen target categories in an open-vocabulary setting. The GPT-5.6 Luna variant achieves 53.0% SR and 34.0% SPL on HM3D-OVON Unseen, exceeding SpaceVLN's reported 51.6% and 33.1%. In internal ablations with shared components, progressively adding the navigation graph, LLM decisions, and tools with feedback history improves both SR and SPL on all four benchmarks, supporting the practical value of the overall design.
Critical weaknesses
  1. Limited methodological novelty. The paper primarily integrates executable navigation graphs, LLM tool orchestration, and dedicated spatial computation. Most tools in Table 4 also come from existing methods. The independent technical contribution over tool-use methods such as AgenticNav and existing graph-based navigation methods remains unclear.
  2. Insufficient causal evidence for the core mechanism. Table 2 adds tool access and feedback history simultaneously, without a fixed-orchestration control or a separate history ablation. It therefore cannot distinguish whether the gains come from additional information, memory, or adaptive tool selection.
  3. Insufficiently controlled comparisons with external baselines. RATIO uses SAM 3, but Table 1 does not clarify whether the baselines were rerun with shared perception modules and comparable computational budgets. Differences in foundation models and computational resources may therefore contribute to the reported gains.
Questions for authors
  1. What is the independent technical contribution over the closest tool-use and graph-based navigation methods?
  2. Could you compare against fixed orchestration using the same tools, history, and budget, and separately ablate feedback history?
  3. Please clarify the sources of baseline results and provide comparisons with representative strong baselines using shared perception components and comparable budgets.
1 comment
AuthorsOpenReview ↗

Thank you for recognizing the navigation representation and tool interaction design, and for your suggestions on the independent contribution, mechanism analysis, and baseline comparisons. We address these points below and describe the corresponding revisions and planned experiments.

Weakness 1 and Question 1

Thank you for the suggestion. We fully agree that the independent contribution should be clarified and would also like to explain the existing experimental evidence. We compare RATIO with three related groups of methods: semantic exploration, map reasoning, and LLM decisions (Table 1). Progressive ablations with shared components further support the contributions of the executable navigation graph, LLM decisions, and tool interaction (Table 2).

The core innovation is the combination of the executable navigation graph, tool-selection mechanism, and LLM. The executable navigation graph provides task evidence and navigable positions; the LLM selects tools according to the current task state and uses tool feedback to adjust subsequent decisions. Most existing tool-using navigation methods organize tools around perception, planning, and control, and modular methods already support replacing these components. For example, in a fixed environment, the best-performing perception tool can generally be selected in advance, making repeated online selection unnecessary. RATIO considers that different navigation stages may require different strategies. We therefore design tools for evidence acquisition, target candidate search, broad exploration, and multi-view verification, allowing the LLM to select the navigation capabilities needed as the task progresses to improve exploration efficiency. Section 3.4.2 describes these tools, and Figure 4(a) in Section 4.4 further illustrates how tool use changes across navigation stages.

The method overview and contribution paragraphs in the Introduction, together with the final paragraph of Section 2.3, LLM-Driven Navigation Decisions, did not sufficiently explain the role of this combination or the design of tools around the strategy needs of different navigation stages. We have revised these passages to explain how the executable navigation graph supports tool selection, how tool feedback affects subsequent decisions, and how the tools address different navigation stages. We have also connected the tool-use patterns in Section 4.4 with the ablation results in Table 2 to clarify the value of this design for navigation success and path efficiency.

Weakness 2 and Question 2

Thank you for the suggestions. We fully agree that stronger experimental evidence is needed for the core mechanism, and partially agree with the fixed-orchestration comparison requested in Question 2. Several tools draw on existing methods, including CogNav and UniGoal, which are compared in Table 1; Table 4 in Appendix D lists the corresponding tool sources. On HM3Dv1, RATIO achieves 36.4% SPL, compared with 26.2% for CogNav and 25.1% for UniGoal. We had not sufficiently articulated the conclusion from this comparison and have now added a statement after the corresponding results in Section 4.2, explaining the value of orchestrating existing navigation capabilities on demand for path efficiency.

For tool feedback history, we agree that an independent ablation with and without this history is needed to complete the evaluation. The current comparison from C to D reflects the combined contribution of tool interaction and tool feedback history. Due to time constraints, we will add this ablation in future work, keeping the executable navigation graph, tools, LLM, and budgets fixed while comparing retained and disabled tool feedback history. The experiment will be added to the ablation study.

Weakness 3 and Question 3

Thank you for raising the issue of fair comparison. We agree that the sources of baseline results and reproduction settings should be clarified. We attempted to reproduce some baselines, but our local results did not reach the metrics reported in the literature. For ApexNav, a reproduction using the smallest segmentation model and BLIP-2 did not reach the reported results. Under our local configuration, the original perception pipeline also incurred greater overhead than the configuration using SAM 3. We therefore also attempted a reproduction using SAM 3 for detection and segmentation.

Given concerns that differences in CPUs, GPUs, storage devices, and runtime configurations could affect reproduction results, we ultimately used baseline results recorded in other papers for Table 1. We have identified the cited sources and original result provenance in the revision. RATIO uses SAM 3 primarily to integrate detection and segmentation and simplify the perception pipeline; our research focus remains the executable navigation graph and task-adaptive tool orchestration. To examine these designs, the internal ablations share perception, map updates, path planning, and the executor, supporting the contributions of the executable navigation graph, LLM decisions, and tool interaction.

Reviewer_U3UMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RATIO, an LLM agent for zero-shot object navigation. It builds an executable navigation graph whose nodes are reachable positions (frontiers, uncertain regions, target and supporting regions) and whose edges carry path cost, uncertainty and simple spatial relations. At each decision point, the LLM chooses which navigation tools to call, such as candidate search, experience retrieval or multi-view verification, and then picks the next node. The main claim is that letting the LLM decide which capability to use, instead of following a predefined routine, works better. RATIO reports the best SR on HM3Dv1, HM3Dv2, MP3D and HM3D-OVON Unseen.

Critical strengths
  • Restricting decisions to reachable nodes is a good design. Every LLM decision must land on a node that A* has verified as reachable, which avoids a common failure of LLM-based navigation, where the selected goal is not reachable.
  • Keeping uncertain cells as a separate map state is useful. The uncertain state is carried into node evidence and edge attributes, so the agent can tell which paths pass through poorly observed areas, which most scene graphs and frontier maps do not record.
Critical weaknesses
  • According to Table 4, six of the seven tools come from prior work (CogNav, TrajRAG, UniGoal, EfficientNav). Section 2.3 contrasts RATIO with Fast-Slow ReAct, which already does event-triggered reasoning and retrieval for ObjectNav, but there is no experiment showing that RATIO's on-demand selection is what makes the difference. The A-to-B ablation supports the value of the executable navigation graph, but I think the paper should make clearer what is new beyond the graph itself.

  • The SR margins over the strongest baselines are small (0.1 over CogNav on HM3Dv1, 0.9 over OpenFrontier on HM3Dv2), and RATIO is behind DRIVE-Nav in SPL on HM3Dv2 and MP3D. The SPL gain on HM3Dv1 is large compared with CogNav (36.4 vs. 26.2) but only 0.5 above DRIVE-Nav. Since RATIO uses SAM 3 and a recent LLM, while most baselines use older models, a baseline rerun with the same perception module would help show how much of the ranking comes from the method itself. This does not affect the internal ablations, which share the same perception.

  • The appendix sets the success distance to 0.20 m (0.25 m on OVON) without saying how it is measured. If it is the distance to target viewpoints, it is twice the standard Habitat ObjectNav value of 0.1 m, which matters when most SR margins over the baselines are below one point. Section 3.4.1 also says the agent issues STOP "when the agent is within the success distance of the target", but the paper never describes the actual stop criterion, so it is not clear whether the policy uses an estimated target position or ground-truth distance. These points affect the comparison in Table 1, though not the internal ablations.

Questions for authors
  • Section 3.4.1 says the agent issues STOP once it is within the success distance of the target, and Appendix E lists this distance as an evaluation parameter, yet the paper never describes how the stop decision is made. What is the exact stop criterion, and does it use any ground-truth information such as Habitat's distance-to-goal?
  • Fig. 4(b) shows that only about 4 of roughly 50 episodes are still running near step 450, so about 90% of episodes end before the 500-step limit, which does not fit an SR of 47.8 to 78.2 if STOP were issued only within the true success distance. Are there other termination conditions, such as a forced stop when no frontier remains, and which benchmark and selection rule were used for the Fig. 4 episodes?
  • The success distance is 0.20 m on HM3D and MP3D and 0.25 m on OVON, while the baseline numbers appear to be taken from the original papers, which mostly use the standard 0.1 m to target viewpoints. How is this distance measured, and what are RATIO's results under the same success criterion as the baselines?
  • Replacing the direct LLM decision with rules costs up to 10.7 SR points (C vs. B), yet switching from GPT-5.6 Luna to DeepSeek V4.1 Flash changes SR by at most 0.4 points on every benchmark. Could you report setting C with DeepSeek and with a much smaller LLM, the per-episode agreement of success between the two RATIO variants, and how often the LLM in setting D simply adopts the top tool recommendation?
  • Table 5 says experience retrieval returns historical segments with their actual outcomes, but the paper does not say whether this memory is kept across evaluation episodes. If it is, do the results depend on episode order, and how do they change when the memory is reset for each episode?
  • Fig. 3 colors some trajectory segments as "Frontier recovery", which is not one of the seven tools in Table 4. What is this module, and how often is it used?
  • Fig. 4(b) reports roughly 80K to 160K tokens per ongoing episode in each step interval, but no wall-clock time or total cost is given. What are the total token cost and runtime per episode, and how do they compare with CogNav or Fast-Slow ReAct?
  • Backbone A shares the perception and semantic fusion of ApexNav but is about 15 SR points below it on HM3Dv2. What exploration policy does Backbone A use?
1 comment
AuthorsOpenReview ↗

Thank you for your careful review and specific suggestions. Below, we clarify the stopping and evaluation conditions, tool use, and experience memory, and describe planned studies on LLM scale and runtime overhead.

Weakness 1

We partially agree and clarify the existing design. Table 2 compares C (Direct Agent), where the LLM directly selects a node on the executable navigation graph, with D (Full RATIO), which adds tool calls and tool feedback history for selection on demand. This assesses their combined contribution; further evidence is needed for tool selection itself. We will add appendix ablations randomly selecting one tool or different numbers of tools. Each group will be compared with LLM selection under matched call counts and budgets, sharing perception, the graph, tool implementations, and feedback history.

Weakness 2

We agree that component configurations should be clearer and have specified them in the revision. External comparisons evaluate complete systems with different perception, mapping, and decision designs, making all components difficult to match. SAM 3 integrates target detection and instance segmentation to reduce perception modules. Our focus is the executable navigation graph and task-adaptive tool orchestration. Internal ablations share perception, map updates, path planning, and the executor to assess the graph, LLM decisions, and tool interaction.

Weakness 3 and Questions 1 and 3

We agree and have distinguished stopping from success evaluation in Appendix E. The policy uses the robot's current pose and the target geometric center estimated from real-time perception. It issues STOP when the detected target is visible and its estimated distance satisfies the stopping threshold. Habitat then uses the robot pose and target ground-truth pose to measure robot-center-to-target-center distance and determine success. Success thresholds are 0.20 m on HM3Dv1, HM3Dv2, and MP3D, and 0.25 m on HM3D-OVON Unseen. We have clarified both distance sources and their roles.

During baseline reproduction, we observed that cabinet and table edges could keep the robot outside 0.1 m of a target center despite reaching its vicinity. We increased the center-distance thresholds to reduce continued navigation caused by these constraints and have explained this choice. This criterion differs from target-viewpoint distance. Unified-protocol results require reevaluation with matched distance definitions, stopping rules, and thresholds; current results use the conditions stated above.

Question 2

Please see the combined response above for stopping conditions. Figure 4(b) uses HM3Dv1, as now stated in the caption. Frontier exhaustion alone does not trigger a forced STOP.

The approximately 90% of episodes ending before 500 steps include successes and failures. Execution failures or incorrect stopping can end episodes early. A false-positive target may satisfy the estimated stopping distance, while Habitat rejects success using the actual target position. Termination and success therefore have different statistical meanings.

Question 4

We partially agree. RATIO can accommodate smaller LLMs such as Llama and Qwen. Given navigation's reasoning and tool-calling requirements, we prioritized GPT-5.6 Luna and DeepSeek V4.1 Flash through official APIs, with frozen parameters and no navigation-specific training. Future appendix experiments will compare LLM scales in setting C and the full system.

Question 5

We fully agree that the scope of experience memory was omitted from the description. Our zero-shot setting evaluates each episode independently. Experience retrieval retains only historical segments and their actual outcomes generated earlier in the current episode, and its memory is reset at the start of every episode. The reported results already use this setting. Experience retrieval therefore does not depend on records from other evaluation episodes and introduces no order dependence through experience accumulated across episodes. We have clarified this in Table 5 of Appendix D and in Appendix E.

Question 6

Thank you for this insight. This was an omission in the description. Frontier recovery is a recovery strategy in the navigation control process: when the executable navigation graph has no selectable nodes, the system resumes exploration using geometric frontiers. We have added this explanation to the caption of Figure 3.

Question 7

Thank you for the suggestion. We fully agree. In future work, we will report total tokens per episode, end-to-end runtime, and API costs, and compare with CogNav and Fast–Slow ReAct under comparable hardware, model-service, and budget conditions to further analyze computational overhead and navigation efficiency.

Question 8

Backbone A uses purely geometric frontier exploration and selects navigation goals from frontiers in the geometric occupancy grid map. It shares perception, mapping, and path planning modules, providing the basis for adding the graph, LLM decisions, and tool interaction. We have clarified this omitted policy description in Section 4.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for recognizing RATIO's design and for your suggestions on experimental rigor. Below, we clarify the comparison settings, information sources, and independent contribution, and describe the corresponding revisions and planned experiments.

Weakness 1

We agree that component configurations should be clearer and have specified them in the revision. Existing object navigation methods use different perception, mapping, and decision designs, so external comparisons reflect complete systems, and matching every component is difficult. RATIO uses SAM 3 to integrate target detection and instance segmentation, reducing the number of perception modules. Our focus is the executable navigation graph and task-adaptive tool orchestration, with semantic segmentation serving as a perception component. Internal ablations share perception, map updates, path planning, and the executor; their results support the graph, LLM decisions, and tool interaction.

Weakness 2 and Question 2

We have clarified the construction of the executable navigation graph. Environmental inputs consist of real-time RGB-D observations and an odometry pose, without preloading target-instance ground-truth poses from the simulator. Section 3.2.1 describes depth projection using odometry poses and ray casting to construct the geometric occupancy grid map. In Section 3.2.2, segmentation and semantic fusion across frames provide object point clouds and cell-wise semantic scores; free cells satisfying the semantic response threshold and object-cluster projections form semantic regions. Appendix B and Section 3.3.1 describe incremental frontier extraction, clustering, and observation positions generated around semantic regions, which produce executable navigation nodes. Path planning and region relations on the current map provide edge attributes.

Thus, target candidates, semantic regions, and edge attributes derive from online observations without additional privileged information. Robot pose processing follows ApexNav under the accurate-pose assumption. The compared methods use the same types of raw inputs: target category, RGB-D observations, and robot pose. Table 2 additionally shares perception and navigation components across ablations, supporting the full tool interaction design under matched input conditions.

Weakness 3 and Question 1

We fully agree that the independent contribution needed clarification. Table 1 compares semantic exploration, map reasoning, and LLM decision methods, while progressive ablations with shared components support the graph, LLM decisions, and tool interaction (Table 2).

The core innovation combines the executable navigation graph, tool-selection mechanism, and LLM. The graph provides task evidence and navigable positions; the LLM selects tools according to the task state and uses tool feedback to adjust subsequent decisions. Most existing tool-using navigation methods organize tools around perception, planning, and control, and modular methods already support component replacement. For a fixed environment, the best-performing perception tool can generally be selected in advance. RATIO considers that different navigation stages may require different strategies. Its tools address evidence acquisition, target candidate search, broad exploration, and multi-view verification, allowing the LLM to select capabilities as navigation progresses while preserving precise and reproducible spatial computation.

We have revised the Introduction's method overview and contributions and the final paragraph of Section 2.3 to explain this combination and the tools' stage-specific roles. Section 3.4.2 describes the tools, and Figure 4(a) shows changing tool use across stages. We have connected this behavior analysis with Table 2 to explain the design's value for navigation success and path efficiency.

Weakness 4

We fully agree. In future work, we will conduct a more detailed performance analysis using per-decision latency, tool calls per episode, and the incorrect STOP rate to examine decision overhead, tool use, and stopping reliability. We will also analyze failure cases to examine how individual stages affect navigation outcomes.

Question 3

The ablations in Section 4.3 share perception, map updates, path planning, and the executor. The results from C to D demonstrate the overall contribution of tool interaction and tool feedback history. Table 4 in Appendix D identifies the tool sources: these tools organize existing navigation capabilities as callable tools, preserving their underlying capabilities. Based on this design and the experimental results, we consider the LLM's task-dependent tool selection and its use of tool outputs to adjust navigation decisions an important source of the performance gains.

We agree that matching or randomizing tool outputs is valuable. Future work will compare the full system with random tool selection and random executable-node selection across object navigation tasks, keeping perception, the executable navigation graph, tool implementations, and budgets fixed, to examine tool selection and node decisions separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.