acceptodds
Under review as a conference paper at ICLR 2027

NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime

11 views

Abstract

Language models trained with long-horizon agentic reinforcement learning can generalize knowledge through reasoning, express precise actions, and pursue goals over many steps, raising the ceiling on what an embodied agent can understand and decide. Physical interaction, however, remains the domain of action policies, which provide dense, low-latency control. We present NavGPT-3, a harness that connects the two models, with an OS-like runtime built above it: reasoning, acting, and monitoring run as threads with their own context, tools, and permissions, while the runtime schedules them and decides which thread controls the robot's motion, so that the robot can react to sudden real-world events through interruption and thread switching. Beneath it, our action policy NavGPT VLA, trained on 19.28M examples, allocates visual tokens using codec allocation, in proportion to scene change; its 8B model alone reaches 74.51 SR on R2R-CE and leads RxR-CE with 78.19 SR. With the complete harness, NavGPT-3 sets the state of the art on R2R-CE (81.51 SR) and, for the first time, brings an autonomous agent to human level: on RxR-CE it matches human followers in success (90.43 vs. 90.4 SR) and path fidelity (78.47 vs. 77.7 nDTW) at 1 min 22 s per episode, versus roughly 3 min for a human. We comprehensively ablate the harness design and the interaction between the two models, showing how tools and the action policy shape the path from language-model reasoning to physical control: when NavGPT VLA executes the route, the reasoning loop shortens and the system's minimum reaction time falls from 3–19 s per language-model decision to 0.5–1 s per action-policy step (1–2 Hz). These results show that designing this embodied interface is central to connecting frontier language-model intelligence with low-level physical control. We will release all models, code, and evaluation records.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-and-Language Navigation, Embodied AI, MLLM, LLM, Agentic Embodied System, Harness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_276sOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NavGPT-3 is a harness for embodied agents that runs multiple low-level robot motion action policies in threads, and have a OS-like runtime schedule and monitor the threads so that it can react to sudden real-world events through interruption and thread switching. A planner controls semantic sequencing and task completion, a VLA supplies high-frequency execution, and a harness connects the planner and the VLA with the environment. An OS-like runtime controls logical threads, their lifecycle and permissions, and motion authority. For the VLA to run real-time and to manage the token budget, a change score is computed to select the most relevant image frames to pass to the VLA.

Critical strengths
  • NavGPT-3 contributes substantial redesign and improvements to the embodied system, adding a reasoning harness that drives decision making of the embodied agent, whereas NavGPT-2’s reasoning capabilities were limited in scope.
  • The separation of an OS-like runtime and threaded action policies are a sophisticated design, moving towards a more robust system required in robotic tasks (analogous to state machines in classical robotics, but via a harness rather than hard-coded policies)
  • The concept of motion authority is interesting and useful to enable real-time execution of actions or intervention by safety checks, while longer planning is running in the background.
Critical weaknesses
  • The introduction paragraphs (as well as the rest of the paper) have strong characteristics of AI-written text (a lot of colons, semi-colons, listing, fragmented sentences that make sense individually but sound clunky). While it is acceptable to use AI as an assistive tool to write the text, it makes it difficult for the reader to parse the text, since the flow is unnatural and the sentences do not connect well. For instance, there are lists of items every few lines (e.g. LL: 056-057, LL: 059-060, LL: 068-069, LL: 071-072, LL: 075-076, LL: 077-079). Use lists sparingly. They carry a lot of information, but they obscure the bigger picture and progression of the narrative.
  • In section 3.2.1., the authors introduce a change score to allocate tokens for images that changed. However, this seems like a crude metric, since the robot camera can rotate ever so slightly which would result in a drastic change in pixel values, so correcting for the camera positions may make more sense.
  • In the experiments, it is difficult to assess whether the NavGPT-3 method out-performs other baselines because of the method itself (runtime + planner + VLA setup), or because of using a stronger underlying model (Opus 5, GPT-6 Astra), since some other baselines use weaker LLMs as planners / orchestrators.

In general, the text needs a significant rewrite for better readability. The method itself seem interesting and have meaningful contribution.

Questions for authors
  • L012 mentions “express precise actions” for the planning agent so the distinction between long-horizon agents (L011) and the action policies (L015) is unclear.
  • L016: Unclear what “the two models” refer to, since it is not introduced before (presumably about the planner vs lower-level VLA).
  • L162: What does “annotation and termination preserve meaning” mean?
  • LL203-205: What do you mean by “Planning during motion, awaiting a decision before acting, and responding to an external event are coordination patterns, not fixed runtime components.”?
  • How does the system perform with the runtime features turned off? Can any of the performance gains be attributed to the runtime itself as opposed to the planner / VLA?
  • Are there any assumptions about sensory information (e.g. depth, odometry) by the harness, and do they match the assumptions by other baselines?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 276s (Part 1)

We thank the reviewer for the careful reading and the specific line references.

W1 and Q1–Q4.

"The introduction paragraphs (as well as the rest of the paper) have strong characteristics of AI-written text (a lot of colons, semi-colons, listing, fragmented sentences […]). […] Use lists sparingly. They carry a lot of information, but they obscure the bigger picture and progression of the narrative."

Thank you for this feedback. We agree with the reviewer that the listed passages were hard to follow. In the revised paper we rewrote the list-style sentences at LL056–079 of the introduction as connected sentences, and we revised the four passages the reviewer asked about. All changes are marked in blue.

PassageRevised wording
L012, "express precise actions""express actions as tool calls". A language model acts through tool calls, one decision at a time, whereas an action policy outputs motion directly and at high frequency.
L016, "the two models""the language-model Planner and the action policy".
L162, "annotation and termination preserve meaning""An annotation attaches semantic information to a place so that later calls can refer to it. Termination is the Planner's explicit decision that the instruction is complete."
LL203–205, "coordination patterns, not fixed runtime components""A main thread can run a VLA operation while a background thread reasons over the executed route and a monitor watches for external events. These threads run concurrently, and only one of them may move the robot at a time."

Q5.

"How does the system perform with the runtime features turned off? Can any of the performance gains be attributed to the runtime itself as opposed to the planner / VLA?"

Thank you for this question. We answer its two parts step by step.

(1) Can any of the benchmark gains be attributed to the runtime? We would like to clarify that existing benchmarks do not capture the advantage of asynchronous execution. R2R-CE and RxR-CE measure navigation capability, mainly as success rate (Tables 1 and 3), and cannot evaluate the value of the runtime's asynchronous execution. The advantage of the runtime lies in inference time and latency on a real device, so it is the measurement on the robot that shows it.

(2) What the runtime contributes. The runtime is responsible for reaction time on a real robot. Without it, execution is sequential: every tool call, including a call to the VLA, makes the caller wait for the result, which produces time bubbles. The runtime represents the Planner's reasoning, the VLA's action, and a LiDAR monitor as separate threads and runs them concurrently, which compresses the bubbles.

(3) How the system performs with the runtime features turned off. On the Go2, we hold the models and tools fixed and turn the two runtime features off and on. According to Table 6, with both features off, the mean time to repair a route is 12.8 s and the time to halt is 1,840 ms. Running reasoning and action concurrently reduces the repair time to 9.6 s. With the continuous monitor as well, the repair time is 7.9 s and the time to halt is 180 ms. Task success changes little, from 22 to 25 of 30 trials, so the gain of the runtime is in reaction time.

AuthorsOpenReview ↗

Response to Reviewer 276s (Part 2)

W2.

"[…] this seems like a crude metric, since the robot camera can rotate ever so slightly which would result in a drastic change in pixel values, so correcting for the camera positions may make more sense."

Thank you for this suggestion. Correcting for pose is a sensible idea, and we respectfully explain why we chose a score that does not need it. Pose correction does not fit how the policy is trained. The training data include internet video, for which no camera pose exists, and sources whose poses differ widely in quality. A score that needs pose would exclude the former and depend on a pose estimator for the latter. Our score uses the video stream alone, which is what lets it apply to the whole training mixture.

We would like to clarify that the score is not a raw comparison of pixel values. It models motion. Inspired by I/P-frame video coding, it measures the magnitude of the motion between consecutive frames of each camera, and we allocate tokens according to that magnitude. The flow fields in the figure of Section 3.2.1 visualize this motion. The score runs no network and costs a few arithmetic operations per frame.

We would also like to clarify that a slight rotation has only a small effect on the score. The score is computed on a low-resolution thumbnail, where a rotation of one degree shifts the image by less than one pixel, so the measured motion stays small. A larger turn does raise the score, and it also brings new content into view, which is when a frame deserves more tokens. The score only redistributes a fixed token budget, with a floor and a cap for every image, so its effect on any one frame is bounded.

According to the codec study in Appendix C, the training recipe that uses this allocation improves the intermediate 8B policy from 69.5 to 71.6 SR on R2R-CE and from 74.4 to 77.0 on RxR-CE. The recipe also tunes the vision encoder, so this result supports the recipe as a whole.

W3.

"[…] it is difficult to assess whether the NavGPT-3 method out-performs other baselines because of the method itself (runtime + planner + VLA setup), or because of using a stronger underlying model (Opus 5, GPT-6 Astra), since some other baselines use weaker LLMs as planners / orchestrators."

Thank you for raising this concern. For the policy, we would like to clarify that the comparison involves no frontier model. NavGPT VLA is evaluated alone in the same setting as the other policies. According to Table 3, the 8B model reaches 82.39 SR on RxR-CE against 76.5 for the best prior method. On R2R-CE it reaches 76.94 SR, second to Robostral Navigate at 77.4, with lower SPL.

For the complete system, we agree that the rows with Astra and Opus are not directly comparable with baselines that use weaker language models, and we do not rank them that way. Their purpose is to show how far frontier reasoning can raise navigation and what limits it. Table 1 holds the Planner fixed for this reason. It shows that a strong model with primitive actions is already accurate but slow and costly, that spatial tools raise its success, and that the VLA shortens its action update from seconds to 0.78 s (details in the response to Reviewer M5Aw, Q5).

Q6.

"Are there any assumptions about sensory information (e.g. depth, odometry) by the harness, and do they match the assumptions by other baselines?"

Thank you for this question. Yes, the complete system makes such assumptions. Specifically, the VLA itself uses only RGB and the instruction. Its 8B model takes four views, as Qwen-RobotNav does in its panoramic configuration. The complete system in simulation additionally uses depth and simulator pose to build its map. Section 4.4 of the revised paper now states this difference. On the Go2, the VLA uses the front RGB camera, and the onboard LiDAR serves only the protective monitor.

Reviewer_276sOpenReview ↗

Thank you for your response

Thank you for the response and clarifications. With the clarifications and promised improvements to the writing of the paper, I am leaning towards a weak accept, but with the condition that reviewer 44ku's review will be disregarded, since they did not perform a proper comprehensive review. (No matter how well the paper is written, identifying no weaknesses shows lack of thoroughness of the review more than the quality of the paper.) However this problem is with the reviewer rather than the authors, so I will update my scores.

Thank you for your constructive feedback and reassessment. We will carefully incorporate the promised writing improvements in the revised manuscript.

Reviewer_44kuOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

NavGPT-3 combines a high-level language-model planner with a trained navigation VLA. The planner receives route evidence, maps, and execution feedback and can inspect progress, return to visited locations, and repair incomplete instruction execution. The VLA uses a Qwen3-VL backbone and a waypoint prediction head, trained with navigation and language/grounding supervision. Additional contributions include scene-change-based visual token allocation and a runtime coordinating reasoning, action execution, monitoring, and priority interruption.

Critical strengths
  1. The method has a coherent motivation beyond a generic fast/slow architecture. Making long VLA executions inspectable and repairable addresses missed instruction clauses and premature termination. Distinguishing policy termination from semantic task completion is particularly well motivated.
  2. Controlled comparisons support the usefulness of the interface. With the same Astra planner and 8B VLA, adding map/backtracking tools improves SR from 75% to 82% while reducing reported token use from 227.9k to 155.0k on the fixed 100-episode subset. This supports improved use of the models without increasing reported token consumption; it does not establish lower total compute or end-to-end cost.
  3. Full-split gains are substantial and include trajectory quality. Relative to the final 8B VLA, the Astra system improves R2R SR from 74.51% to 81.51% and RxR SR from 78.19% to 90.43%. RxR SPL improves from 68.98 to 74.31. These results support meaningful system-level utility.
  4. The paper separately tests visual allocation and runtime mechanisms. Visual allocation improves results across model sizes and tasks in an intermediate training configuration. The Go2 factorial experiment separates overlapping execution from priority interruption, providing more direct evidence than a qualitative demonstration of responsiveness.
Critical weaknesses

I did not identify a major weakness that undermines the main results.

Questions for authors
  1. Beyond the existing benchmark comparisons with Qwen-RobotNav and NavMCP, which specific interface or runtime mechanisms do the authors consider the main methodological advance? A concise comparison would help distinguish the contribution from improvements in the underlying navigation policy.
  2. More frequent inspection improves navigation performance but increases token consumption. What inspection frequency would the authors recommend in practice to balance task success, responsiveness, and inference cost?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 44ku

We thank the reviewer for the encouraging review and the two questions.

Q1.

"Beyond the existing benchmark comparisons with Qwen-RobotNav and NavMCP, which specific interface or runtime mechanisms do the authors consider the main methodological advance? A concise comparison would help distinguish the contribution from improvements in the underlying navigation policy."

Thank you for this suggestion. We give the concise comparison below.

SystemMain contribution
Qwen-RobotNavA navigation model with configurable navigation modes and observation settings, which returns the images of its trajectory to an upper-level agent
NavMCPSeparate channels for intent, observation, and persistent memory, used to answer embodied questions
NavGPT-3A comprehensive study of the harness needed for embodied navigation, VLA delegation that reduces token use and reaction time, a runtime above the harness that runs reasoning, action, and monitoring asynchronously, and a navigation model trained with codec token allocation

Our advance over these systems is fourfold. We comprehensively study what harness is actually needed for embodied navigation, namely which context and tools let a frontier model navigate well (Table 1, Section 4.2). We identify the problem that such a harness causes, which is high token consumption and slow reaction. We solve this problem with VLA delegation inside the harness and a runtime above it, which runs reasoning, action, and monitoring asynchronously so that the robot does not stop while the Planner reasons. Finally, we train a stronger video navigation model with codec token allocation for streaming video input. As reported in Table 3, it reaches 82.39 SR on RxR-CE, against 76.5 for Qwen-RobotNav 8B.

Q2.

"More frequent inspection improves navigation performance but increases token consumption. What inspection frequency would the authors recommend in practice to balance task success, responsiveness, and inference cost?"

Thank you for this practical question. Our recommendation is that the inspection frequency should follow how well the VLA can execute the task.

(1) In-domain tasks need few inspections. When the task is within the VLA's training distribution, as in the benchmarks, the VLA executes reliably. We recommend that the Planner inspect only when a VLA operation returns, which is the released default and the cheapest setting.

(2) Real-world tasks need more frequent inspection. In a real environment the VLA generalizes less well, and more frequent inspection lets the Planner catch the errors of the smaller model and correct them promptly.

(3) The practical limit is continuous motion, not every step. Because the runtime runs inspection concurrently with execution, the robot does not stop while the Planner inspects. How often the Planner can inspect then depends on the network and the GPU resources available. We recommend inspecting as frequently as these resources allow while the robot still moves continuously. Inspecting at every single step is not necessary.

Thank you to the authors for their careful response. The comparison with related work and the explanation of the inspection-frequency trade-off address my questions. Good luck!

Thank you for your thoughtful feedback and for confirming that our responses addressed your questions. We sincerely appreciate your time and consideration.

Reviewer_M5AwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents NavGPT-3, a hierarchical navigation framework that integrates an LLM-based planner, a vision-language-action (VLA) policy, and a runtime system. The proposed Harness provides spatial tools, context management, and execution feedback to support long-horizon navigation, while the runtime coordinates asynchronous execution, motion authority, and safety monitoring. In addition, the VLA predicts waypoint sequences and uses scene-change-aware visual token allocation to improve navigation efficiency. Experiments in simulation and real-world environments demonstrate the effectiveness of the overall system.

Critical strengths

This paper presents a relatively complete navigation framework that combines high-level planning based on LLM, action execution based on VLA, and runtime coordination to support long-range navigation. This framework supports tool-assisted spatial reasoning, execution feedback, route recovery, and asynchronous control for safety. The authors conducted extensive experiments in multiple navigation benchmarks and real robot scenarios, demonstrating the practical application potential of this framework. Additionally, ablation experiments analyzed the impact of different tools and Planner–VLA interactions on navigation performance and computational overhead, providing valuable references for understanding the roles of each component.

Critical weaknesses

1、The navgpt VLA predicts eight waypoints but executes only the first one before generating a new prediction. However, the paper does not clarify the typical spatial distance between consecutive waypoints. If the waypoint spacing is too large, the environment may change substantially before the next prediction, undermining the intended benefit of closed-loop execution. Conversely, if the spacing is too small, frequent replanning may introduce considerable inference overhead and reduce real-time efficiency. Could the authors report the distribution of inter-waypoint distances and evaluate how waypoint spacing and execution frequency affect navigation performance and real-world deployment efficiency?

2、The proposed Harness appears to primarily serve as a tool integration and orchestration layer. Many of its functions, such as obtaining first-person views from the robot, accessing environmental maps, and constructing occupancy maps, can either be directly provided by simulators or physical robot systems or are relatively standard operations. Therefore, the authors should further clarify what substantive novelty the Harness offers beyond the unified organization and orchestration of these tools. Furthermore, how does the proposed Harness differ from or relate to the concept of Harness Engineering currently discussed in the AI community? Is the authors' use of the term Harness intended to reflect or build upon the principles of Harness Engineering? The authors are encouraged to clarify the core technical contributions of the Harness and distinguish them from existing approaches.

3、Figures 1 and 2 contain several abbreviations and symbols whose meanings are not adequately explained in the captions or legends, making the figures difficult to interpret.

4、The existing ablation studies demonstrate the effectiveness of tools such as Map, Backtrack, and VLA, but do not sufficiently establish the independent contribution of the Harness itself. Incorporating map information or a VLA policy can inherently improve navigation performance, which does not necessarily demonstrate the novelty of the Harness, as these tools primarily provide additional navigational information and execution capabilities. For example, maps are already commonly used as inputs in many navigation methods. The authors should better distinguish the performance gains attributable to the added tool capabilities from those arising specifically from the Harness architecture.

5、The paper utilized powerful closed-source models such as GPT-6 Astra and Claude Opus 5, while many comparison methods employed smaller open-source models or their fine-tuned versions. Therefore, it is currently unclear whether the performance improvement of NavGPT-3 mainly stems from the proposed system architecture or the stronger reasoning and tool utilization capabilities of the underlying LLM. In particular, GPT-6 Astra itself may already possess strong robot navigation and control capabilities. It is recommended that the authors conduct more comparable experiments to further distinguish the performance gains brought about by the underlying model capabilities and the proposed architecture separately.

Questions for authors

see the weakness.

8 comments
AuthorsOpenReview ↗

Response to Reviewer M5Aw (Part 1)

We thank the reviewer for the detailed and constructive review.

Q1.

"The navgpt VLA predicts eight waypoints but executes only the first one before generating a new prediction. However, the paper does not clarify the typical spatial distance between consecutive waypoints. […] Could the authors report the distribution of inter-waypoint distances and evaluate how waypoint spacing and execution frequency affect navigation performance and real-world deployment efficiency?"

Thank you for this question. We first explain how the waypoints are defined (1), and then answer the three requests in turn: the distribution of inter-waypoint distances (2), the effect of the executed waypoint on navigation performance (3), and real-world deployment efficiency (4). We respectfully note that the spacing does not set how often the system replans, as (3) and (4) explain.

1. Waypoints are normalized, so their physical spacing depends on the embodiment. The policy is trained across embodiments, from indoor robots to outdoor driving and internet video, whose motion differs by two orders of magnitude. We therefore train on waypoints normalized by scales specific to each embodiment, for example (1.0 m, 0.433 m, 2.094 rad) for Habitat navigation and (55.3 m, 11.2 m, 2.7 rad) for nuScenes driving. The model predicts eight cumulative (x, y, θ) waypoints in this normalized space, and the scales of the deployed embodiment convert them back to metres and radians. The same normalized prediction is thus a short, tightly curved path for an indoor robot and a long, gently curved one for a car. The prompt tells the model which embodiment it is controlling.

2. Distribution of inter-waypoint distances. For indoor navigation, after conversion to metres, 3,196 predictions of the 4B policy on 30 RxR-CE episodes give:

QuantityMean ± SD
Spacing between consecutive waypoints0.078 ± 0.027 m
Distance to the executed waypoint (the second of the eight)0.160 ± 0.062 m
Distance to the eighth waypoint (8 × mean spacing)0.63 m

3. Effect on navigation performance. The benchmark executes the second waypoint and predicts again from a new observation. We choose a near waypoint because it is the most accurate. Accuracy falls with distance along the prediction, and executing the eighth waypoint instead lowers the success rate by 1.32% for the 4B model on the R2R-CE Val-Unseen split. The drop is small because even the eighth waypoint is only about 0.63 m ahead, inside the view that produced it, so the scene changes little between predictions.

4. Real-world deployment efficiency. On a robot, execution is asynchronous: the policy predicts continuously, and the robot tracks the latest prediction without stopping. During execution, we monitor the inference rate on the GPU and adjust which waypoint the robot tracks. The condition is that the distance the robot travels between two predictions stays within the distance to the tracked waypoint. According to Table 1, our 8B model takes 0.78 s per prediction on a single 4090 GPU, which is 1.28 predictions per second. At an assumed speed of 0.3 m/s, the robot travels 0.47 m between receiving one prediction and the next. The eighth waypoint is 0.63 m ahead, so the condition holds and the robot never stops. When the inference rate is higher, the robot tracks a nearer and more accurate waypoint. Predicting eight waypoints is therefore what lets the system adapt to the available inference rate, and waypoint spacing adds no inference overhead.

AuthorsOpenReview ↗

Response to Reviewer M5Aw (Part 2)

Q2.

"The proposed Harness appears to primarily serve as a tool integration and orchestration layer. […] the authors should further clarify what substantive novelty the Harness offers beyond the unified organization and orchestration of these tools. Furthermore, how does the proposed Harness differ from or relate to the concept of Harness Engineering currently discussed in the AI community?"

Thank you for this question. Yes, we use "harness" in the sense of Harness Engineering, and we agree that its individual tools are standard. We respectfully argue that the paper does not claim them as new. Our contribution is threefold. We comprehensively study what harness is actually needed for embodied navigation, namely which context and tools let a frontier model navigate well (Table 1, Section 4.2). We identify the problem that such a harness causes, which is high token consumption and slow reaction. We solve this problem with VLA delegation inside the harness and a runtime above it.

1. A harness-only system consumes too many tokens and reacts too slowly. According to Table 1, with the harness alone, Astra reaches 83 SR but spends 190.8k tokens per episode and reasons for 6.2 s before every action, and Opus spends 483.4k tokens and 19.3 s for 73 SR. A robot cannot wait that long for each step. Beyond success rate, we therefore focus on token use and reaction time.

2. The VLA reduces both. According to Table 1, when the Planner hands step-by-step execution to the VLA, tokens fall to 155.0k and 185.6k and the action is updated every 0.78 s. Success stays at 82 and 79 SR, because the Planner can still inspect the returned route and correct it.

3. The runtime is a separate layer above the harness, and it is where we place the novelty. The harness is responsible for performance, but its execution is fundamentally sequential: every tool call, including a call to the VLA, makes the caller wait for the result. On a robot this produces the time bubbles. The runtime represents each activity as a separate thread and determines which threads can run concurrently. In our Go2 experiments there are three threads: the Planner reasoning over the executed route, the VLA acting, and a LiDAR monitor. The reasoning thread and the VLA thread run concurrently, and the monitor thread runs continuously in parallel with both. Like an operating system, the runtime overlaps these threads in time, which compresses the bubbles. The runtime is thus responsible for reaction time. According to Table 6, with models and tools held fixed, overlapping reasoning with action shortens the mean time to repair a route from 12.8 to 9.6 s, and the continuous monitor shortens the time to halt from 1,840 to 180 ms.

Q3.

"Figures 1 and 2 contain several abbreviations and symbols whose meanings are not adequately explained in the captions or legends, making the figures difficult to interpret."

Thank you for pointing this out. We expanded both captions so that each figure can be read on its own. Figure 1 now defines the context, call, evidence, and trajectory symbols, and the abbreviations LM, VLA, and CE. Figure 2 defines the thread lanes, motion authority, events, acknowledgements, and the interruption and resumption transitions, and states that its time axis is schematic.

AuthorsOpenReview ↗

Response to Reviewer M5Aw (Part 3)

Q4.

"The existing ablation studies demonstrate the effectiveness of tools such as Map, Backtrack, and VLA, but do not sufficiently establish the independent contribution of the Harness itself. […] The authors should better distinguish the performance gains attributable to the added tool capabilities from those arising specifically from the Harness architecture."

Thank you for this comment. We would like to respectfully clarify that we do not claim novelty in the design of the tools. The motivation behind this experiment is to determine what a frontier model can achieve when it is given a complete navigation harness, and the experiment reveals a vital problem: a harness-only system succeeds, but it consumes too many tokens and reacts too slowly for a robot. The gains that the reviewer asks us to separate are measured in three experiments: (1) measures the tool capabilities, (2) measures the harness with the tools held fixed, and (3) measures the architecture above the harness, which is the runtime.

(1) Gains from the tool capabilities (Table 1). With the Planner fixed, we add tool groups to the basic observation and movement tools, so each row of Table 1 isolates what one capability contributes. Map and Backtrack raise success from 72 to 83 SR for Astra and from 66 to 73 for Opus. The token and reaction-time figures for each configuration are discussed under Q2.

(2) The harness with the tools held fixed (Section 4.2). Apart from its tools, the harness decides which information the Planner receives and how that context is managed. It keeps the observed map and the numbered visited places across calls, and returns keyframes of each VLA rollout. Holding the tools fixed, we vary how this context is presented to the Planner in five ways. According to the visual-prompt study in Section 4.2, SR stays within 76–78 for Astra and 73–77 for Opus, while tokens vary from 160k to 210k and from 139k to 173k. What the agent needs is therefore the information itself, and its presentation mainly determines the token cost.

(3) Gains from the architecture above the harness (Section 4.4, Appendix F). The results above come from simulator benchmarks, which cannot evaluate the value of the runtime's asynchronous execution. This value appears only on a real robot. We therefore measure it on the Go2 with the models and tools fixed and only the runtime varied (Table 6), as described under Q2.

Q5.

"[…] it is currently unclear whether the performance improvement of NavGPT-3 mainly stems from the proposed system architecture or the stronger reasoning and tool utilization capabilities of the underlying LLM. In particular, GPT-6 Astra itself may already possess strong robot navigation and control capabilities. It is recommended that the authors conduct more comparable experiments […]"

Thank you for this question. We agree with the reviewer that a good part of the complete system's result comes from the Planner. The paper separates the sources as follows.

According to Table 3, without any Planner, the 8B VLA reaches 76.94 SR on R2R-CE and 82.39 on RxR-CE, against 77.4 and 76.5 for the best prior policies. This comparison involves no closed-source model. With the system fixed and the Planner changed, Opus reaches 79.01 and 84.80, and Astra reaches 81.51 and 90.43. The two Planners differ by up to 5.6 points, so the strength of the model clearly matters.

With the Planner fixed and the interface changed, Table 1 bears out the reviewer's conjecture in part. Given only primitive actions, Astra already reaches 72 SR, level with the trained VLA at 71 on the same episodes. It does so at 325.0k tokens, 32 turns, and 3.3 s of reasoning per step, which is impractical on a robot. Spatial tools raise the same model to 83 SR, and the VLA brings the action update to 0.78 s. Our claim is therefore about use and not about ranking: a frontier model can navigate, and the interface and the runtime are what make that ability affordable and fast enough to act on. The revised paper now separates the two readings in Section 4.4: the standalone VLA rows compare with prior policies, and the system rows show what frontier Planners reach with the complete interface.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the three reviewers for their careful reading and constructive comments. We first summarize the central claim of the paper and then address two concerns shared across the reviews. All changes in the revised paper are marked in blue.

Central claim. Our experiments show that a frontier model with good tools and prompts already navigates well. According to Table 1, GPT-6 Astra with spatial tools reaches 83 SR on the R2R-CE subset, but at 190.8k tokens per episode and 6.2 s of reasoning before every action, and Claude Opus 5 needs 483.4k tokens and 19.3 s. A robot cannot wait that long for every step. This is why many robot demonstrations of language-model agents act in chunks and not continuously. The robot executes one action, stops while the model reasons, and then executes the next. These time bubbles are the main obstacle to using frontier models in the physical world. Beyond success rate, minimal reaction time is therefore one of the metrics we focus on most, together with token use. NavGPT-3 bypasses the waiting directly: the action is updated every 0.78 s, and the robot no longer waits for the Planner's reasoning, because reasoning and acting run asynchronously. It does so with two distinct layers.

  • The harness decides what the Planner sees and what it can do. It provides the context, the spatial tools, and delegation to the VLA, and it is responsible for navigation performance and token use. According to Table 1, with the VLA executing inside the harness, tokens fall from 190.8k to 155.0k for Astra and from 483.4k to 185.6k for Opus, while success stays at 82 and 79 SR.
  • The runtime is a new layer that we introduce above the harness, and it is responsible for reaction time. Execution in a harness is fundamentally sequential: every tool call, including a call to the VLA, makes the caller wait for the result. The runtime instead represents each activity as a separate thread and, like an operating system, runs the threads concurrently. In our Go2 experiments, the Planner's reasoning, the VLA's action, and a LiDAR monitor run as three such threads. According to Table 6, overlapping reasoning with action shortens the mean time to repair a route from 12.8 to 9.6 s, and the continuous monitor shortens the time to halt from 1,840 to 180 ms.

The policy is compared with prior work without any frontier model. NavGPT VLA is evaluated alone, in the same setting as prior navigation policies, so this comparison involves no closed-source model. As reported in Table 3 of the paper, the 8B policy reaches 82.39 SR on RxR-CE, against 76.5 for the best prior method, and 76.94 SR on R2R-CE, second to Robostral Navigate at 77.4.

The complete-system results show what a frontier Planner can reach, and at what cost. As reported in the same table, the complete system reaches 81.51 SR on R2R-CE and 90.43 SR on RxR-CE with Astra, and 79.01 and 84.80 with Opus. We agree with the reviewers that these rows are not directly comparable with baselines that use weaker language models, and we do not rank them that way. Their purpose is to show how far frontier reasoning can raise navigation. As reported in Table 1, that accuracy comes with 190.8k to 483.4k tokens per episode and 6.2 to 19.3 s per action when the harness is used alone, which is the problem that the VLA and the runtime address.

Area_Chair_PyqDOpenReview ↗

Reminder: Author Response and Reviewer Discussion

Dear Reviewers,

We are halfway through the discussion period. Please review the authors’ responses, address any remaining concerns, and engage in the discussion where appropriate. If your assessment remains unchanged, please briefly acknowledge that you have considered the authors’ responses.

Thank you for your time and contributions.

AuthorsOpenReview ↗

Author Discussion Summary and Manuscript Revisions

Dear AC and Reviewers,

We sincerely thank the Area Chair for overseeing our submission and all three reviewers for their time and constructive feedback. As the discussion period comes to a close, we briefly summarize the main points and revisions below.

Acknowledged strengths. Reviewers recognized the framework’s practical potential and the analysis of tool and Planner–VLA interactions (M5Aw), the value of motion authority for concurrent planning and execution (276s), and the motivation for making extended VLA execution inspectable and repairable (44ku). Reviewer 44ku also highlighted the benchmark results and the separate evaluation of visual allocation and runtime mechanisms.

Main concerns and responses.

  • Contribution beyond standard tools and relation to prior work (M5Aw, 44ku). We clarified that individual navigation tools are standard. Our study examines the context and capabilities needed for frontier-model navigation, the token and reaction-time bottlenecks that remain, and how VLA delegation and a separate runtime address them. We also provided a direct comparison with Qwen-RobotNav and NavMCP.
  • Runtime attribution (M5Aw, 276s). We distinguished navigation capability from physical execution latency. In the Go2 comparison with models and tools fixed, overlap reduces mean route-repair time from 12.8 to 9.6 s under the same queued event policy. Combining overlap with priority interruption reduces p95 halt latency from 1,840 to 180 ms (Table 6).
  • Underlying model strength (M5Aw, 276s). We acknowledged the contribution of frontier Planners and separated those system results from the standalone policy comparison. Without a Planner, the 8B NavGPT VLA achieves 82.39 SR on RxR-CE, compared with Qwen-RobotNav 8B’s reported 76.5 (Table 3). Table 1 examines tool availability and VLA delegation for each Planner, reporting both navigation performance and resource use.
  • Waypoint execution and inspection frequency (M5Aw, 44ku). We explained embodiment-specific waypoint normalization, the distinction between benchmark execution and asynchronous robot control, and how tracked targets relate to inference rate. We recommend few checks for reliable in-domain execution and more frequent review for unfamiliar real-world tasks as resources permit continuous motion.
  • Codec, sensing, and readability (276s, M5Aw). We explained how the RGB change score reallocates visual tokens within a fixed budget without requiring camera pose. We clarified which sensors the standalone policy and the complete system use, and revised the introduction, component definitions, and figure captions.

Revisions. Changes are marked in blue. The revised manuscript clarifies the roles of the harness, runtime, and models, expands the symbol definitions in Figures 1 and 2, explains sensing assumptions, and brings the controlled Go2 runtime comparison into the main text. The main paper remains within ten pages.

We thank Reviewer 44ku for confirming that the related-work comparison and the explanation of inspection frequency addressed both questions. In our rebuttal, we addressed each concern raised by Reviewer 276s, including readability, codec sensitivity to camera motion, sensing assumptions, and the contributions of the Planner and runtime. We appreciate their acknowledgment of our clarifications and planned writing improvements. Reviewer M5Aw retained their rating, and we have asked which concerns remain unresolved so we can respond directly.

We sincerely thank the Area Chair for considering this summary and all reviewers for their time and feedback. We would be glad to provide any further clarification.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.