- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
NavGPT-3 is a harness for embodied agents that runs multiple low-level robot motion action policies in threads, and have a OS-like runtime schedule and monitor the threads so that it can react to sudden real-world events through interruption and thread switching. A planner controls semantic sequencing and task completion, a VLA supplies high-frequency execution, and a harness connects the planner and the VLA with the environment. An OS-like runtime controls logical threads, their lifecycle and permissions, and motion authority. For the VLA to run real-time and to manage the token budget, a change score is computed to select the most relevant image frames to pass to the VLA.
Critical strengths
- NavGPT-3 contributes substantial redesign and improvements to the embodied system, adding a reasoning harness that drives decision making of the embodied agent, whereas NavGPT-2’s reasoning capabilities were limited in scope.
- The separation of an OS-like runtime and threaded action policies are a sophisticated design, moving towards a more robust system required in robotic tasks (analogous to state machines in classical robotics, but via a harness rather than hard-coded policies)
- The concept of motion authority is interesting and useful to enable real-time execution of actions or intervention by safety checks, while longer planning is running in the background.
Critical weaknesses
- The introduction paragraphs (as well as the rest of the paper) have strong characteristics of AI-written text (a lot of colons, semi-colons, listing, fragmented sentences that make sense individually but sound clunky). While it is acceptable to use AI as an assistive tool to write the text, it makes it difficult for the reader to parse the text, since the flow is unnatural and the sentences do not connect well. For instance, there are lists of items every few lines (e.g. LL: 056-057, LL: 059-060, LL: 068-069, LL: 071-072, LL: 075-076, LL: 077-079). Use lists sparingly. They carry a lot of information, but they obscure the bigger picture and progression of the narrative.
- In section 3.2.1., the authors introduce a change score to allocate tokens for images that changed. However, this seems like a crude metric, since the robot camera can rotate ever so slightly which would result in a drastic change in pixel values, so correcting for the camera positions may make more sense.
- In the experiments, it is difficult to assess whether the NavGPT-3 method out-performs other baselines because of the method itself (runtime + planner + VLA setup), or because of using a stronger underlying model (Opus 5, GPT-6 Astra), since some other baselines use weaker LLMs as planners / orchestrators.
In general, the text needs a significant rewrite for better readability. The method itself seem interesting and have meaningful contribution.
Questions for authors
- L012 mentions “express precise actions” for the planning agent so the distinction between long-horizon agents (L011) and the action policies (L015) is unclear.
- L016: Unclear what “the two models” refer to, since it is not introduced before (presumably about the planner vs lower-level VLA).
- L162: What does “annotation and termination preserve meaning” mean?
- LL203-205: What do you mean by “Planning during motion, awaiting a decision before acting, and responding to an external event are coordination patterns, not fixed runtime components.”?
- How does the system perform with the runtime features turned off? Can any of the performance gains be attributed to the runtime itself as opposed to the planner / VLA?
- Are there any assumptions about sensory information (e.g. depth, odometry) by the harness, and do they match the assumptions by other baselines?
4 comments
Response to Reviewer 276s (Part 2)
W2.
"[…] this seems like a crude metric, since the robot camera can rotate ever so slightly which would result in a drastic change in pixel values, so correcting for the camera positions may make more sense."
Thank you for this suggestion. Correcting for pose is a sensible idea, and we respectfully explain why we chose a score that does not need it. Pose correction does not fit how the policy is trained. The training data include internet video, for which no camera pose exists, and sources whose poses differ widely in quality. A score that needs pose would exclude the former and depend on a pose estimator for the latter. Our score uses the video stream alone, which is what lets it apply to the whole training mixture.
We would like to clarify that the score is not a raw comparison of pixel values. It models motion. Inspired by I/P-frame video coding, it measures the magnitude of the motion between consecutive frames of each camera, and we allocate tokens according to that magnitude. The flow fields in the figure of Section 3.2.1 visualize this motion. The score runs no network and costs a few arithmetic operations per frame.
We would also like to clarify that a slight rotation has only a small effect on the score. The score is computed on a low-resolution thumbnail, where a rotation of one degree shifts the image by less than one pixel, so the measured motion stays small. A larger turn does raise the score, and it also brings new content into view, which is when a frame deserves more tokens. The score only redistributes a fixed token budget, with a floor and a cap for every image, so its effect on any one frame is bounded.
According to the codec study in Appendix C, the training recipe that uses this allocation improves the intermediate 8B policy from 69.5 to 71.6 SR on R2R-CE and from 74.4 to 77.0 on RxR-CE. The recipe also tunes the vision encoder, so this result supports the recipe as a whole.
W3.
"[…] it is difficult to assess whether the NavGPT-3 method out-performs other baselines because of the method itself (runtime + planner + VLA setup), or because of using a stronger underlying model (Opus 5, GPT-6 Astra), since some other baselines use weaker LLMs as planners / orchestrators."
Thank you for raising this concern. For the policy, we would like to clarify that the comparison involves no frontier model. NavGPT VLA is evaluated alone in the same setting as the other policies. According to Table 3, the 8B model reaches 82.39 SR on RxR-CE against 76.5 for the best prior method. On R2R-CE it reaches 76.94 SR, second to Robostral Navigate at 77.4, with lower SPL.
For the complete system, we agree that the rows with Astra and Opus are not directly comparable with baselines that use weaker language models, and we do not rank them that way. Their purpose is to show how far frontier reasoning can raise navigation and what limits it. Table 1 holds the Planner fixed for this reason. It shows that a strong model with primitive actions is already accurate but slow and costly, that spatial tools raise its success, and that the VLA shortens its action update from seconds to 0.78 s (details in the response to Reviewer M5Aw, Q5).
Q6.
"Are there any assumptions about sensory information (e.g. depth, odometry) by the harness, and do they match the assumptions by other baselines?"
Thank you for this question. Yes, the complete system makes such assumptions. Specifically, the VLA itself uses only RGB and the instruction. Its 8B model takes four views, as Qwen-RobotNav does in its panoramic configuration. The complete system in simulation additionally uses depth and simulator pose to build its map. Section 4.4 of the revised paper now states this difference. On the Go2, the VLA uses the front RGB camera, and the onboard LiDAR serves only the protective monitor.
Thank you for your response
Thank you for the response and clarifications. With the clarifications and promised improvements to the writing of the paper, I am leaning towards a weak accept, but with the condition that reviewer 44ku's review will be disregarded, since they did not perform a proper comprehensive review. (No matter how well the paper is written, identifying no weaknesses shows lack of thoroughness of the review more than the quality of the paper.) However this problem is with the reviewer rather than the authors, so I will update my scores.
Thank you for your constructive feedback and reassessment. We will carefully incorporate the promised writing improvements in the revised manuscript.
Response to Reviewer 276s (Part 1)
We thank the reviewer for the careful reading and the specific line references.
W1 and Q1–Q4.
Thank you for this feedback. We agree with the reviewer that the listed passages were hard to follow. In the revised paper we rewrote the list-style sentences at LL056–079 of the introduction as connected sentences, and we revised the four passages the reviewer asked about. All changes are marked in blue.
Q5.
Thank you for this question. We answer its two parts step by step.
(1) Can any of the benchmark gains be attributed to the runtime? We would like to clarify that existing benchmarks do not capture the advantage of asynchronous execution. R2R-CE and RxR-CE measure navigation capability, mainly as success rate (Tables 1 and 3), and cannot evaluate the value of the runtime's asynchronous execution. The advantage of the runtime lies in inference time and latency on a real device, so it is the measurement on the robot that shows it.
(2) What the runtime contributes. The runtime is responsible for reaction time on a real robot. Without it, execution is sequential: every tool call, including a call to the VLA, makes the caller wait for the result, which produces time bubbles. The runtime represents the Planner's reasoning, the VLA's action, and a LiDAR monitor as separate threads and runs them concurrently, which compresses the bubbles.
(3) How the system performs with the runtime features turned off. On the Go2, we hold the models and tools fixed and turn the two runtime features off and on. According to Table 6, with both features off, the mean time to repair a route is 12.8 s and the time to halt is 1,840 ms. Running reasoning and action concurrently reduces the repair time to 9.6 s. With the continuous monitor as well, the repair time is 7.9 s and the time to halt is 180 ms. Task success changes little, from 22 to 25 of 30 trials, so the gain of the runtime is in reaction time.