- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces BATON, a LLM + VLA framework for solving long horizon manipulator tasks. In long horizon manipulation, prior works relying on end-to-end VLAs encounter exponentially growing compute cost and compounding errors that lead to low overall success rate. This work instead digs into the subtask level: it uses LLM agents to define the success boundaries between each subtask, and verifies the success of each subtask before continuing. This reduces the computation cost to linear. Furthermore, within each subtask, BATON (1) verifies the subtask is in a valid start state, (2) selects a move that is compatible with next-step geometric constraints. The authors performed experiments on the RoboMemArena benchmark and observed 37.7% improvement in success rate as compared to baseline “Harness VLA”.
Critical strengths
- Identifying the importance of verifying the start of a subtask (innovation transition) and compatibility with the future state (lookahead transition) is novel.
- Ablation experiments show all of the proposed mechanisms are effective and necessary.
Critical weaknesses
-
The paradigm of the proposed method falls into the category of hierarchical VLAs, and yet in the related works there is insufficient discussion on this topic. Furthermore, the idea of verifying subtask completion has been proposed in the work “SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Model (Yang et al.)”. In addition, similar ideas like the “handoff transition” were introduced as the “initiation set” in “Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining (Konidaris et al.)”.
-
It is not meaningful to compare against single-stage end-to-end VLAs that are apparently weaker in reasoning capability. Unlike these single-stage methods, the proposed method has better capability, but more execution cost.
-
The formulation described by Eq.2 assumes a linear path of subtasks. This does not fully represent the reality where multiple paths could lead to the same completion result.
-
In Section 4, it is recommended to analyze the effectiveness of using SAM-3 as the verifier since it is prone to occlusions. In such cases, how will the model handle verifier mistakes?
-
This work is heavily built on top of Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents [44] with the additional transition-aware mechanisms. It is a bit of an overclaim to frame the proposed method as a foundational new long-horizon manipulator framework. It is recommended to highlight more appropriately as, for example, “Subtask transition awareness for long-horizon manipulation.”
Questions for authors
-
What does “promotion” mean in Fig.1? This word is not mentioned anywhere in the text.
-
The term “coding agent” (readers may assume it is the agent that write program code) is mentioned repeatedly throughout the text, but there is no description how “coding” is done in the proposed work.
-
Line 275: it is a bit difficult to understand how do you define the success criteria for the “verified unit”. Do you mean as long as one subtask could lead to a valid start on the next subtask, it is considered successful? Does this involve an actual simulator rollout?
-
In Section 3.3, could the authors give examples of what data is stored in M?
9 comments
Response to Reviewer xvQq (2/2)
For W5 (relation to Harness VLA and BATON contribution):
Agentic robotics is becoming an increasingly important direction, and to our knowledge, we are the first to tackle its long-horizon bottleneck by breaking the task into subtasks and the transitions between them. We clarify that our novelty is identifying and resolving the bottleneck of agentic robotics, which is fundamentally different from adding mechanisms to Harness VLA.
First, BATON identifies the unique bottleneck of agentic robotics at long horizons: exploration cost. The paradigm is promising because an LLM agent can discover its own solution through exploration, without training. On long-horizon tasks, however, that exploration becomes prohibitively expensive for two reasons. Whole-task exploration cannot tell which stage caused a failure, and the subtasks depend on each other, so pieces that each succeed alone still break when chained. This bottleneck is intrinsic to the paradigm, not specific to Harness VLA.
Second, BATON addresses each of these issues directly. To identify which stage failed, BATON introduces agentic subtask exploration, which explores each subtask separately and attributes every failure to a single stage. To handle inter-subtask dependencies, BATON introduces the inter-subtask transition memories (handoff and lookahead), which record the conditions under which subtasks compose. To further secure the handover within each subtask, BATON adds the invocation transition, which hands control to the VLA only when the scene is ready. Overall, these reduce exploration cost from exponential to linear, which is why BATON performs strongly under a limited budget.
We therefore respectfully disagree that the framing is an overclaim. Our contribution lies in identifying the bottleneck that blocks agentic robotics from long-horizon tasks, and in proposing a solution that directly addresses each of the issues identified. The suggested “subtask transition awareness” covers only the transitions and omits the subtask-level exploration that they serve.
For Q1 (“promotion” in Fig. 1):
“Promotion” refers to the bottom-up composition step. Once a subtask and its boundary with the next subtask are verified, the two are merged into a larger verified unit, and this repeats until the full task is formed.
For Q2 (“coding agent”):
The term refers to the agent backbone we use, an off-the-shelf coding agent, not to code generation. In our method, the agent does not write program code; it issues structured primitive calls and reads and writes the language memory. To avoid this confusion, we have replaced “coding agent” with “LLM agent” throughout in the updated version.
For Q3 (success criteria for a verified unit):
A subtask succeeds when its own goal is achieved (e.g., the object is inside the microwave), regardless of the arm’s final state. A valid start on the next subtask is handled by the transition at the boundary, not by the subtask’s success criterion. A pair of subtasks becomes a verified unit when the successor also succeeds, starting from the state its predecessor leaves (Sec. 3.2). Both exploration and verification are performed with actual simulator rollouts.
For Q4 (BATON memory structure):
We clarify the memory structure of BATON, which is illustrated in Fig. 2 (right) together with when each memory is read and written. In brief, the memory of a long-horizon task consists of skill memory and transition memory:
Skill memory records how each subtask is solved.
Transition memory records how subtasks connect, in three kinds:
- Invocation: a readiness condition before each VLA call, e.g., release only when the wrist image shows the object inside the target.
- Handoff: an entry condition at each boundary, e.g., reset the wrist to top-down after opening the microwave door.
- Lookahead: an execution constraint from later subtasks, e.g., grasp a box at yaw 0 so it can later be lowered into the drawer.
In addition, every failed attempt during exploration is written as failure memory, which is read before the next attempt to avoid repeating the same mistake.
References:
[1] Yang et al. SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Model. arXiv:2509.14138, 2025.
[2] Konidaris and Barto. Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining. In NeurIPS, 2009.
3 Days Remaining: We would like to learn your opinion and address any remaining concerns.
Dear Reviewer xvQq,
We know this is a very busy time for reviewers, but as the discussion period is closing in 3 days, we want to send a gentle follow-up. We have submitted our response to your thoughtful questions. In our rebuttal, we did our best to address all the questions and provide additional context. Please let us know if there is anything else we can address to help resolve any remaining concerns.
Best regards,
The Authors
Response to Reviewer xvQq (1/2)
We sincerely thank you for the valuable questions and comments. Here are our responses:
For W1 (BATON vs hierarchical VLAs, SeqVLA, skill chaining):
We clarify that BATON’s contribution lies in identifying and addressing the long-horizon exploration bottleneck of agentic robotics. BATON does not belong to hierarchical VLA and differs from SeqVLA[1] and initiation sets[2].
First, BATON is an agentic robotics method, not a hierarchical VLA. In a hierarchical VLA, a high-level planner helps the VLA reason, but every action still comes from the VLA’s output. In BATON, the LLM agent produces the execution through analytic primitives and calls the frozen VLA as a function only for contact-rich segments. Moreover, we have already discussed this line of work in Sec. 2 line 181 as an adjacent paradigm: “VLM planners orchestrate VLAs as callable tools; dual-system models couple a slow reasoner to a fast policy.”
Second, BATON differs from SeqVLA[1] in both the problem and the means. SeqVLA finetunes π0 with a completion head to detect when a subtask is done, giving a trained policy an exit signal. It involves no exploration and does not check whether the successor can start, restore a disturbed entry state, or adapt execution to what follows. BATON’s transitions provide these without training.
Third, BATON differs from initiation sets[2] in how the condition is obtained and enforced. Skill chaining is already cited in Sec. 3.3 ([16]). Its initiation set is a classifier learned offline per skill. Our entry condition is stated in language, verified on live observations, attached to the boundary rather than a skill, and actively restored when violated.
We want to clarify the positioning and contribution of BATON: it identifies and addresses the exploration bottleneck that agentic robotics faces at long horizons. An LLM agent can discover its own solution through exploration, but at long horizons whole-task exploration cannot tell which stage failed and becomes prohibitively expensive. BATON explores at the subtask level to localize failures and reduce this cost, and it adds transition memories so that the separately explored subtasks compose.
For W2 (BATON vs end-to-end VLAs):
We clarify that the comparison with end-to-end VLAs is meaningful for two reasons: they show where agentic robotics stands on long-horizon tasks, and they hold advantages over BATON on factors other than reasoning.
First, these baselines show the level that agentic robotics reaches on its own. Harness VLA, a pure agentic method using the same LLM as BATON, reaches only 23.1% TSR and trails end-to-end methods such as PrediMem (38.5%) and FrameSamp+Modul (46.1%). A stronger LLM planner therefore does not by itself give agentic methods an advantage on long-horizon tasks.
Second, these baselines are privileged in training data and memory. Several have trained memory mechanisms (MemoryVLA, MemER, PrediMem), and all are trained on RoboMemArena’s own data, so their policies are tuned to these scenes. BATON uses a frozen VLA never finetuned on the benchmark, and its only adaptation is 20 exploration episodes on a single seed. That BATON outperforms these privileged baselines under such a limited budget demonstrates the superiority of our method.
For W3 (linear formulation of Eq. 2):
We clarify that Eq. 2 describes the subtask sequence of a single execution, not the only way to complete the task.
Although multiple paths may lead to the same result, any one execution follows exactly one of them as a sequence of subtasks. Completing the task requires only one valid path, so BATON’s goal is to find and verify one, and the existence of alternatives does not affect this. Moreover, BATON is not bound to the path it explored. Its memory stores language-level lessons about subtasks and transitions rather than a fixed trajectory, and the agent applies them to the current observation at runtime.
For W4 (SAM-3 as the verifier):
We clarify that the verifier is the LLM agent rather than SAM-3, and that verifier mistakes are recoverable.
First, the verifier is the LLM agent. To judge whether the gripper is aligned, the agent checks on the wrist-camera image whether the grasped object lies within the bounding box of the destination object (Sec. 3.3). SAM-3 serves only as the segmentation module that provides these regions.
Second, SAM-3 does not affect our comparison. BATON uses the same SAM-3 segmentation module as Harness VLA (Sec. 4.1), so any segmentation error affects both methods equally.
Third, verifier mistakes are recoverable in both directions. If a ready scene is judged not ready, the agent simply re-stages and checks again. If an unready scene is judged ready, the resulting failure is recorded in failure memory during exploration and avoided in later attempts. In neither case does the error go unnoticed.