acceptodds
Under review as a conference paper at ICLR 2027

Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-Aware Memory

Abstract

Long-horizon robot manipulation chains many contact-rich skills into one multi-stage task. Vision-language-action (VLA) and world-action models (WAMs) increasingly master the individual skills. However, the chain itself still fails: errors compound beyond the policy's ability to correct, and the execution of one subtask silently constrains the next. Coding agents offer a promising pathway: freeze the VLA and put an LLM agent in charge of it. The agent plans in language, moves in free space with analytic primitives, and invokes the VLA only for the contact-rich segments, writing all adaptation into language memory. Yet applied directly to long horizons, the agent-plus-VLA recipe breaks twice. 1) Its competence is acquired through whole-task exploration at test time, whose cost is exponential in the number of stages: where a single stage requires episodes on average, a -stage task requires on the order of , and a failed episode does not identify the stage responsible. 2) It has no representation of the transitions: the VLA primitive carries an exit condition but no entry condition, and a subtask can succeed in a form its successor cannot use. We present BATON, which addresses the two failures in turn. Against 1), BATON makes the subtask the unit of exploration. Each subtask is explored in the inexpensive short-horizon regime, and its solution is stored in memory. A long-horizon trajectory is then composed from these solutions rather than discovered whole. Exploration cost thus becomes linear in the number of stages (), and every failure is attributed to a single stage. Against 2), BATON equips this exploration with a transition-aware memory. Within a subtask, a verifier agent governs the invocation transition: the VLA is invoked only after the wrist view confirms that the scene is ready. Across subtasks, handoff transition restores an entry state disturbed by the predecessor's residue, and lookahead transition determines the execution strategy whose outcome the successor can inherit. On the long-horizon manipulation benchmark RoboMemArena, BATON improves task success rate by 37.7% and cumulative success rate by 29.7% over the current SoTA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
robot manipulation, coding agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xvQqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces BATON, a LLM + VLA framework for solving long horizon manipulator tasks. In long horizon manipulation, prior works relying on end-to-end VLAs encounter exponentially growing compute cost and compounding errors that lead to low overall success rate. This work instead digs into the subtask level: it uses LLM agents to define the success boundaries between each subtask, and verifies the success of each subtask before continuing. This reduces the computation cost to linear. Furthermore, within each subtask, BATON (1) verifies the subtask is in a valid start state, (2) selects a move that is compatible with next-step geometric constraints. The authors performed experiments on the RoboMemArena benchmark and observed 37.7% improvement in success rate as compared to baseline “Harness VLA”.

Critical strengths
  • Identifying the importance of verifying the start of a subtask (innovation transition) and compatibility with the future state (lookahead transition) is novel.
  • Ablation experiments show all of the proposed mechanisms are effective and necessary.
Critical weaknesses
  1. The paradigm of the proposed method falls into the category of hierarchical VLAs, and yet in the related works there is insufficient discussion on this topic. Furthermore, the idea of verifying subtask completion has been proposed in the work “SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Model (Yang et al.)”. In addition, similar ideas like the “handoff transition” were introduced as the “initiation set” in “Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining (Konidaris et al.)”.

  2. It is not meaningful to compare against single-stage end-to-end VLAs that are apparently weaker in reasoning capability. Unlike these single-stage methods, the proposed method has better capability, but more execution cost.

  3. The formulation described by Eq.2 assumes a linear path of subtasks. This does not fully represent the reality where multiple paths could lead to the same completion result.

  4. In Section 4, it is recommended to analyze the effectiveness of using SAM-3 as the verifier since it is prone to occlusions. In such cases, how will the model handle verifier mistakes?

  5. This work is heavily built on top of Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents [44] with the additional transition-aware mechanisms. It is a bit of an overclaim to frame the proposed method as a foundational new long-horizon manipulator framework. It is recommended to highlight more appropriately as, for example, “Subtask transition awareness for long-horizon manipulation.”

Questions for authors
  1. What does “promotion” mean in Fig.1? This word is not mentioned anywhere in the text.

  2. The term “coding agent” (readers may assume it is the agent that write program code) is mentioned repeatedly throughout the text, but there is no description how “coding” is done in the proposed work.

  3. Line 275: it is a bit difficult to understand how do you define the success criteria for the “verified unit”. Do you mean as long as one subtask could lead to a valid start on the next subtask, it is considered successful? Does this involve an actual simulator rollout?

  4. In Section 3.3, could the authors give examples of what data is stored in M?

9 comments
AuthorsOpenReview ↗

Response to Reviewer xvQq (1/2)

We sincerely thank you for the valuable questions and comments. Here are our responses:

For W1 (BATON vs hierarchical VLAs, SeqVLA, skill chaining):

We clarify that BATON’s contribution lies in identifying and addressing the long-horizon exploration bottleneck of agentic robotics. BATON does not belong to hierarchical VLA and differs from SeqVLA[1] and initiation sets[2].

First, BATON is an agentic robotics method, not a hierarchical VLA. In a hierarchical VLA, a high-level planner helps the VLA reason, but every action still comes from the VLA’s output. In BATON, the LLM agent produces the execution through analytic primitives and calls the frozen VLA as a function only for contact-rich segments. Moreover, we have already discussed this line of work in Sec. 2 line 181 as an adjacent paradigm: “VLM planners orchestrate VLAs as callable tools; dual-system models couple a slow reasoner to a fast policy.”

Second, BATON differs from SeqVLA[1] in both the problem and the means. SeqVLA finetunes π0 with a completion head to detect when a subtask is done, giving a trained policy an exit signal. It involves no exploration and does not check whether the successor can start, restore a disturbed entry state, or adapt execution to what follows. BATON’s transitions provide these without training.

Third, BATON differs from initiation sets[2] in how the condition is obtained and enforced. Skill chaining is already cited in Sec. 3.3 ([16]). Its initiation set is a classifier learned offline per skill. Our entry condition is stated in language, verified on live observations, attached to the boundary rather than a skill, and actively restored when violated.

We want to clarify the positioning and contribution of BATON: it identifies and addresses the exploration bottleneck that agentic robotics faces at long horizons. An LLM agent can discover its own solution through exploration, but at long horizons whole-task exploration cannot tell which stage failed and becomes prohibitively expensive. BATON explores at the subtask level to localize failures and reduce this cost, and it adds transition memories so that the separately explored subtasks compose.

For W2 (BATON vs end-to-end VLAs):

We clarify that the comparison with end-to-end VLAs is meaningful for two reasons: they show where agentic robotics stands on long-horizon tasks, and they hold advantages over BATON on factors other than reasoning.

First, these baselines show the level that agentic robotics reaches on its own. Harness VLA, a pure agentic method using the same LLM as BATON, reaches only 23.1% TSR and trails end-to-end methods such as PrediMem (38.5%) and FrameSamp+Modul (46.1%). A stronger LLM planner therefore does not by itself give agentic methods an advantage on long-horizon tasks.

Second, these baselines are privileged in training data and memory. Several have trained memory mechanisms (MemoryVLA, MemER, PrediMem), and all are trained on RoboMemArena’s own data, so their policies are tuned to these scenes. BATON uses a frozen VLA never finetuned on the benchmark, and its only adaptation is 20 exploration episodes on a single seed. That BATON outperforms these privileged baselines under such a limited budget demonstrates the superiority of our method.

For W3 (linear formulation of Eq. 2):

We clarify that Eq. 2 describes the subtask sequence of a single execution, not the only way to complete the task.

Although multiple paths may lead to the same result, any one execution follows exactly one of them as a sequence of subtasks. Completing the task requires only one valid path, so BATON’s goal is to find and verify one, and the existence of alternatives does not affect this. Moreover, BATON is not bound to the path it explored. Its memory stores language-level lessons about subtasks and transitions rather than a fixed trajectory, and the agent applies them to the current observation at runtime.

For W4 (SAM-3 as the verifier):

We clarify that the verifier is the LLM agent rather than SAM-3, and that verifier mistakes are recoverable.

First, the verifier is the LLM agent. To judge whether the gripper is aligned, the agent checks on the wrist-camera image whether the grasped object lies within the bounding box of the destination object (Sec. 3.3). SAM-3 serves only as the segmentation module that provides these regions.

Second, SAM-3 does not affect our comparison. BATON uses the same SAM-3 segmentation module as Harness VLA (Sec. 4.1), so any segmentation error affects both methods equally.

Third, verifier mistakes are recoverable in both directions. If a ready scene is judged not ready, the agent simply re-stages and checks again. If an unready scene is judged ready, the resulting failure is recorded in failure memory during exploration and avoided in later attempts. In neither case does the error go unnoticed.

AuthorsOpenReview ↗

Response to Reviewer xvQq (2/2)

For W5 (relation to Harness VLA and BATON contribution):

Agentic robotics is becoming an increasingly important direction, and to our knowledge, we are the first to tackle its long-horizon bottleneck by breaking the task into subtasks and the transitions between them. We clarify that our novelty is identifying and resolving the bottleneck of agentic robotics, which is fundamentally different from adding mechanisms to Harness VLA.

First, BATON identifies the unique bottleneck of agentic robotics at long horizons: exploration cost. The paradigm is promising because an LLM agent can discover its own solution through exploration, without training. On long-horizon tasks, however, that exploration becomes prohibitively expensive for two reasons. Whole-task exploration cannot tell which stage caused a failure, and the subtasks depend on each other, so pieces that each succeed alone still break when chained. This bottleneck is intrinsic to the paradigm, not specific to Harness VLA.

Second, BATON addresses each of these issues directly. To identify which stage failed, BATON introduces agentic subtask exploration, which explores each subtask separately and attributes every failure to a single stage. To handle inter-subtask dependencies, BATON introduces the inter-subtask transition memories (handoff and lookahead), which record the conditions under which subtasks compose. To further secure the handover within each subtask, BATON adds the invocation transition, which hands control to the VLA only when the scene is ready. Overall, these reduce exploration cost from exponential to linear, which is why BATON performs strongly under a limited budget.

We therefore respectfully disagree that the framing is an overclaim. Our contribution lies in identifying the bottleneck that blocks agentic robotics from long-horizon tasks, and in proposing a solution that directly addresses each of the issues identified. The suggested “subtask transition awareness” covers only the transitions and omits the subtask-level exploration that they serve.

For Q1 (“promotion” in Fig. 1):

“Promotion” refers to the bottom-up composition step. Once a subtask and its boundary with the next subtask are verified, the two are merged into a larger verified unit, and this repeats until the full task is formed.

For Q2 (“coding agent”):

The term refers to the agent backbone we use, an off-the-shelf coding agent, not to code generation. In our method, the agent does not write program code; it issues structured primitive calls and reads and writes the language memory. To avoid this confusion, we have replaced “coding agent” with “LLM agent” throughout in the updated version.

For Q3 (success criteria for a verified unit):

A subtask succeeds when its own goal is achieved (e.g., the object is inside the microwave), regardless of the arm’s final state. A valid start on the next subtask is handled by the transition at the boundary, not by the subtask’s success criterion. A pair of subtasks becomes a verified unit when the successor also succeeds, starting from the state its predecessor leaves (Sec. 3.2). Both exploration and verification are performed with actual simulator rollouts.

For Q4 (BATON memory structure):

We clarify the memory structure of BATON, which is illustrated in Fig. 2 (right) together with when each memory is read and written. In brief, the memory of a long-horizon task consists of skill memory and transition memory:

Skill memory records how each subtask is solved.

Transition memory records how subtasks connect, in three kinds:

  • Invocation: a readiness condition before each VLA call, e.g., release only when the wrist image shows the object inside the target.
  • Handoff: an entry condition at each boundary, e.g., reset the wrist to top-down after opening the microwave door.
  • Lookahead: an execution constraint from later subtasks, e.g., grasp a box at yaw 0 so it can later be lowered into the drawer.

In addition, every failed attempt during exploration is written as failure memory, which is read before the next attempt to avoid repeating the same mistake.

References:

[1] Yang et al. SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Model. arXiv:2509.14138, 2025.

[2] Konidaris and Barto. Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining. In NeurIPS, 2009.

AuthorsOpenReview ↗

3 Days Remaining: We would like to learn your opinion and address any remaining concerns.

Dear Reviewer xvQq,

We know this is a very busy time for reviewers, but as the discussion period is closing in 3 days, we want to send a gentle follow-up. We have submitted our response to your thoughtful questions. In our rebuttal, we did our best to address all the questions and provide additional context. Please let us know if there is anything else we can address to help resolve any remaining concerns.

Best regards,

The Authors

Reviewer_SHd4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses long-horizon manipulation by exploring subtasks separately, composing verified solutions hierarchically, and storing conditions governing their transitions. Its memory supports checking readiness before VLA invocation, restoring entry conditions between subtasks, and selecting strategies compatible with subsequent subtasks. The VLA remains frozen. On RoboMemArena, BATON achieves 63.5% task success and 82.9% cumulative success, compared with 23.1% and 37.9% for its matched Harness VLA baseline. Evaluation also includes ablations of the three transition mechanisms and two real-world tasks. The paper attributes its gains to retaining reusable partial solutions and explicitly managing dependencies between skills.

Critical strengths
  1. The method addresses a consequential limitation of skill composition. Explicitly representing when a skill can start, whether its output supports the next skill, and whether a locally successful strategy remains compatible with future requirements is a useful contribution. The framework gives concrete operational meaning to the observation that individually successful skills need not compose reliably.

  2. The performance improvements are substantial and supported by relevant controls. BATON and Harness VLA share the frozen policy, coding-agent backbone, perception system, and initial skill memory. Table 3 retains hierarchical exploration while separately removing invocation, handoff, and lookahead mechanisms; each removal reduces performance. These are meaningful component ablations, and the real-world results provide additional evidence of practical utility.

Critical weaknesses
  1. The claimed exponential-to-linear reduction in exploration cost is not established. The (T^K) argument requires assumptions about stage success probabilities and independence that do not follow from a mean time to first success for an adaptive agent. Conversely, the (KT) claim assumes that skills and boundaries can be explored at bounded cost, without accounting for reaching their initial states, replaying prefixes, or revisiting previously successful strategies after downstream failures. No measured scaling curves, environment-step counts, token costs, or wall-clock comparisons substantiate the claim. The shared budget of 20 episodes per task is consequently difficult to interpret when one method explores whole tasks and the other explores subtasks and compositions.

  2. Some implementation details needed to reproduce the comparison remain underspecified. Figure 2 describes memory categories and read/write triggers, but concrete memory formats, agent prompts, and implementation-level retrieval and update procedures remain unclear. The Harness VLA baseline also warrants clarification: the referenced framework retains intermediate feedback and failure evidence, so the paper should specify which mechanisms are active in its implementation and distinguish them from reusable successful subtask solutions.

Questions for authors
  1. What counts as one exploration episode for an isolated subtask, a composed pair, and a complete task? How are resets, prefix execution, retries, and composition verification charged against the 20-episode budget?

  2. Are total environment steps, agent calls or tokens, and exploration time available for the matched comparison? These measurements would clarify whether the hierarchy improves resource efficiency as well as final success.

  3. How does the Harness VLA [1] baseline retain information from unsuccessful attempts in this implementation? Please distinguish failure feedback and general operating rules from reusable successful subtask solutions, since the referenced framework already includes failure memory.

[1] Zhang et al. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents. arXiv:2607.08448, 2026.

3 comments
AuthorsOpenReview ↗

Response to Reviewer SHd4 (1/2)

Thank you very much for your acknowledgement of our work and your constructive questions. Here are our response:

For W1:

We address the two concerns separately below: W1.1 on exploration complexity, and W1.2 on implementation details.

- W1.1 (exploration complexity):

We clarify that BATON reduces exploration cost from exponential to linear without contradicting the inter-subtask dependencies, supported by both a formal justification and experimental evidence.

First, subtask exploration gives BATON a linear cost, while whole-task exploration costs exponentially. T^K and TK are order-of-magnitude estimates under the idealized assumption of fixed, independent per-stage success rates, but the gap itself does not depend on this assumption. A whole-task episode succeeds only if all K subtasks succeed in that same episode, so the first success takes on the order of T^K episodes. BATON explores each of the K subtasks and K−1 boundaries once and keeps every verified element in memory, so its cost is on the order of TK.

Second, exploring elements independently does not contradict the dependencies. The handoff transition restores the entry state the next subtask expects without changing either subtask, so nothing is re-explored. The lookahead transition handles the case where an earlier execution is incompatible with what follows: BATON re-explores only that earlier subtask, at most once per boundary, and the dependency does not propagate further back. The extra cost is additive, so the total stays linear.

Third, our experiments confirm both points. Under the same budget (20 episodes per task, a 3,000-step cap, the same LLM agent), BATON reaches 63.5% TSR versus 23.1% for Harness VLA (Tab. 1). On the many-stage tasks, whole-task exploration barely reaches a first success, while BATON succeeds on many subtasks and boundaries. Measuring this gap directly is infeasible: in the worst case, whole-task exploration costs C = T^K · c, where c is the cost of one episode. A RoboMemArena task has about 6 subtasks and each episode exceeds 1,000 steps, so C surpasses any realistic budget even for small T.

- W1.2 (implementation):

We have supplemented additional implementation details in the appendix and address each point in below response to Q1, Q2 and Q3.

For W2 (implementation details):

We have added further implementation details to the appendix, including the agent prompts and the update procedures. The concrete memory format and the implementation-level retrieval are the same as in Harness VLA. For the Harness VLA baseline’s mechanisms and their distinction from reusable subtask solutions, please see our response to Q3.

For Q1 (episode definition and budget):

We clarify what counts as one episode and how each item is charged.

The 20-episode budget is counted at the task level: one episode is one pass over the task, in which each unit under exploration is attempted once (e.g., the first pass attempts every subtask once and counts as a single episode). Each episode is capped at 3,000 environment steps and 3,600 s in total, shared among the units explored, the same as one whole-task attempt of Harness VLA. Each unit starts from the benchmark’s initialization of its first subtask, so no prefix outside the unit is executed. A failed unit is written into failure memory and retried from the same initialization in the next pass. Success of a subtask or a composed pair is verified by the benchmark’s subtask completion checker, which has no additional cost.

For Q2 (resource efficiency):

We clarify that both methods share identical resource caps, and that the hierarchy improves resource efficiency, which in turn drives final success.

First, the hierarchy spends the same budget where it is needed. Both methods are capped at 3,000 environment steps per episode. On tasks where Harness VLA fails, it typically exhausts its full budget while stuck at a single stage. Subtask exploration instead directs the budget to that stage and its transitions.

Second, the resource efficiency translates directly into success. Under the same budget, subtask exploration alone, without any transition memory, already improves TSR by 4.2 points and CSR by 18.5 points over Harness VLA (Tab. 3). The full BATON further achieves improvements of 40.4 points in TSR and 45.0 points in CSR (Tab. 1).

AuthorsOpenReview ↗

Response to Reviewer SHd4 (2/2)

For Q3 (BATON vs HarnessVLA baseline):

We clarify the memory mechanisms of our Harness VLA baseline and how BATON differs from them.

Our implementation keeps all of Harness VLA’s original mechanisms active, which retain three kinds of information:

  • Failure feedback: each failed attempt is kept as negative evidence, so later attempts avoid the same mistake.
  • General operating rules: task-independent success rules and failure models, such as re-staging after an empty grasp.
  • Reusable successful solutions: recorded only when the whole task succeeds, so a subtask that succeeds within a failed episode is not kept as a solution.

BATON keeps the first two unchanged and differs in the third. Because it explores at the subtask level, every successful subtask is retained as a reusable solution, and every failure is localized to a single subtask or composition.

AuthorsOpenReview ↗

3 Days Remaining: We would like to learn your opinion and address any remaining concerns.

Dear Reviewer SHd4,

We know this is a very busy time for reviewers, but as the discussion period is closing in 3 days, we want to send a gentle follow-up. We have submitted our response to your thoughtful questions. In our rebuttal, we did our best to address all the questions and provide additional context. Please let us know if there is anything else we can address to help resolve any remaining concerns.

Best regards,

The Authors

Reviewer_FjyoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BATON extends a coding agent over a frozen VLA with hierarchical subtask exploration and transition-aware memory. It checks readiness before VLA invocation, restores entry conditions between subtasks, and selects strategies according to downstream requirements. Evaluation covers 26 RoboMemArena tasks, two real-robot tasks, and ablations of the three transition mechanisms.

Critical strengths
  1. The reported improvements are substantial, and the comparison with Harness VLA shares the underlying components.

  2. The transition ablations and real-robot experiments provide useful evidence that managing subtask boundaries improves execution.

Critical weaknesses
  1. The central claim that exploration cost falls from exponential to linear is not established. It assumes that subtasks and boundaries can be solved independently with bounded effort, despite the dependencies motivating the method. Neither a formal justification nor a scaling experiment supports this claim.

  2. The methodological advance is incremental. Hierarchical skill composition, entry-condition checks, and downstream-aware execution are established ideas. The paper does not clearly establish what its language-memory formulation contributes beyond adding these mechanisms to Harness VLA.

  3. The exploration protocol is insufficiently specified. Equal episode budgets need not imply equal interaction or inference costs, and subtask initialization and reset access are unclear. The absence of a separate hierarchical-exploration ablation further limits attribution of the gains.

Questions for authors
  1. Under what assumptions does the linear exploration-cost claim hold, and is it supported by measured costs across different task lengths?

  2. How are isolated subtasks initialized, and how are resets, environment steps, and model calls counted within the exploration budget?

  3. How does hierarchical exploration perform without any transition memory under the same total interaction budget?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Fjyo (1/2)

We sincerely thank you for the valuable questions and comments. Here are our responses:

For W1 (linear vs. exponential cost):

We clarify that BATON reduces exploration cost from exponential to linear without contradicting the inter-subtask dependencies, supported by both a formal justification and experimental evidence.

First, subtask exploration gives BATON a linear cost, while whole-task exploration costs exponentially. A whole-task episode succeeds only if all K subtasks succeed in that same episode, so the first success takes on the order of T^K episodes. BATON explores each of the K subtasks and K−1 boundaries once and keeps every verified element in memory, so its cost is on the order of TK.

Second, exploring elements independently does not contradict the dependencies. The handoff transition restores the entry state the next subtask expects without changing either subtask, so nothing is re-explored. The lookahead transition handles the case where an earlier execution is incompatible with what follows: BATON re-explores only that earlier subtask, at most once per boundary, and the dependency does not propagate further back. The extra cost is additive, so the total stays linear.

Third, our experiments confirm both points. Under the same budget (20 episodes per task, a 3,000-step cap, the same LLM agent), BATON reaches 63.5% TSR versus 23.1% for Harness VLA (Tab. 1). On the many-stage tasks, whole-task exploration barely reaches a first success, while BATON succeeds on many subtasks and boundaries. Measuring this gap directly is infeasible. In the worst case, whole-task exploration costs C = T^K · c, where c is the cost of one episode. A RoboMemArena task has about 6 subtasks and each episode exceeds 1,000 steps, so C surpasses any realistic budget even for small T.

For W2 (novelty):

Agentic robotics is becoming an increasingly important direction, and to our knowledge, we are the first to tackle its long-horizon bottleneck by breaking the task into subtasks and the transitions between them. We clarify that our novelty is identifying and resolving agentic robotic’s bottleneck, which is fundamentally different from adding mechanisms to Harness VLA.

First, BATON identifies the unique bottleneck of agentic robotics at long horizons: exploration cost. The paradigm is promising because an LLM agent can discover its own solution through exploration, without training. On long-horizon tasks, however, that exploration becomes prohibitively expensive for two reasons. Whole-task exploration cannot tell which stage caused a failure, and the subtasks depend on each other, so pieces that each succeed alone still break when chained. This bottleneck is intrinsic to the paradigm, not specific to Harness VLA.

Second, BATON addresses each of these issues directly. To identify which stage failed, BATON introduces agentic subtask exploration, which explores each subtask separately and attributes every failure to a single stage. To handle inter-subtask dependencies, BATON introduces the inter-subtask transition memories (handoff and lookahead), which record the conditions under which subtasks compose. To further improve the transition within each subtask, BATON adds the invocation transition, which hands control to the VLA only when the scene is ready. Overall, these reduce exploration cost from exponential to linear, which is why BATON performs strongly under a limited budget.

We therefore disagree that the advance is incremental. Our contribution lies in identifying the bottleneck that blocks agentic robotics from long-horizon tasks, and in proposing a solution that directly addresses each of the issues identified.

For W3 (exploration protocol):

The three points raised here are addressed in our responses to the below three questions: Q1, Q2 and Q3.

For Q1 (assumptions and task length):

Both points are addressed in W1; we summarize the conclusions here.

First, BATON’s exploration cost is linear because it decomposes the full task into subtasks and boundaries and explores each once within a bounded budget.

Second, the results across task lengths are consistent with this linear cost. RoboMemArena tasks vary from 3 to 9 subtasks. On occlusion, the longest category (about 1,400 steps per task), whole-task exploration (Harness VLA) drops to 0.0% TSR because its exponential cost exceeds the budget, while BATON, whose cost grows linearly with the number of subtasks, stays within the same budget and reaches 50.0% TSR.

AuthorsOpenReview ↗

Response to Reviewer Fjyo (2/2)

For Q2 (implementation detail):

We clarify each detail below and supplement more implementation details in the appendix. Isolated subtasks are initialized with the benchmark’s own subtask initialization settings, and all exploration runs on a single exploration seed. A reset happens only when an attempt fails: the scene returns to the same initialization and the agent explores again, now with the failed attempt written into failure memory. The budget is counted at the task level: one episode corresponds to one pass over the full task, and both methods share the same 20-episode budget. Each episode is capped at 3,000 environment steps and 3,600 s of wall time, shared across the subtasks explored in that episode, so the per-episode budget is identical to that of Harness VLA. Model calls are not separately counted.

For Q3 (additional exp on no transition memory):

We provide an additional ablation of hierarchical exploration without any transition memory, under the same budget, and have added it to Table 3.

Table 3. Ablation of BATON's transition mechanisms (TSR / CSR, %). The row marked with † is newly added in this rebuttal.

VariantTransOccCntSeqAvg
w/o all transitions †50.0 / 61.57.3 / 40.618.6 / 57.375.0 / 93.127.3 / 56.4
w/o invocation52.5 / 63.536.4 / 66.542.9 / 61.880.0 / 93.447.3 / 68.9
w/o handoff57.5 / 67.19.1 / 44.421.4 / 78.577.5 / 93.330.4 / 64.6
w/o lookahead62.5 / 70.818.2 / 47.828.6 / 83.382.5 / 93.837.7 / 68.0
BATON (full)75.0 / 83.350.0 / 78.664.3 / 83.387.5 / 93.863.5 / 82.9

Without any transition memory, the average TSR / CSR drops to 27.3 / 56.4, the lowest among all BATON variants, showing that transition memory is essential for long-horizon tasks. Compared with Harness VLA (23.1 / 37.9), hierarchical exploration alone still raises CSR substantially, as it solves individual subtasks; but without transition memory, these subtasks fail to compose, so TSR stays low.

AuthorsOpenReview ↗

3 Days Remaining: We would like to learn your opinion and address any remaining concerns.

Dear Reviewer Fjyo,

We know this is a very busy time for reviewers, but as the discussion period is closing in 3 days, we want to send a gentle follow-up. We have submitted our response to your thoughtful questions. In our rebuttal, we did our best to address all the questions and provide additional context. Please let us know if there is anything else we can address to help resolve any remaining concerns.

Best regards,

The Authors

Comments on the paper

3 comments
Area_Chair_gtYvOpenReview ↗

Please start the discussion on the rebuttals

Dear Reviewers,

Thanks a lot for your help reviewing this submission! We have received the rebuttals of the authors, and the opinions for this paper are diverse currently, so I would like to hear everyone's opinion after the authors' rebuttal to help me form my final meta-review and recommendation for this paper.

Can I ask everyone to share a little bit about their post-rebuttal feelings about the paper by replying to this message thread, and update their original review/rating of the paper accordingly at your earliest convenience? Thanks a lot in advance!

Best, your AC

Area_Chair_gtYvOpenReview ↗

Please reply to the authors's rebuttals

Dear Reviewers,

Please reply to the authors's rebuttals!

AuthorsOpenReview ↗

Thank You for Your Positive Assessment!

Dear Reviewer SHd4,

Thank you for taking the time to read our rebuttal and for confirming that it addressed your main concerns. We sincerely appreciate your positive assessment and your constructive feedback, which has helped us improve the paper. In the revised version, we will include the clarifications from the discussion: the formal justification of the linear exploration cost, the episode and budget definitions, and the comparison of memory mechanisms with Harness VLA.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.