acceptodds
Under review as a conference paper at ICLR 2027

Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models

Abstract

Vision-language-action (VLA) models are effective robot action executors, but they remain limited on long-horizon tasks due to the dual burden of extended closed-loop planning and diverse physical operations. We therefore propose VLAs-as-Tools, a strategy that distributes this burden across a high-level vision language model (VLM) agent for temporal reasoning and a family of specialized VLA tools for diverse local physical operations. The VLM handles scene analysis, global planning, and recovery, while each VLA tool executes a bounded subtask. To tightly couple agent planning with VLA tool execution in long-horizon tasks, we introduce a VLA tool-family interface that exposes explicit tool selection and in-execution progress feedback, enabling efficient event-triggered agent replanning without continuous agent polling. To obtain diverse specialized VLA tools that faithfully follow agent invocations, we further propose Tool-Aligned Post-Training (TAPT), which constructs invocation-aligned training units for instruction following and adopts tool-family residual adapters for efficient tool specialization. Across five simulated benchmarks, VLAs-as-Tools improves the success rate of pi_0.5 by 8.9 percentage points on average and enhances invocation fidelity by 15.0 points in Non-biased Rate. On three real-world Franka tasks, it improves success rate from 16.7% to 58.3% under matched data and compute. Code will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Agent, Vision Language Action Model.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sFncOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VLAs-as-Tools, a hierarchical system for long-horizon manipulation in which a high-level VLM agent (e.g., Gemini 3 Flash) decomposes tasks and handles replanning, while a VLA executes bounded subtasks as "tools." Two components are introduced: (1) a VLA tool-family interface, where each call is a (tool-family label, scene-grounded instruction) pair (e.g., grasp('bowl')), and the VLA returns a scalar per-call progress signal from an auxiliary head, which a threshold-based monitor uses to trigger replanning; and (2) Tool-Aligned Post-Training (TAPT), which segments demonstrations into invocation-level clips using an automatic VLM-assisted labeling pipeline (11 tool families), routes each family through its own LoRA adapter on a shared backbone, adds a progress-regression loss, and optionally runs subtask-level GRPO with simulator-state completion predicates. The paper evaluates OpenVLA, OpenVLA-OFT, and π0.5 on LIBERO-Long, an 8-task single-arm subset of RoboTwin, CALVIN, RoboMemArena, RoboLab-120, and a self-built counterfactual suite (LIBERO-IF), as well as three real-world Franka tabletop tasks (20 trials each). Reported gains include +3.6 to +5.2 points on LIBERO-Long, up to +35.5 points on RoboTwin-SA, +15.0 points in Non-biased Rate on LIBERO-IF, and an increase from 16.7% to 58.3% in average real-world success over a fine-tuned π0.5.

Critical strengths
  1. The evaluation is broad and covers multiple backbones. The method is tested on three VLA backbones and six simulated benchmarks, plus real-robot experiments, and it improves consistently over the corresponding single-policy baselines. The appendix also contains useful practical results: adding a tool family with a new adapter preserves earlier skills (97.6% retention vs. 86.4% for sequential fine-tuning, Table 8), and progress-triggered replanning cuts VLM calls by more than two orders of magnitude compared with periodic VLM monitoring (Table 14). This breadth is the main argument for acceptance, because it shows the recipe is not tied to a single model or benchmark.

  2. The invocation-fidelity study addresses a real failure mode. LIBERO-IF (Table 3) directly measures whether a VLA follows the invoked command rather than reproducing training-task behavior. The component ablation shows that each part of TAPT, especially the tool-family adapters (40.6 → 54.0 Faithful Rate for OpenVLA-OFT), improves instruction-faithful execution. This is the most targeted evidence in the paper, and it supports the claim that the executor has to be trained to be called reliably.

Critical weaknesses
  1. The paper combines known components without a focused scientific question. Hierarchical VLM planning over learned skills (SayCan, Inner Monologue, RT-H, Hi Robot, π0.5's high-level subtask prediction), per-skill LoRA experts (TAIL, AtomicVLA), VLM-orchestrated policy primitives (RoboClaw), and subtask-level progress or RL signals all have close precedents. The formalization in Sec. 3 is a generic description of hierarchical control and yields no testable insight. The paper's own ablations also weaken its more distinctive parts: under TAPT, the interface adds only about 1 point on LIBERO-Long (Table 4), a coarse taxonomy performs almost as well as the proposed one (Table 7), and the choice of VLM planner hardly matters (Table 13). The paper reports system-level gains, but the reader learns little that transfers beyond this particular pipeline.
  2. The evidence does not isolate the contribution against fair or strong baselines. The only hierarchical baseline sends subtask calls to a VLA trained on full-task instructions, so its drop in performance is expected. The paper never tests a planner combined with a single VLA trained with standard SFT on the same segmented data, nor existing hierarchical VLA systems such as π0.5's native high-level inference or Hi Robot. TAPT also receives extra DROID-split mid-training that the baselines do not appear to receive, the largest gain comes from RoboTwin-SA, an 8-task subset the authors selected themselves, and no variance is reported. The three real-world tasks are simple cube/bowl tasks, and the largest gain ("Conditioned") comes from observation aliasing that any stateful planner resolves by design. The real-robot results therefore show that hierarchy helps a memoryless π0.5, not that TAPT or the progress interface does.
Questions for authors

What is the success rate of a "planner + single VLA SFT on the same invocation-segmented data" baseline, with no tool-family adapters, no progress head, and no DROID-split post-training, using VLM-based or timeout-based subtask termination? This is the most direct test of whether the TAPT-specific components matter. A related question: what happens if the same baseline is run on the real robot?

1 comment

Thank you again for your thoughtful feedback, and if you have a chance to review our rebuttal, we’d be happy to clarify any remaining questions.

Reviewer_uPGLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VLAs-as-Tools, a framework in which a high-level VLM agent invokes specialized VLA executors for bounded manipulation subtasks. Tool-Aligned Post-Training combines invocation-aligned training units with tool-family residual adapters, while a learned progress signal supports event-triggered replanning. Evaluation covers five simulation benchmarks, counterfactual instruction following, component ablations, and three real-world tasks. Under matched real-world demonstrations, training budgets, and initial configurations, aggregate success increases from 16.7% to 58.3%. The main contribution is the combination of an explicit agent–executor interface and training intended to make local execution more faithful to that interface.

Critical strengths
  1. A coherent approach to aligning training with hierarchical deployment. Segmenting demonstrations into bounded invocations, specializing execution by tool family, and exposing progress feedback address concrete difficulties in composing VLA behaviors. The counterfactual instruction-following experiment is particularly useful because it evaluates whether the executor follows the requested local goal rather than merely reporting full-task success.

  2. Broad empirical coverage with encouraging physical results. The paper examines multiple backbones, both imitation-learning and reinforcement-learning settings, and several benchmarks. The real-world comparison uses the same task demonstrations and initial configurations for both methods, and the reported improvement is substantial.

Critical weaknesses
  1. The progress signal’s reliability and independent contribution remain only partially characterized. Appendix A explicitly describes its supervision as a normalized temporal proxy rather than a calibrated estimate of semantic completion. The paper does not quantify false completion decisions, missed stagnation, or detection delay, though these errors directly determine whether the agent advances or replans.

  2. Table 4 provides useful component ablations, and Table 14 establishes system-level recovery and efficiency benefits. However, neither isolates the learned progress signal against alternative monitoring strategies with the trained executor held fixed. Given prior hierarchical control and skill-specialization approaches such as Hi Robot [1] and AtomicVLA [2], which the submission discusses in §2, this comparison would help identify the specific benefit of the proposed feedback interface.

[1] Shi et al. Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models. ICML 2025.

[2] Zhang et al. AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robots. CVPR 2026.

Questions for authors
  1. For the simulation results in Table 1, do the standalone and direct-planner baselines receive comparable DROID data exposure and training budgets? With the TAPT executor and planner fixed, how does progress-triggered execution compare with fixed-duration or timeout-based execution?

  2. Are statistics available for false completion, missed stagnation, and detection delay?

  3. Appendix F requires both stacks to be completed in the Conditioned task, but also describes success as determined from the final physical state. How are the intermediate stacking and restoration stages verified?

Reviewer_623kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes the VLAs-as-Tools hierarchical framework, which treats the VLA model as a callable physical tool to be combined with the high-level VLM for planning, and enhances the compliance of instructions through TAPT alignment training, thereby improving the performance of the robot in long-term sequence manipulation tasks. The effectiveness has been verified through tests on multiple benchmarks.

Critical strengths
  1. This work addresses the actual needs of complex robotic operation tasks, and overcomes the inherent shortcomings of the VLA model in terms of insufficient planning reasoning and fault recovery capabilities under long-term sequence tasks.
  2. The VLAs-as-Tools solution is proposed, which to a certain extent compensates for and balances the two deficiencies of the end-to-end VLA, namely strong physical execution but weak long-term reasoning, and the VLM-driven intelligent agent planning ability but insufficient underlying execution reliability.
Critical weaknesses
  1. Using VLA as a tool and having it scheduled by the high-level intelligent agent is the mainstream approach for agentic robots. There are already representative works such as HarnessVLA.
  2. Although Franka real machine experiments were conducted, they were limited to single-arm tabletop object manipulation and did not demonstrate the advantages of the proposed method.
  3. Dependent on training data, the zero-shot capability for unseen primitives is weak: When faced with robot action primitives that did not appear in the training, this method cannot perform zero-shot execution.
Questions for authors

See the above weaknesses.

5 comments
AuthorsOpenReview ↗

New Real-World Multi-Stage Bimanual Manipulation Experiments.

Takeaway: On a new Cobot Magic bimanual platform across three 8→16→24-call multi-stage tasks (60 paired trials per method), VLAs-as-Tools raises average full-task success from 20.0% (standalone π0.5\pi_{0.5}) and 38.3% (same planner + shared IA-SFT) to 73.3% under strictly matched downstream demonstrations and budgets (Table R3).

Experimental setup and bimanual TAPT pipeline. Complementing our single-arm Franka Research 3 evaluation (Table 5, raising average success from 16.7% to 58.3%), we evaluated π0.5\pi_{0.5} on a second physical embodiment (Cobot Magic) across 3 progressive long-horizon bimanual tasks requiring cross-workspace relays, precision stacking, and articulated drawer manipulation. Because DROID-split in §5.1 covers only single-arm manipulation, we instantiate TAPT's intermediate IA Post-train stage on the bimanual embodiment—playing the exact role of DROID-split on single-arm robots—with 200K steps on RoboMIND 2.0, segmented into invocation-aligned clips by combining our automatic labeling pipeline (Appendix B, Table 6) with RoboMIND 2.0's native subtask annotations. During downstream adaptation, all compared methods use the exact same 50 physical demonstrations per task, a strictly matched 50K-step downstream budget, and 20 paired randomized initial configurations per task (following Table 6, each pick-and-place stage comprises 4 calls: reach, grasp, move, release):

  • Task 1 — Bimanual Sort (8 nominal calls, 2 stages): Left arm picks up the cylinder and places it into the left bowl (4 calls); right arm picks up the small cube and places it into the right bowl (4 calls).
  • Task 2 — Relay Stack (16 nominal calls, 4 stages): Both the small cube and cylinder start in the left-arm-only workspace and must be relayed to the right-arm-only target zone for assembly—(i) left arm transfers the small cube to the center handover zone (4 calls) → (ii) right arm picks up the small cube from the center zone and places it at the right target spot as a base (4 calls) → (iii) left arm transfers the cylinder to the center handover zone (4 calls) → (iv) right arm picks up the cylinder from the center zone and stacks it precisely atop the small cube (4 calls).
  • Task 3 — Drawer Retrieve & Stow (24 nominal calls, 6 stages across 6 tool families reach, grasp, move, release, pull, push): (i) Left arm grasps the handle and pulls the drawer open (reach, grasp, pull, release, 4 calls) → (ii) left arm retrieves the cube from inside the drawer to the center handover zone (4 calls) → (iii) right arm transfers the tabletop cylinder from the right zone to the center handover zone (4 calls) → (iv) right arm picks up the cube from the center zone and places it into the right bowl (4 calls) → (v) left arm picks up the cylinder from the center zone and stows it inside the drawer (4 calls) → (vi) left arm grasps the handle and pushes the drawer closed (reach, grasp, push, release, 4 calls).

To verify that physical gains genuinely stem from TAPT and progress feedback rather than merely adding a stateful planner over segmented clips, we compare against both Standalone Fine-Tuned π0.5\pi_{0.5} and Same Stateful VLM Planner + Shared IA-SFT π0.5\pi_{0.5} (using the exact same planner and segmented downstream bimanual data, without tool-family residual adapters or a learned progress head, advancing subtasks via periodic VLM visual checks every 5s).

Table R3: New Real-World Bimanual Results on Cobot Magic (π0.5\pi_{0.5}, 20 Trials/Task). Matched downstream demos & budget; task cells report Full-Task SR / Step Progress.

MethodPlannerTAPT + Prog.T1: Sort (8 calls)T2: Relay Stack (16 calls)T3: Drawer Stow (24 calls)Avg. Prog. (%) ↑Avg. SR (%) ↑
Standalone Fine-Tuned π0.5\pi_{0.5}6/20 (30.0%) / 93/160 (58.1%)3/20 (15.0%) / 79/320 (24.7%)3/20 (15.0%) / 174/480 (36.3%)39.712/60 (20.0%)
Same Planner + Shared IA-SFT π0.5\pi_{0.5}✓11/20 (55.0%) / 118/160 (73.8%)7/20 (35.0%) / 189/320 (59.1%)5/20 (25.0%) / 211/480 (44.0%)58.923/60 (38.3%)
VLAs-as-Tools (Same Planner + TAPT π0.5\pi_{0.5}, Ours)✓✓17/20 (85.0%) / 144/160 (90.0%)15/20 (75.0%) / 275/320 (85.9%)12/20 (60.0%) / 327/480 (68.1%)81.444/60 (73.3%)

Thank you again for your thoughtful feedback, and if you have a chance to review our rebuttal, we’d be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Summary of the rebuttal.

Dear Reviewer 623k,

Thank you again for your constructive feedback. As the discussion period draws to a close, we wanted to briefly recap how our rebuttal and new experiments address your three concerns:

  • Distinction from HarnessVLA: Rather than orchestrating frozen VLAs, we post-train the executor for reliable tool calls, lifting fixed-plan counterfactual fidelity on LIBERO-IF from 24.8% to 54.8%.
  • Real-world evaluation: On three new bimanual tasks (8 to 24 tool calls) under matched demos and budgets, our method reached 73.3% full-task success versus 38.3% for a shared executor with the same planner.
  • Unseen primitives: Given demonstrations of new primitives, our adapter expansion learns them without replaying old data while retaining 97.6% of prior skill performance.

We completely understand that a busy schedule may not have left time to join the discussion, and we sincerely hope you will take our detailed rebuttal and additional experiments into account during the subsequent review stage. We have put tremendous effort into this work, and with the help of your review, we believe the paper has improved substantially. We would be truly grateful if you would consider raising your rating.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for Submission 12832 (VLAs-as-Tools)

Summary for Submission 12832 (VLAs-as-Tools)

We thank the reviewers for their feedback. During the rebuttal, we provided point-by-point responses and added controlled simulation studies and bimanual robot experiments to address the main concerns about our contribution, experimental fairness, progress feedback, and physical applicability.

Our central contribution is to train pretrained VLAs into physical tools that an agent can reliably invoke and monitor. This addresses a concrete execution problem—VLAs trained on full tasks can follow familiar trajectory patterns even when an agent requests a different local action. Across five simulation benchmarks, our method improves π0.5’s average success by 8.9 percentage points, with further gains on both single-arm and bimanual robots.

  • Established the need for executor alignment. Directly adding a planner to a full-task VLA reduced LIBERO-Long success from 92.4% to 80.4%. With the high-level plan fixed on LIBERO-IF, TAPT raised faithful execution from 24.8% to 54.8%, demonstrating improved adherence to the agent’s instructions.

  • Added controlled shared-executor baselines. With the planner, progress interface, and DROID exposure held fixed, tool-family adapters increased LIBERO-Long success from 96.4% to 97.2%. The corresponding executor-only comparison improved from 91.2% to 96.0%, supporting the contribution of tool specialization under matched training conditions.

  • Isolated the benefit of progress feedback. With the RoboTwin executor fixed, progress-triggered recovery improved success from 39.4% to 57.5% over timeout-based recovery and raised stall-alarm precision from 31.7% to 64.9%. It also required 95.8% fewer VLM calls than periodic visual checks. We quantified false completions and discussed the remaining limitations of progress estimation.

  • Extended physical validation to longer bimanual tasks. Across three new tasks requiring 8–24 tool calls, our method achieved 73.3% success, compared with 38.3% for a shared executor using the same stateful planner and matched downstream training. These results demonstrate gains with planner memory held fixed.

  • Clarified data efficiency and tool-library expansion. With ten demonstrations per LIBERO-Long task, our method improved OpenVLA-OFT success by 11.5 percentage points. After learning new pull and push primitives without replay, prior-skill success remained 97.6%, compared with 86.4% under sequential fine-tuning.

We sincerely thank Reviewer uPGL for their careful review and constructive feedback. We are pleased that our responses and additional experiments have addressed their concerns and strengthened the paper.

We have also provided comprehensive, point-by-point responses to Reviewers 623k and sFnc, supported by additional experiments and clarifications. We hope this evidence resolves their concerns as well. As the discussion period concludes, we thank all reviewers and the Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.