acceptodds
Under review as a conference paper at ICLR 2027

ACTION–EFFECT CONTRACTS: BRIDGING TASK INTENT AND PHYSICAL EFFECTS IN AGENTIC ROBOTS

Abstract

Vision-language-action (VLA) models map language instructions and visual ob- servations directly to continuous robot actions. Real-world robot tasks are often long-horizon and compositional, requiring agentic robot systems to plan and ex- ecute sequences of interdependent subtasks. In mainstream designs, a high-level agent intervenes through task-flow decisions such as subtask invocation, stopping, replanning, and human assistance, while the low-level VLA determines the specific effects of task execution. This architecture leaves the agent without direct involve- ment in action execution. To increase the agent’s participation during execution, we introduce Action–Effect Contracts, a semantic Agent–VLA interface that lets the high-level agent specify the physical effect required at the current task stage. At each decision point, a frozen VLA proposes multiple short action candidates, and an effect ranker selects the candidate most likely to realize the requested effect. The ranker jointly conditions on the current image, robot state, effect request, and candidate action through candidate-dependent spatial attention. Same-state counterfactual rollouts provide candidate-level matching labels and within-group ranking supervision. We instantiate the high-level agent as a fine-tuned Qwen3-VL- 8B-Instruct model that generates online effect requests, and use a frozen π0.5 action policy for short-horizon manipulation in LIBERO. Across nine LIBERO-90 tasks, our method improves requested-effect matching in all four stages and raises task success from 42.06% to 51.27%. In four real-world manipulation tasks performed by a robot arm, success increases from 48.33% to 58.33%. These results show that high-level effect requests can effectively guide VLA action execution and improve task completion.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Hierarchical Robot Control, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a3uYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Action–Effect Contracts, a semantic interface designed to bridge high-level task intentions and low-level physical action execution in agentic robotic systems. The framework employs a fine-tuned Qwen3-VL-8B model to generate physical-effect requests, including approach, lift, transport, and place. A frozen π0.5 VLA generates multiple candidate action chunks, while an effect-conditioned ranker evaluates their compatibility with the requested physical effects. The ranker incorporates candidate-conditioned spatial attention and is trained using candidate-level classification and pairwise ranking losses, with supervision obtained from same-state counterfactual executions. The method is evaluated on nine LIBERO-90 tasks and four real-world manipulation tasks, achieving improvements over direct VLA execution and random candidate selection. Overall, the paper addresses a meaningful problem and proposes a practical framework for coordinating high-level reasoning with low-level robotic execution.

Critical strengths
  1. Meaningful problem formulation and practical Agent–VLA integration. The paper addresses an important limitation of hierarchical robotic systems, where high-level agents typically influence task-level decisions but have limited involvement in low-level action execution. The proposed Action–Effect Contracts establish an explicit interface between task intentions and physical action effects. This provides a practical mechanism for guiding action selection without modifying the pretrained VLA policy.
  2. Effective candidate-level supervision and encouraging empirical results. The same-state counterfactual supervision provides meaningful training signals by comparing the physical outcomes of different candidate actions under identical initial conditions. The method achieves consistent improvements over direct policy execution and random candidate selection in both simulation and real-world manipulation experiments. The ablation studies also support the contributions of the individual modules.
Critical weaknesses
  1. Insufficient comparisons with competitive baselines. My primary concern is the limited set of comparison methods. The main experiments compare the proposed approach only against direct execution of the frozen π0.5 policy and random selection among four candidate actions. Although these comparisons demonstrate that the proposed ranker is more effective than naive selection strategies, they are insufficient to establish its advantages over existing action-ranking, goal-conditioned scoring, or model-based planning approaches. The component ablations are useful, but they do not replace comparisons with independent and competitive alternatives. In particular, the authors should consider including: • A rule-based or finite-state effect-request generator combined with the same action ranker, to evaluate whether the high-level VLM provides benefits beyond conventional manipulation-stage estimation. • A goal-conditioned action ranker or learned value critic operating on the same frozen VLA candidates, with comparable training data and computational budgets. • A representative model-based or VLM-guided candidate-selection baseline, such as an appropriately adapted VLMPC-style approach, where a fair comparison is feasible. These comparisons would substantially strengthen the empirical justification for the proposed design.
  2. Moderate methodological novelty and limited generalization evidence. The overall framework primarily integrates existing components, including a pretrained VLM, a frozen VLA, a visual encoder, and a learned ranking network. While the explicit Action–Effect Contract is a useful interface, the classification and pairwise ranking objectives are relatively standard. The paper does not sufficiently demonstrate why this particular formulation is preferable to alternative goal-conditioned action evaluation methods. Furthermore, the experiments mainly focus on grasping and object-placement tasks. The limited task diversity makes it difficult to assess whether the four predefined effect categories generalize to more complex interactions or long-horizon manipulation scenarios. Therefore, the current evidence does not fully support broader claims about the general applicability of the framework.
Questions for authors

Q1. Comparison with stronger action-selection baselines Why are the main experiments limited to direct π0.5 execution and random candidate selection? Could the authors compare against a learned goal-conditioned action scorer or value-based candidate selector using the same frozen VLA, candidate pool, and training data? Such an experiment would help determine whether the observed improvements arise specifically from the proposed Action–Effect Contract formulation or from learned candidate ranking more generally. Q2. Necessity of the high-level VLM Since the effect vocabulary contains only four predefined manipulation stages, could the authors compare the fine-tuned Qwen3-VL agent against a rule-based or finite-state effect-request generator? This comparison is important for establishing the necessity of high-level vision-language reasoning in the proposed framework. Q3. Generalization and candidate-selection limitations How does the approach behave when none of the generated candidate actions can realize the requested effect? Could the authors report the frequency of such cases and evaluate whether the ranker can reliably distinguish feasible from infeasible candidate groups? This analysis would help clarify the practical limitations of the current action-selection mechanism.

2 comments

Thank you for recognizing the Agent–VLA connection and same-state supervision. Our strongest evidence is improved task completion through request-conditioned selection with a frozen policy and matched candidate budget.

Q1. Value beyond generic reranking. Holding images, state, actions and request-matching targets fixed, providing the request raises SR from 40.79% to 51.27% (+10.48 pp). Candidate-dependent attention adds +8.73 pp over global pooling; ranking supervision adds +7.62 pp over classification alone. These ablations connect the effect target, visual evidence and selection objective to task success.

Supplementary prompt-conditioning and external-verification controls:

MethodSuccesses / trialsSR
Request appended to VLA instruction113/27041.85%
Random115/27042.59%
Effect ranker134/27049.63%
Transferred CoVer scorer136/27050.37%
Oracle effect selector141/27052.22%

The ranker completes 21 more episodes than direct language conditioning and 19 more than Random. CoVer provides an independent verification reference using its public Bridge-trained scorer, with action/gripper conversion and history adaptation. Its result concerns that transferred component; the table does not establish statistical equivalence between scorers.

These rows share geometry requests, frozen pi05_libero, K=4, H=5, 1.5 cm thresholds and 30 episodes/task. The main study uses Qwen requests, 1 cm thresholds and 70 episodes/task. Comparisons remain within their respective protocols.

The contribution is a testable intermediate physical objective, linked to candidate evaluation through executed outcomes. A request-blind progress scorer instead targets any positive effect. A task-language value critic and VLMPC constitute different controls; the reported experiments do not establish superiority over those untested alternatives. The controlled request ablation directly supports the specific benefit of supplying the requested effect.

Q2. Role of the requester. Qwen generates requests from task language, two-round global/wrist images and robot state. A conventional, simulator-privileged geometry requester reaches 49.63% with the frozen ranker, demonstrating compatibility with different request sources.

The 10.48 pp main-study request-ablation gap establishes the value of the request input. The Qwen and geometry SR values come from different protocols, so they do not isolate the benefit of model size. Our empirical claim concerns the interface; a same-input, same-label lightweight comparison would separately test the choice of requester.

Q3. Candidate coverage and behavior without a match. For a valid request, the learned selector executes the highest-scoring candidate and updates its observation after five controls. If the pool contains no matching candidate, this relative-ranking rule still applies; scores are not used as calibrated rejection probabilities.

Requested effectReported Oracle coverageNo matching candidate
Approach35.85%64.15%
Lift25.98%74.02%
Transport79.67%20.33%
Place48.19%51.81%

The last column is 100% minus coverage on the corresponding evaluated pools. Coverage bounds selector Hit when states, requests, candidates and label rules coincide; it is not a cross-trajectory success bound.

An independent 452-pool geometry diagnostic evaluates feasible-pool detection using the maximum ranker score. AUROC is 0.681 pooled and 0.548/0.891/0.762/0.667 by stage, with 336/60/45/11 pools. These measurements characterize stage-dependent discrimination, including the near-chance approach value and small placement sample. The deployed mechanism uses relative ranking; no validated feasibility-rejection threshold is assumed.

Scope and practical value. Ours improves over compute-matched Random on all nine simulation tasks (+48/630 successes) and all four robot objects (+14/120). These results support execution-time coordination for the evaluated grasping/placement tasks and real-world adaptation; the four effects are not claimed to cover every manipulation domain.

We hope the controlled ablations, independent comparisons and explicit coverage analysis address your questions, and would be grateful if you would consider raising your score.

Thank you for your response. I maintain my original score.

Reviewer_qLNbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Action–Effect Contracts, a mechanism through which a high-level agent — a Qwen3-VL-8B fine-tuned with LoRA — issues a stage-level "effect request" rt∈{approach,lift,transport,place}r_t \in \{\text{approach}, \text{lift}, \text{transport}, \text{place}\} at every decision point within a manipulation subtask. Rather than committing to its default output, a frozen π0.5\pi_{0.5} policy generates K=4K{=}4 candidate action chunks, each spanning H=5H{=}5 control steps. A learned "effect ranker" then scores each candidate — jointly attending over DINOv2 visual patch features, the robot state, the requested effect, and the proposed action via candidate-dependent spatial attention — and the highest-scoring chunk is executed. The ranker is trained in simulation via same-state counterfactual rollouts: each candidate is replayed from an identical saved state, and geometric rules assign binary match targets as well as within-group ranking pairs. Across nine LIBERO-90 pick-and-place tasks, success rate rises from 42.06% (Default) and 43.65% (Random under the same candidate budget) to 51.27%; on four real-world plate-placement tasks, it improves from 48.33% (Default) to 58.33%.

Critical strengths
  • Honest compute-matched control. Random selection under the same candidate budget is reported throughout, isolating the true reranking gain (+7.62 pp in sim) from the best-of-4 effect alone; ablations and protocol details are unusually explicit.

  • Clean supervision with an informative negative result. Same-state counterfactual rollouts yield clean candidate-level labels, and the ablation showing that request-blind selection underperforms even direct execution is a genuinely useful observation.

Critical weaknesses
  • Thin novelty, oversold as a "semantic interface." The method is best-of-N reranking of a frozen policy with a learned outcome classifier — well established in value-guided/critic-reranking work, none of which is discussed — and the "contract" reduces to a four-way stage one-hot with a 4-entry embedding table: the VLM never describes an effect in language.

  • The "agent" is decorative. Requests are supervised by the same geometric rules that generate the labels, the agent just outputs one of four tokens at every chunk boundary, and the trivial rule-based-requester baseline (zero cost in simulation, where labels come from geometry) is never run; the agent's request accuracy is never reported either.

  • Headline evidence is circular or fragile. The headline "Hit" metric is scored by the exact annotation function the ranker is trained on; the real-world gains on 120 trials are not statistically significant (p≈0.12/0.07p\approx0.12/0.07, no CIs or seed variance anywhere); and no oracle bound shows how far 51.27% is from the ceiling set by request accuracy plus candidate coverage.

  • Limited external validity. Tasks are cherry-picked into a 10–90% difficulty band of a benchmark that fine-tuned policies essentially saturate (~98%), and all thirteen tasks are variants of pick-and-place-to-target, so the numbers are not comparable to standard LIBERO-90 results.

Questions for authors
  1. How are the K=4K{=}4 candidates sampled from a flow-matching π0.5\pi_{0.5} (temperature/noise seeds)? Candidate diversity determines reranking headroom.
  2. What are the ranker training-set sizes (groups/episodes) in sim and real? For real-world "same starting conditions" groups, how are resets performed, and what is inter-annotator agreement on effect outcomes?
  3. Which π0.5\pi_{0.5} checkpoint is used — was it fine-tuned on LIBERO-90 demonstrations, and with what recipe?
  4. What is the agent's request accuracy against the geometric ground truth, and what end-to-end SR would a rule-based requester achieve in simulation?
  5. Oracle experiment: if the ranker always selected a request-matching candidate when one exists, what SR/Hit results? Where does 51.27% sit relative to this bound?
1 comment

Thank you for recognizing the compute-matched control and same-state supervision. Our evidence links request conditioning to higher terminal success.

Contribution: an execution-grounded interface. Four testable effects connect requests to candidate-dependent visual attention and scoring. Same-state executions supervise selection while pi0.5 stays frozen. Main-study ablations quantify each component:

VariantTask SRFull-model advantage
Full model51.27%—
Without request input40.79%+10.48 pp
Global visual pooling42.54%+8.73 pp
Without ranking loss43.65%+7.62 pp
Without robot state47.46%+3.81 pp

Retaining request-matching targets and other inputs, the no-request ablation isolates a 10.48 pp benefit of request access. Our contribution couples established best-of-K selection and classification/ranking tools through physical-effect supervision.

Task success provides an independent outcome. Hit evaluates the supervised physical-effect objective on held-out transitions. Terminal success is evaluated separately through benchmark completion and, on the robot, release and stable placement.

SettingDefaultRandomOurs
Simulation265/630275/630323/630
Real robot58/12056/12070/120

Ours completes 48 more simulation episodes than Random (+7.62 pp), improving on all nine tasks. Real-robot gains over Random are 3/3/5/3 successes across the four objects, totaling +14/120 (+11.67 pp). Descriptive 95% Wilson intervals for simulation Ours/Random are [47.37,55.15]%/[39.83,47.55]%; for the robot they are [49.39,66.76]%/[37.98,55.56]%. These pooled intervals assume independent trials and describe proportions, not paired significance or seed variance.

Agent role and rule-based reference. The requester identifies the needed effect; the ranker evaluates candidates against it. Qwen uses task language, image history and robot state. With geometry requests, the frozen ranker achieves 134/270 (49.63%) versus Random's 115/270 (42.59%): +7.04 pp, demonstrating compatibility with conventional requests.

Supplementary results use geometry requests, 1.5 cm thresholds and 30 episodes/task; main results use Qwen, 1 cm and 70 episodes/task. Cross-protocol SR differences do not measure requester quality. SR and Hit do not measure request accuracy; 8B-specific benefits require an input-matched comparison.

Oracle reference and sampling headroom. With the same supplementary setup, Oracle achieves 141/270 (52.22%), compared with the learned ranker's 134/270 (49.63%): a seven-success gap. Oracle preserves Random when it already matches or no match exists; otherwise it samples a matching candidate. This local-effect reference is not a terminal-success optimum or directly compared with the main-study 51.27%.

Each proposal uses a distinct deterministic seed to initialize Gaussian flow noise, with 10 integration steps and the first five controls of each native 10-step output. All 16,533 inspected pools have four numerically distinct chunks. In a separate 452-pool diagnostic, 338 pools have no matches, 87 have four, and 27 have both types. Oracle coverage is 25.22% there, defining the fixed-pool Hit ceiling.

Training and evaluation scope. The available simulation ranker cache has 1,987 training groups/7,948 candidates from 480 episodes and 743 validation groups/2,972 candidates from 278 episodes. The 48/28 task identities are disjoint; cache states are 0–9 and supplementary test states 30–44. We use released OpenPI pi05_libero. Its published recipe specifies physical-intelligence/libero, pi05_base initialization, batch 256, 30,000 steps, peak LR 5e-5, 10,000 warmup steps and EMA 0.999; this identifies the release rather than a demonstration-level LIBERO-90 audit.

Real-world training uses executed candidate groups and human outcome labels. The reported 30 trials/object are evaluation counts. Training totals, reset tolerances and annotator agreement require their records; no values are inferred here.

The nine-task 10–90% difficulty subset tests intervention under matched budgets. Results support within-subset grasping/placement gains and adapted physical deployment, not full-suite or long-horizon performance.

We hope these clarifications and controlled results address your concerns, and would appreciate your consideration of a higher score.

Reviewer_qthROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Action–Effect Contracts to guide action selection from a frozen VLA with high-level effect requests from a planning agent. A fine-tuned Qwen3-VL-8B model requests one of four effects. In the LIBERO setup, a learned ranker selects among four short action chunks from π0.5. The ranker uses effect-matching and pairwise ranking supervision from candidate executions, including simulator rollouts from the same starting state. Evaluation covers nine selected LIBERO-90 tasks and four real-world placement tasks, with additional real-world training data. The paper reports overall improved task success rates in both simulation and on the real robot.

Critical strengths
  1. The paper proposes to execute candidates from the same simulator state that provides direct supervision of their physical effects. This is a reasonable way to compare candidate outcomes while controlling the starting conditions.

  2. The paper reports higher task success than both direct policy execution and random candidate selection on all nine simulation tasks and all four real-world tasks.

Critical weaknesses
  1. The paper lacks a baseline that directly adds the agent-generated effect request to the VLA instruction. The current comparisons do not establish the need for an additional trained ranker or its advantage over direct instruction conditioning. Please compare the two approaches using the same request generator, frozen VLA, and update frequency.

  2. The paper lacks key literature for comparisons, which may undermine its novelty. CoVer-VLA [1], RoboMonkey [2], and V-GPS [3] already use learned evaluators to guide action selection. Comparisons with Default, Random, and internal ablations do not show whether the proposed effect requests and supervision from executed outcomes improve on these approaches. A controlled comparison with CoVer-VLA's scoring component would be particularly informative.

  3. The practical value of frequent execution-time intervention, i.e., candidate action pool generation and selection, is difficult to assess without measuring its inference cost. In simulation, each five-step decision round involves a Qwen3-VL-8B request, four VLA proposals, and effect ranking. No end-to-end decision latency or episode duration is reported, and matching control-step budgets does not establish the tradeoff between success and elapsed time.

References

[1] Jacky Kwok et al. Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignment. ECCV, 2026.

[2] Jacky Kwok et al. RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models. CoRL, 2025.

[3] Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance. CoRL, 2024.

Questions for authors
  1. Why does Table 5 mark a lift request as unmatched when the action also achieves transport? How often does this choice affect training labels and Hit rates?

  2. Table 2 uses each method's own trajectories. Do the Hit gains persist on the same held-out states, requests, and candidate pools?

  3. With the ranker fixed, does Qwen3-VL-8B improve task success rates over a lightweight stage predictor trained on the same inputs and labels? Is using a 8B model necessary?

2 comments

Thank you for highlighting same-state supervision and the consistent task-level gains. Our central contribution is an execution-grounded interface that makes the current requested effect actionable for a frozen VLA.

Why request-conditioned selection matters. In the main study, Ours achieves 51.27% success versus 43.65% for compute-matched Random: +7.62 percentage points (pp), or 48 additional successes in 630 trials, with gains on all nine tasks. Removing the request input while retaining the image, state, actions and request-matching training targets yields 40.79%. This 10.48 pp ablation gap directly supports conditioning candidate evaluation on the current request.

Direct instruction conditioning and independent verification. The supplementary comparison fixes the requester, frozen pi05_libero checkpoint, K=4, H=5 and request-update frequency:

MethodSuccesses / trialsSR
Request appended to VLA instruction113/27041.85%
Random115/27042.59%
Effect ranker134/27049.63%
Transferred CoVer scorer136/27050.37%
Oracle effect selector141/27052.22%

The ranker completes 21 more episodes than direct language conditioning (+7.78 pp) and 19 more than Random (+7.04 pp). Direct conditioning appends the effect and samples uniformly from four proposals. This comparison tests where the same request enters execution.

CoVer, RoboMonkey and V-GPS establish the importance of learned verification. Our contribution specifies a request-dependent physical target and grounds candidate ordering in executed outcomes. The CoVer row evaluates its public Bridge-trained scoring component with action/gripper and history adaptation; its 50.37% and our 49.63% are descriptive results, not a statistical-equivalence claim. Request-specific value is established by the controlled ablation above.

These supplementary rows use geometry requests, 1.5 cm thresholds and 30 episodes/task. Main-study results use Qwen requests, 1 cm thresholds and 70 episodes/task. Gains are calculated within each protocol.

Execution cost. A separate geometry-request timing study measures median proposal/scoring times of 348.91/9.16 ms. Median decision-loop time, including simulation execution, is 442.47 ms for the ranker and 430.71 ms for Random; episode medians are 34.68 and 33.72 s. These measurements quantify selection cost at a fixed four-proposal budget. They exclude Qwen inference and therefore describe the geometry-request loop, not full Agent latency or a main-study success-per-second comparison.

Simultaneous effects. The deterministic priority place > transport > lift > approach assigns one label per transition. A transition labeled transport is unmatched to a lift request even if height increases: Hit measures exact requested-category agreement. In an independent 452-pool diagnostic (1,808 candidates), 111 candidates satisfy multiple conditions; 95/156 lift-satisfying candidates receive transport/place labels. Among 411 candidates satisfying the requested independent condition, 20 receive another label; this removes all exact matches in five pools. These are diagnostic counts, not historical training-overlap estimates.

Identical-pool evaluation. On those 452 geometry-request pools, expected Random Hit is 21.63%, ranker Hit 21.90%, and Oracle coverage 25.22%; the ranker difference is +0.27 pp, with a trajectory-cluster 95% interval of [-0.57, +1.21] pp. There are 27 mixed-match pools where selection can change binary Hit. This separates fixed-state ordering from online trajectories. The main-study online gains of 9.30/8.11/11.94/8.67 pp across the four stages describe each method's visited states; the independent task-success comparison establishes the closed-loop benefit.

Requester choice. Qwen3-VL-8B maps task language, two-round global/wrist images and robot states to requests. Geometry requests also operate with the same interface, as the 49.63% supplementary result shows. The supported claim concerns request-conditioned execution and requester interchangeability. An input- and label-matched lightweight comparison is required to attribute benefits specifically to 8B capacity; the reported SR and Hit do not measure request-classification accuracy.

Thank authors for the detailed response. I'd keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.