acceptodds
Withdrawn from ICLR 2027

Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action models, one-shot imitation, learning from human video, robot manipulation, flow matching, task specification, cross-embodiment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ebuPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PEWAM, a World-Action Model based on object effect representations. The method converts demonstrations into an Effect Program—comprising object keypoint trajectories, contact points, and final states—to decouple task objectives from embodiment-specific execution actions. Using multi-stream Flow Matching, the model jointly learns object state transitions and robot actions, enabling program editing and closed-loop replanning. Evaluations across LIBERO, Meta-World, RLBench, diverse robotic arms, and real-world setups demonstrate its capabilities in single-demonstration task execution, cross-embodiment transfer, and disturbance recovery.

Critical strengths
  1. The task representation is innovative. Decoupling object manipulation goals from robot actions via the Effect Program enables direct editing of target positions and grasp methods, providing both flexibility and interpretability.

  2. The lightweight model demonstrates practical potential. With only 71.5M parameters, PEWAM achieves high success rates on real robots and exhibits closed-loop disturbance recovery, highlighting its practical value.

Critical weaknesses
  1. Cross-embodiment generalization is limited. While the Effect Program is decoupled from the robot embodiment, cross-arm transfer still relies on kinematic mapping and unified control interfaces, failing to demonstrate generalization across structurally distinct robots.

  2. Experimental validation is insufficient, and the core design's necessity is questionable. The paper lacks systematic evaluations on public benchmarks using standard protocols, and baseline comparisons are inadequate. Furthermore, the full Effect Program underperforms compared to using only final goal states in LIBERO-Goal experiments, failing to justify the necessity and consistent advantage of this complex task representation.

Questions for authors
  1. Current cross-arm transfer relies on Panda joint space mapping and unified operational space control. Could you provide direct execution experiments on unseen robots with significantly different kinematics or action spaces, and compare them with existing adaptation methods? This would verify that the generalization stems from the model itself rather than rule-based kinematic adaptations.

  2. Why are results not reported under the full standard evaluation protocols of public benchmarks like LIBERO and RLBench? Please provide the performance and statistical results for the complete task sets.

  3. Could you compare the method against recent competitive Diffusion Policy, VLA/WAM, and object-centric manipulation approaches under unified training data and evaluation protocols?

  4. Why does the full Effect Program underperform compared to using only final goal states on LIBERO-Goal? Please provide rigorous ablations demonstrating that object trajectories and contact points offer stable and irreplaceable benefits.

Reviewer_XiB7OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PEWAM, a 50M-parameter joint flow-matching policy that programs manipulation tasks from a single demonstration by keeping the physical effect on objects—a 16-step object pose trajectory window E*, terminal goal state G*, and 3D surface contact track c(s)—while discarding the demonstrator's robot motion. Trained with random stream clamping and a paired multi-solution grasp corpus, PEWAM executes closed-loop rollouts via an effect-progress pointer and engagement gate, and transfers across robot arms by mapping joint angles into the training Panda frame via IK_Panda(FK(q)).

Critical strengths

(1) Intuitive 3D contact-point interface ("Points, Not Weights") and closed-loop disturbance recovery: Representing manipulation intent via 3D object poses and explicit 3D surface contact points supports direct spatial shifts and grasp-angle edits, cleanly avoids the sign-flipping rotation errors of 9-D affine weights (Figure 4b), and enables closed-loop recovery after a 5 cm mid-task push (Table 3).

(2) Broad evaluation across five simulation suites, four robot arms, and real-world human videos: Compiling 3D effect programs from single human RGB-D videos achieves 36/40 real-Franka successes versus 22/40 for a goal-image baseline (Table 4), and the appendices transparently report 28 pre-registered checks, negative results, and multi-seed runs.

Critical weaknesses

(1) Full effect trajectories underperform terminal-state-only conditioning on LIBERO and require incompatible coordinate switches across task families: On the primary LIBERO benchmarks, conditioning on the full object trajectory (E*, G*) consistently underperforms clamping only the terminal goal G*: terminal state only beats effect program by 49/90 vs. 40/90 on held-out LIBERO-Goal (Table 7, 3-seed: 49, 50, 32 vs. 40, 38, 33) and by 89/90 vs. 67/90 on LIBERO-Object (Table 16, 2-seed: 89, 84 vs. 67, 65), whereas full trajectories help on articulated RoboCasa tasks (69/90 vs. 50/90, Table 20). Moreover, the default absolute-coordinate program achieves 0% on articulated drawer tasks in LIBERO-Goal split 2 (Table 15) and Meta-World ML10 (drawer-open and lever-pull, Table 17); switching to relative displacements (R5) recovers ML10 drawer-open (53.3% own / 33.3% transferred, 3-seed means 61.1% / 47.8%) but collapses held-out LIBERO-Goal from 40/90 down to 1/90 (Table 5).

(2) Arithmetic errors in Tables 2 and 8 that reverse the multi-seed cross-arm ablation conclusion: In Tables 2 (p. 8) and 8 (p. 18), the three training seeds for the full recipe both (PEWAM) achieve four-arm bowl sums of 89, 74, 72 and bottle sums of 43, 27, 17, whose arithmetic means are 78.33 (bowl) and 29.00 (bottle)—not 83.3 and 27.7 as reported. Correcting 83.3 to 78.33 drops both (PEWAM) (78.33 bowl / 29.00 bottle, total 107.33) below both contact only (82.0 bowl / 48.5 bottle, total 130.5) and neither (78.5 bowl / 41.0 bottle, total 119.5), overturning the claim that the combined recipe achieves the highest multi-seed cross-arm mean and showing that adding the paired corpus degrades multi-seed bottle transfer from 48.5 to 29.0.

(3) Privileged simulator keypoints in main benchmarks and reset-state assumptions in the RGB-D perception loop: Outside Section 4.4, all simulation results in Tables 1–3 and 5–20 read noise-free 3D object keypoints directly from simulator state, creating an input asymmetry when comparing against vision-only baselines such as UWM and Zero-WAM in Table 1. In Table 13 (Appendix F, p. 22), segmenting live keypoints from rendered RGB-D while keeping a state-derived program (P1) drops bowl+bottle success by 35.9% (from 39/60 in P0 to 25/60); although extracting both program and live keypoints from RGB-D (P2) reaches 40/60 on rigid pick-and-place, Line 1108 notes that the tracker prompts SAM 2 with reset-time simulator centers projected into the image and freezes each entity's 3D box size and orientation to their reset values throughout the episode.

Questions for authors
  1. Why does clamping only the terminal state G* outperform the full effect trajectory E* on both LIBERO-Goal (49/90 vs. 40/90, Table 7) and LIBERO-Object (89/90 vs. 67/90, Table 16), and how can a single unified coordinate parameterization handle both LIBERO placement and Meta-World articulated tasks without collapsing one or the other (Tables 5 and 17)?

  2. Could you correct the 3-seed mean sums for both (PEWAM) in Tables 2 and 8 from 83.3 / 27.7 to 78.33 / 29.00 and explain why adding the paired multi-solution corpus on top of contact only lowers the multi-seed four-arm mean from 82.0 / 48.5 to 78.33 / 29.00?

  3. What is PEWAM's success rate on the three held-out LIBERO-Goal tasks in Table 1 when using fully visual RGB-D tracking (P2) without reset-time simulator center prompts or frozen object orientations, including on the rotational turn-the-knob task?

Reviewer_LL2YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PEWAM represents a demonstration as an “effect program” with three parts: object keypoint trajectories, grasp contact points, and a final object configuration. A world-action model takes this program, generates robot actions, and replans from the current scene. The paper shows that editing the program changes placement and grasp, that the method transfers across four robot arms, and that it recovers when objects are disturbed. It completes 40/90 held-out LIBERO-Goal episodes and 36/40 real-robot trials specified by human demonstrations.

Critical strengths
  1. The idea of separating how the object should move from how the demonstrator’s arm moved makes sense to transfer demonstrations between robots, and from humans to robots.
  2. The editing experiments are interesting. Moving the final state changes where the object gets placed, and rotating the contact points changes the grasp. This shows that the model actually responds to specific parts of the program, rather than just recognizing the task and completing it.
Critical weaknesses
  1. The difference from existing point-track methods isn’t well supported. ATM appears in Table 1, but its zero success doesn’t show that it fails because its tracks include the actor’s motion. Both methods use point trajectories to guide execution, so a controlled comparison is needed here. Train on the same data with actor-inclusive tracks versus object-only tracks, then add contacts and the final state one at a time. That would make the contribution much clearer.
  2. In some cases, like held-out LIBERO-Goal and held-out objects, the full program performs worse than the final-state goal. The paper doesn’t really explain when the trajectory helps and when it just over-constrains the policy. Final-state-only results on the real robot and under disturbances would help.
  3. The paper didn't test a simpler policy conditioned on exactly the same program. Without that baseline, it’s hard to tell how much of the gain comes from the representation and how much from the four-stream model.
  4. The disturbance experiments mainly compare against open-loop replay. Any closed-loop policy should beat replay after an object gets pushed, since replay can’t react. A closed-loop baseline conditioned on a goal or on object tracks would be more convincing.
  5. The writing is very AI. A polished proofreading by human is needed.
Questions for authors
  1. why in some cases full program helps while in the others it hurts?
  2. How does the effect representation compare with ATM-style point tracks under the same training conditions?
  3. Is the four-stream world-action model necessary? How would a standard action policy conditioned on the same effect program perform?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.