acceptodds
Under review as a conference paper at ICLR 2027

PlayWAM: Learning Dynamics and Value in World Action Models through Controlled Counterfactual Play

Abstract

World Action Models (WAMs) serve as both dynamics models and action generators. However, this dual role creates conflicting data requirements: dynamics learning benefits from diverse experiences beyond successful execution, whereas action generation requires supervision selected for task success. Our key insight is that simulation enables controlled alternative futures from shared task states, revealing both physical consequences and their task utility. Motivated by this, we introduce PlayWAM, which combines branch-specific learning with controlled counterfactual play—a simulation-native pipeline for generating comparable alternative futures from shared states. We use state cloning and parallel rollouts with controlled interventions to capture nominal, perturbed, failing, and recovering behaviors. PlayWAM learns dynamics from all recorded branches but imitates only task-successful segments; other observed actions condition future prediction rather than serve as imitation targets. A lightweight value head ranks same-state branches and scores parallel future–action candidates, enabling single-round selection without outer-loop refinement. We evaluate on PlayBench, RoboTwin-Play, LIBERO, and eight real-world tasks. Project page: https://stellar-wisp-d1ecc3.netlify.app/

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Robot Learning, Simulation, Counterfactual Learning, Dynamics Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6neLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PlayWAM uses simulation to collect alternative continuations from shared states, including failures and successful recoveries. It learns dynamics from all recorded transitions and imitates only successful, executable action segments. A value head then selects among jointly generated future–action candidates in one round of sampling. The results show substantial robustness gains on PlayBench and RoboTwin-Play, smaller gains from selection on LIBERO, and better performance on eight real-world tasks when structured simulated data replace successful simulated data alongside fixed real demonstrations. I find the approach promising, but several details need clarification to establish why it works and how to reproduce the supervision.

Critical strengths
  1. The distinction between physical evidence and useful behavior is well motivated. Retaining a failed attempt for dynamics learning while imitating its successful correction is a sensible way to use recovery data (Section 5; Eqs. 25–28).
  2. The evaluation covers two simulators and physical robots, with meaningful controls within each model family.
  3. Scoring latent candidates avoids decoding videos and repeatedly refining proposals. Reporting both four and eight candidates makes the success/latency tradeoff visible (Table 3). The timing advantage over WAV is useful as a comparison of the evaluated systems.
Critical weaknesses
  1. The experiments do not yet clearly identify what is gained from learning failure dynamics. Full training adds both dynamics and value supervision, and the all-action ablation changes what the policy imitates. The random-selection result supports an improvement in generation, but does not show that better physical prediction is the reason. A useful control would remove only the failure-video loss while retaining successful action targets and failure-based value pairs. A focused prediction diagnostic and a value head without future-video features would help explain the mechanism. The collapse of WAV and DreamZero on Full data also needs careful interpretation, since they retain their original objectives. One comparison with basic executable-action masking would make the benefit of the full design clearer (Section 6.1; Tables 17–18).
  2. I could not fully reconstruct the value-training pairs. Appendix D.1.2 samples windows around events and uniformly along trajectories, but those trajectories no longer share the same state after divergence. Please explain how the current contexts are matched for preference learning, beyond retaining the original group identifier. It is also unclear how the whole-branch utility in Eq. 29 labels a short window when success depends on a later retry. An example with the two context windows, future intervals, and numerical targets would help. Please include the utility formula and explain how shared-prefix windows and outcomes outside the prediction horizon are handled (Appendices C.4 and D.2.3).
  3. The action masks need a more operational description. Eq. 26 supervises Recovery actions before the failure timestamp, so it matters whether that timestamp marks the unsuitable action or the later detection of failure. Please also define how the executable indicator in Eq. 27 is assigned. For a window that crosses a failure and recovery boundary, it is unclear how the time-indexed masks interact with clean versus noised action inputs and the freezing of action-specific parameters. A short pseudocode example would resolve this. Given the large effect of action supervision in Table 17, these details are central to understanding the result (pp. 22 and 27–28).
  4. The smaller gains are difficult to assess without uncertainty estimates. The paper gives evaluation-trial counts, but not the number of trained models or confidence intervals. This is particularly relevant to the 0.65-point LIBERO improvement with four candidates, the additional 0.1 point with eight, and the small gap to the oracle. Please report paired uncertainty using the shared evaluation seeds and distinguish it from training variability. Since WAV's LIBERO success rates are taken from its paper, small differences against that result should be interpreted cautiously. The large robustness gains are encouraging; the finer comparisons need more support (Tables 3 and 18; Appendix E.2).
Questions for authors
  1. What happens when failure-video supervision alone is removed? How much does selection depend on future-video features, and how does a baseline with executable-action masking compare?
  2. Could you show one complete Stage-2 preference pair, including the contexts, temporal alignment, utility calculation, and handling of outcomes beyond the window?
  3. How are failure onset and executable intervals annotated? Could you show the action inputs, loss masks, and gradient updates for a window spanning failure and recovery?
  4. How many independent training runs were used, and what uncertainty accompanies the within-PlayWAM improvements and oracle comparison? This would help determine which small gains support a firm conclusion.
6 comments

Overall

Thank you for your detailed review. We add targeted controls and paired uncertainty, and clarify branch-outcome supervision, time-aligned pairs, and action-mask routing.

W1 + Q1: Failure-video supervision, value features, and masking

Removing failure-video loss lowers robustness and worsens FVD; removing the explicit video readout weakens selection; executable-action masking substantially improves WAV on Play-Full. These controls use single evaluations, with the original Full reference of 53.7% Robust Avg.

Failure-video control. We remove video loss only on failure-designated dynamics targets, retaining trajectories, eligible action targets/masks, failure-based value labels/pairs, training budget, and B=4. Each generator's head is trained on its frozen features with the same Stage-2 protocol. Robust Avg. falls to 49.9%; both Post-CF and Post-DS fall 3.8 pp, versus 0.7 pp in Nominal. On identical held-out failure/intervention windows, Full's FVD is 184.6 versus 217.1, a 32.5 (15.0%) reduction. Predictions use recorded executed actions, matched contexts, nine-frame futures, and identical feature/sampling settings. This supports improved video prediction and robust execution, but FVD does not establish contact accuracy or a specific physical mechanism.

Video readout. With the same frozen generator, B=4 candidate sets, and ranking/calibration training, Action-only retains action hidden features but omits the explicit video readout. Full gains 2.4 pp (53.7 vs. 51.3). Action tokens may still carry visual information through stream interaction; this isolates the additional video readout, not all visual information.

Masked baseline. WAV+Mask changes only action-target eligibility, retaining WAV's remaining objective without PlayWAM's clean-action dynamics route or action-specific freezing. It raises Play-Full robustness from 10.2% to 29.0%, versus 30.3% on Play-Success. Thus, eligibility explains a substantial part of the degradation; the residual 1.3 pp is descriptive. We will clarify that unmodified WAV/DreamZero results do not show an inherent inability to learn from failures. The remaining PlayWAM gap compares complete systems and cannot be assigned to one routing component.

Table 6-W1-A — Failure-video control and prediction quality

Caption. Single evaluation, B=4; success rates (%). FVD uses the matched recorded-action prediction protocol above.

VariantNominal ↑Post-CF ↑Post-DS ↑Robust Avg. ↑FVD ↓
Full PlayWAM70.357.849.653.7184.6
w/o Failure-video Loss69.654.045.849.9217.1
Full − ablation+0.7+3.8+3.8+3.8−32.5

Table 6-W1-B — Value-head inputs

Caption. Same frozen generator and B=4 candidates; single-evaluation success rates (%).

Value-head inputNominal ↑Post-CF ↑Post-DS ↑Robust Avg. ↑
Action-only69.555.147.551.3
Video + Action (Full)70.357.849.653.7
Full − Action-only+0.8+2.7+2.1+2.4

Table 6-W1-C — Executable-action masking

Caption. Single-evaluation Robust Avg. (%); WAV+Mask changes only action-loss eligibility.

MethodTraining dataAction supervisionRobust Avg. ↑
WAVPlay-SuccessOriginal30.3
WAVPlay-FullOriginal, unmasked10.2
WAV + Action MaskPlay-FullExecutable-action mask29.0
Full PlayWAMPlay-FullBranch-specific supervision53.7

W2 + Q2: Time-aligned pairs and branch-outcome targets

Our comparison is defined over counterfactual continuations from a shared anchor, including their subsequent windows after divergence. Different actions or interventions can lead these continuations to different later states; this divergence does not remove their shared origin or their role as alternative continuations in our controlled-play construction. Subsequent windows remain part of these branches and inherit their respective branch-utility labels. We therefore apply the branch-level preference to time-aligned windows from the first divergence onward, without requiring their current states to remain identical.

Here “same-state” refers to the shared initial state. We exclude identical prefixes and require a utility gap >0.1 (Appendix C.4.2, Eq. (30)). We will clarify this in the manuscript.

The PlayBench implementation of Eq. (29) is

G_i^b=\frac{Y_i^b+0.5P_i^b}{1.5},$$ where Y is terminal success and P the completed fraction of equally weighted task phases; each phase counts once, including completion after retry. Windows inherit source-branch utilities. A failed branch may contain locally appropriate actions: lower rank denotes its recorded branch outcome, not that every local action is poor. A successful Recovery can have G = 1 despite earlier failure; value labels and action eligibility are distinct. **Outcomes can lie beyond the 36-action/9-frame horizon.** The target is complete-branch utility, not window-local return or a guarantee that the outcome is observable within the window. This is outcome-based weak supervision and can introduce temporal credit-assignment noise. We impose no additional current-state matching or horizon-local outcome filter. Huber calibration anchors scores to utility but cannot resolve unobservable events. The WAM stays frozen; Eq. (11) uses calibration weight 0.2. The shuffled-group control (**47.7% vs. Full's 53.7%**, Table 17) supports shared-origin grouping without proving every local preference noise-free. #### Table 6-W2 — A complete Stage-2 pair **Caption.** Schematic drawer example; fixed task goal, CF-altered controller target. Each context has four history frames, proprioception, and instruction; futures have 36 actions/nine frames. Both outcomes occur after the shown windows. Non-overlapping windows illustrate alignment, not sampler stride. Phases: approach, grasp, open. | Field | Nominal branch | CF branch | |---|---|---| | Initial anchor and task | Shared anchor; open the designated drawer | Same anchor and task | | Task-directed target | (1,1,1) | (1,1,1) | | Controller target after intervention | (1,1,1) | (2,2,1) | | Identical prefix | No ranking | No ranking | | First divergent window | Shared context c₀; future [0,36), moving toward the correct target | Shared context c₀; future [0,36), moving toward the perturbed target | | Next aligned window | Own context c₃₆ᴺ; illustrative position (1,0,1); future [36,72) | Own context c₃₆ᶜᶠ; illustrative position (2,1,0); future [36,72) | | Recorded terminal outcome | Success | Failure | | Completed phases | 3 of 3 | 1 of 3 | | Utility target | Y=1, P=1, G=1 | Y=0, P=1/3, G=1/9 | | Pair eligibility | Gᴺ−Gᶜᶠ=8/9>0.1 | Same utility gap | | Ranking at both shown windows | Preferred by branch outcome | Dispreferred by branch outcome |

W3 + Q3: Failure boundaries, masks, and gradient routing

Failure onset marks the start of the unsuccessful attempt, not its later detection. Mixed windows mask action targets by time; parameter freezing applies to dynamics-only updates, not individual tokens.

In Eq. (26), onset is identified retrospectively from attempt outcomes/logs; recovery starts with the successful corrective attempt. The intervening failed interval is excluded from action imitation, while eligible preceding behavior and successful correction remain supervised. Repeated failures follow the same rule. NC records the complete failed execution without retry and provides no action-imitation targets.

For CF/DS, action eligibility follows Eq. (27); all valid transitions retain dynamics supervision (Eq. (25)):

m_{i,t}^{\mathrm{act},m} &= v_{i,t}^mY_i^m e_{i,t}^m,\\ m_{i,t}^{\mathrm{dyn},m} &= v_{i,t}^m. \end{aligned}$$ From phase/attempt logs and intervention timestamps, e = 1 for task-aligned successful execution and 0 for failure-causing intervals or externally injected disturbance commands. Failed branches have no action targets; successful branches contribute eligible actions, including task-directed actions under altered CF settings. These masks differ from value labels. In the joint route, eligible action positions are noised; others use clean detached actions, and action loss is masked. Video targets follow time-aligned supervision roles. In the dynamics-only route, all actions are clean/detached and encoded at action flow time zero; only video loss is optimized, freezing action-specific modules while updating the world path/adapters. The routes are separately normalized and interleaved (Eq. (7); Appendix D.3.1). Joint attention couples positions, so masking action loss does not imply complete gradient isolation of ineligible tokens. We will make this operational description explicit. #### Table 6-W3-A — A mixed 36-step Recovery window **Caption.** Illustrative indices; all transitions valid. Inputs refer to the joint route. Video roles use timestamp alignment, not direct copying of the action mask onto nine frames. | Action interval | Recorded behavior | Action mask | Joint-route action input | Action loss | Video supervision | |---|---|---|---|---|---| | [0,10) | Eligible approach | 1 | Noised | Yes | Yes | | [10,22) | Failed attempt and intervening replanning | 0 | Clean, detached | No | Yes | | [22,36) | Successful corrective retry | 1 | Noised | Yes | Yes | #### Table 6-W3-B — Parameter updates **Caption.** Freezing is route-specific; joint action loss uses eligible positions only. | Parameter group | Executable joint route | Dynamics-only route | |---|---|---| | World path | Trainable | Trainable | | Action-to-video adapters | Trainable | Trainable | | Action expert | Trainable | Frozen | | Action-specific input/output projections | Trainable | Frozen | #### Algorithm 6-W3 — Mixed-window supervision ```text Input: video/actions, context, attempt outcomes, timestamps, validity. Build action masks from Eqs. (26)-(27); keep valid dynamics (Eq. (25)). Align video supervision roles with action timestamps. Joint update: Noise eligible actions; keep others clean/detached. Predict video/action flows. Apply joint video loss and masked action loss; update all components. Dynamics-only update: Use clean/detached actions, encoded at action flow time zero. Noise video at sampled flow time; optimize dynamics video loss. Freeze action-specific modules; update world path and adapters. Normalize routes separately and interleave (Eq. (7)).
Reviewer_seSfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PlayWAM, a framework for training and deploying world action models. PlayWAM leverages simulation controllability to augment training datasets with counterfactual data including various types of failure and play behaviors. The action head is supervised only with successful demonstrations while dynamics and value are supervised with the entire dataset. PlayWAM is tested on newly introduced as well as modified-existing simulated benchmark task setups and shows that it is able to leverage variable quality data to improve task performance. The authors also demonstrate transfer of simulated failure and play experience to improved success on real-world tasks.

Critical strengths

Method provides a mechanism for leveraging failure, recovery and play data for training WAMs without degrading the action-head performance.

Demonstrates sim-to-real transfer of failure and play experience.

Critical weaknesses

Contribution and details are not clear.

Some aspects of the method contribution are not made clear within the main text.

How exactly are the anchor groups generated and annotated? It seems like a very critical part of the method but details are not described in the main text beyond the coarse types of interventions collected, so one cannot judge the feasibility and scalability of this approach. Same goes for the “simulator utilities” used for value learning.

Is your value learning approach novel? How does it relate to previous value-guided action selection?

Limitations are not discussed.

It seems that what you are proposing as a method relies heavily on access to privileged sim information for training. That implies that the method is a sim-to-real approach but that is not highlighted enough in the paper in my opinion. To my understanding, your method relies on having high-quality play data which requires defining useful branch data per task. Value learning also requires defining “simulator utilities” per task. Although parts of the data collection pipeline is automated, it still requires manual definitions per task. What are the limitations of this approach? Discussing these and other limitations will help the reader better understand the method itself I believe.

Questions for authors
  • The value head is trained on differentiating distinct branches within the same group. The action head is only trained on successful trajectories, which means that the value head ranks actions sampled only from the success distribution at test-time. How do you explain the contribution of the value in action selection given the above?
  • The experiments are missing a motivation for the specific baselines used. How do they differ from your approach and how does that help contrast the key aspects of your method?
  • In section 6.3, what do you mean by transfers across simulators? Is this just another benchmark you test your method on or is there an actual learning transfer in this setting?
6 comments

Overall

Thank you for your thoughtful feedback. We clarify the controlled-play pipeline, contributions and baselines, value selection, and evaluation scope and limitations.

W1: Anchor groups, annotations, and scalability

Anchor groups and labels are generated automatically after task configuration; the demonstrated scale is 32,000 groups and 160,000 trajectories across 32 tasks. Collection exceeds 1,000 hours; PlayBench evaluates a 12-task subset (Sec. 4.2).

In Isaac Sim, we randomize an episode, execute a shared prefix, save the anchor state, and restore it independently for five continuations (Sec. 4; Appendix B.2–B.3). Scripted experts compose reusable skills. Nominal records first-attempt success; NC records a complete failed execution without retry; Recovery retains failure and adds bounded corrective retries, sharing the matched NC's failed prefix. CF changes task/skill parameters, such as target pose/contact choice; DS injects action/motion noise, dropout, or delay. Both may succeed or fail. Scene randomization varies across groups; non-intervention factors remain fixed within a group.

Task predicates and simulator state provide success and phase completion; phase/attempt logs and intervention records provide timestamps. Our PlayBench implementation of the outcome/progress utility in Appendix C.4.1, Eq. (29), is

G_i^b=\frac{Y_i^b+0.5P_i^b}{1.5}.$$ Y is terminal success; K is the number of configured phases; binary sₖ denotes phase completion. Phases have equal weight and count once, including completion after retry. Successful Nominal/Recovery branches have G = 1; failure after one/two phases of a three-phase task gives G = 1/9 or 2/9. Success dominates utility, while progress distinguishes failures. Windows inherit source-branch utilities. Same-anchor pairs satisfying the utility-gap threshold are ranked from the first divergent window onward along the shared timeline; identical prefixes are excluded, and later current contexts need not match. Calibration aligns scores to the same utility targets. Snapshot restoration, branching, recording, and annotation run across parallel instances once skills, predicates, interventions, and retry budgets are configured. New tasks may require new skills; automation does not remove task setup. We will clarify this pipeline and utility in the main text and release code, controllers, and configurations upon acceptance.

W2 + Q2: Contributions and baseline rationale

Our contribution integrates controllable data infrastructure, role-dependent supervision, and lightweight candidate selection; we do not claim that ranking or regression losses themselves are new. The parallel collection pipeline produces PlayData and the PlayBench intervention benchmark. Shared-origin branches supply dynamics evidence, eligible action targets, and utility supervision (Sec. 4; Appendix B).

PlayWAM learns dynamics from all valid transitions, imitates eligible successful segments, and trains a detached value readout on frozen contextualized video/action features (Eqs. (7)–(11)). At inference, it generates B joint candidates in one batch and scores completed hidden features once, without RGB decoding, separate value-latent sampling, or outer-loop refinement. “Single-round” retains the ordinary denoising steps.

DreamZero-5B tests joint generation with direct execution; our implementation uses Wan2.2-TI2V-5B (Sec. 6.1). WAV tests value-guided iterative latent-distribution refinement. Table W2 contrasts their designs. All receive identical trajectories within each Nominal, Play-Success, or Play-Full condition, retaining their objectives. Cross-family results compare complete systems: degradation on Play-Full does not establish that those architectures inherently cannot learn from failure.

For component-level evidence, within-PlayWAM controls hold architecture fixed: all-action imitation tests supervision routing, shuffled groups test shared-anchor organization, Recovery-to-Nominal replacement tests recovery experience, and fixed-generator selectors test value selection (Appendix E.4.1–E.4.2, Tables 17–18). We will add this contribution and baseline rationale to the revision.

Table W2 — Infrastructure, supervision, and inference

Caption. Original designs and our PlayBench implementation. “Not introduced” concerns our controlled same-anchor infrastructure; it does not preclude extensions. Method descriptions follow the original papers cited in the manuscript.

AspectDreamZeroWAVPlayWAM
Controlled collection/data/benchmarkNot introducedNot introducedParallel snapshot/restore, five modes, automatic labels; PlayData and PlayBench
World–action modelJoint video–action generationVideo, trajectory-value, action modulesJoint video–action generation
Play-Full supervisionOriginal joint objectiveOriginal video/value/action objectivesAll valid dynamics; eligible successful actions; branch utilities
ValueNo dedicated selectorFlow-based trajectory-value module; discounted returnsDetached hidden-feature readout; ranking + Huber
Action selectionDirect generated actionsIterative latent-distribution refinement, then action generationB parallel candidates; one latent readout and selection

W3: Privileged information and limitations

Privileged simulation information is used offline, not at deployment; collection is automatic only after task-specific setup. Experts and annotations use object poses, robot/articulation states, contacts, and task predicates. Deployed PlayWAM uses images, proprioception, and instruction without querying these privileged states or experts.

The physical-robot study is simulation-assisted learning with real demonstrations: each of eight tasks uses 100 successful real demonstrations and 200 simulated trajectories. Full replaces the simulated subset with structured PlayData while fixing the real data (Sec. 6.5).

New tasks require skill composition, phase/success/failure predicates, interventions, and retry rules, and may need new controllers. The utility formula is shared, but its inputs require task-specific definitions. Available skills, designed interventions, and predicates limit coverage and may omit failures or execution-quality factors. Simulator contact and sensing fidelity further limit physical transfer. Reusable skills reduce, but do not eliminate, engineering effort.

We will discuss these limitations and release the pipeline, atomic-skill library, and task configurations upon acceptance. We will continue expanding AtomicSkill coverage to support more behaviors and reduce task-composition effort.

Q1: Why value selection helps success-supervised actions

Learning from successful actions does not guarantee that every sampled candidate succeeds; the value head selects among their different predicted consequences. The generator approximates the demonstration distribution. Sampling variation, model error, and CF/DS-shifted states can still cause missed grasps, lost contact, or incomplete tasks.

Dynamics training includes failed transitions; video/action features therefore support learning utility beyond action likelihood. Action supervision shapes proposals, while branch preferences and utilities train selection (Eqs. (7)–(11)). With the same frozen Play-Full generator and B=4 candidate sets, Random, Action likelihood, and Full obtain 46.7%, 50.35%, and 53.7% Robust Avg. (original Table 18). Full gains 7.0 pp over Random and 3.35 pp over likelihood, isolating the selection rule under a fixed generator and budget.

Q3: What evaluation across simulators means

This is an independently trained evaluation on a second simulator, not transfer of an Isaac Sim-trained checkpoint to RoboTwin. We apply controlled play and learning to eight SAPIEN RoboTwin tasks with platform-specific experts/configurations.

Within each model family, architecture, pretrained initialization, trajectory count, and evaluation remain fixed. Normal uses 150 nominal trajectories per task; Play-Full uses 50 Nominal trajectories plus 50 failure–recovery pairs, also totaling 150 (Sec. 6.1; Appendix E.3). PlayWAM's Robust Avg. rises from 34.6% to 64.9% (+30.3 pp), supporting applicability in this second setting. We will rename Sec. 6.3 “RQ2: Paired Play Improves Robustness across Simulators” and revise the transfer wording accordingly.

Reviewer_ZK1NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper is based on the following key idea: if we have access to a robot simulator, such as Isaac Sim, we can branch out from any given state si (with history/context ci) and generate B possible futures, by perturbing the environment, then applying the task-specific scripted expert) and simulator to generate B possible action continuations a(1:H) and video continuations v(1:H). Each tajectory is then scored with success or failure metrics R(1:B). They then train a WAM video-action diffusion model on this data, where the action trajectories are only supervised for successful rollouts. Finally they train a value function with a ranking loss to predict which of the B rollouts is the best one to pick; this avoids the need to do iterative refinement of the planned action chunk. (The value function is an MLP trained on latent features derived from the WAM's rollouts.)

Critical strengths

Assessment. This is a neat way to distill the knowledge of a simulated environment (and hand-written expert controller) into a diffusion-based WAM that can be applied to other environments, including real ones. They show good empirical gains on various synthetic benchamrsk (PlayBench, RoboTwinPlay and Libero. and some good results on a real aloha bimanual arm.

Critical weaknesses

No error bars on results. The Libero results in table 3 seem to be at ceiling for all methods.

Small typos (not critical)

Line 278, Claude latex error 'Let mj,b= ⊮[b∈Ej]'.

Line 280. What are hidde features Hj,b? Is this activations from some layer of the DIT used inside of the diffusion model?

Line 315. Why did you use a ranking loss to train the value function and not a MSE regression loss?

Line 321. What is calibrated loss Lcal?

It was very unclear how the action sequences are derived from each branch. I had to dig into the appendix to see that they are generated from an expert written script. It would be good to give more details on this, otherwise the method is not reproducible.

Questions for authors

Do your expert written scripts, used to create action sequences for each of your branched trajectories, rely on access to privileged info (eg true proprio state and object locations) or does it work from pixels?

3 comments

Overall

We sincerely thank you for your careful review and positive assessment of our work. We especially appreciate your recognition of the idea of distilling simulator and scripted-expert knowledge into a WAM, and of the gains across simulated benchmarks and real-world manipulation. Your constructive suggestions help us improve both clarity and reproducibility. Below, we add evaluation uncertainty and clarify the value objective, hidden features, and expert data-generation pipeline.

W1: Error bars and the LIBERO ceiling

The value selector improves LIBERO performance without PlayData, with a positive paired gain across three evaluation seeds; B=4 remains our preferred efficiency–performance configuration. LIBERO uses official demonstrations and calibration-only value training (Sec. 6.1), testing the architecture's value independently of our play-data pipeline.

We evaluate fixed checkpoints three times with different evaluation seeds on LIBERO and PlayBench. LIBERO variants share task initializations/seeds within each repeat. We report sample SD and 95% t-CIs; paired intervals use within-repeat differences, with t(0.975,2)=4.30265. Due to computational constraints, we add evaluation repeats, not independent training seeds: these intervals quantify evaluation variability conditional on the trained models.

B4−Direct is +0.57 pp [0.254, 0.879], positive in all three repeats despite limited headroom. B8−B4 is +0.11 pp [−0.469, 0.685]. Together with B=4's 1.71× higher reference inference rate (4.03 vs. 2.35 Hz), this supports B=4 as our primary configuration. PlayBench Full reaches 53.28 ± 1.82% Robust Avg., with a mean CI of [48.758, 57.797]%. We will add these estimates to the manuscript; single-evaluation controls remain separately identified.

Table W1-A — LIBERO evaluation repeats

Caption. Fixed checkpoints; three paired evaluation seeds. Success rates are percentages.

StatisticDirectB=4B=8
Eval 197.998.598.7
Eval 298.699.399.1
Eval 398.599.099.3
Mean ± evaluation SD98.3 ± 0.498.9 ± 0.499.0 ± 0.3
95% t-CI[97.5, 99.2][97.9, 99.9][98.2, 99.8]

Table W1-B — Full PlayWAM evaluation repeats on PlayBench

Caption. Fixed TableTop/Mobile checkpoints; the same 12 tasks and weights per repeat. Robust Avg. averages Post-CF and Post-DS. Scores are percentages.

StatisticNominal (%)Post-CF (%)Post-DS (%)Robust Avg. (%)
Eval 170.354.250.552.3
Eval 267.860.443.852.1
Eval 370.760.150.755.4
Mean ± evaluation SD69.6 ± 1.558.2 ± 3.548.3 ± 3.953.3 ± 1.8
95% t-CI[65.8, 73.4][49.5, 67.0][38.6, 58.0][48.8, 57.8]

W2: Indicator-function notation

The symbol is an indicator function; we will correct the notation without changing the supervision rule. Immediately before Eq. (6), Sec. 5.1, the expression should be:

mj,b=1[b∈Ej]m_{j,b}=\mathbf{1}[b\in\mathcal E_j]

Here ℰⱼ contains eligible joint video–action supervision segments: 1 if eligible, 0 otherwise.

W3: Source of the hidden features

The hidden features H come from the last WAM Transformer block, immediately before the video/action velocity-output projections. These DiT tokens represent the future video/actions of branch b in group j conditioned on visual history, current robot state, and instruction (Sec. 5.1, Eq. (6)).

The head separately mean-pools video/action tokens, projects them to a shared space, adds the flow-time embedding, and predicts a scalar with an MLP (Eq. (8); Appendix D.2.3). Stage 2 freezes the WAM and detaches its features, training only the head. At inference, completed candidates are scored at τ = 0 without RGB decoding or a separate value encoder. We will state the extraction location explicitly.

W4 + W5: Why ranking, and what calibration does

Ranking directly supervises candidate ordering; Huber calibration anchors scores to branch utility. Their combination outperforms either term alone and MSE in our controls. Selection depends on relative scores (Eq. (9)), whereas MSE fits absolute targets. A common score offset changes MSE but leaves ranking and the selected candidate unchanged.

Expanding the Huber term specified before Eq. (11),

\mathcal L_{\mathrm{cal}} &= \mathbb E_{w\sim\mathcal W_{\mathrm{cal}}}[\operatorname{Huber}(\hat v_w-G_w)],\\ \mathcal L_V &= \mathbb E_j[\mathcal L_{\mathrm{rank}}^j]+0.2\,\mathcal L_{\mathrm{cal}}. \end{aligned}$$ Here the expectation is over valid calibration windows; each predicted value is regressed to its source branch's normalized utility. The weight 0.2 is in Appendix D.4, Table 9. Our PlayBench implementation of Eq. (29) uses: $$G=\frac{Y+0.5P}{1.5}.$$ Here Y is terminal success and P the fraction of completed, equally weighted task phases. Ranking supplies relative preferences; calibration constrains score offsets and magnitudes against the utility scale. Both train only the head, with the WAM frozen. Play-Full windows inherit source-branch utilities. Pairs originate from the same initial anchor and satisfy the utility-gap condition in Appendix C.4.2, Eq. (30). Ranking starts at the first divergent window and continues along the shared timeline; identical prefixes are excluded, and later contexts need not match. Huber-only and MSE-only use identical regression windows/targets. With generator, head architecture, features, and B=4 fixed, Full improves over MSE-only by **4.3 pp** and Ranking-only by **1.5 pp** (Table W4). These single-evaluation controls support the combined objective. We will clarify its definition and motivation. #### Table W4 — Value-objective ablation on PlayBench **Caption.** Same frozen generator, head architecture, features, and B=4. Single evaluations; Full uses the original 53.7% reference, distinct from Table W1-B. Scores are percentages. | Value objective | Nominal ↑ | Post-CF ↑ | Post-DS ↑ | Robust Avg. ↑ | |---|---|---|---|---| | Ranking + Huber (Full) | **70.3** | **57.8** | **49.6** | **53.7** | | Ranking-only | 69.9 | 56.4 | 48.0 | 52.2 | | Huber-only | 68.8 | 53.8 | 47.0 | 50.4 | | MSE-only | 68.4 | 52.9 | 45.9 | 49.4 | ### W6 + Q1: Expert actions, privileged information, and reproducibility **Actions come from task-configured scripted experts using privileged simulator state offline; deployed PlayWAM uses visual history, proprioception, and the instruction only.** The Isaac Sim experts compose reusable atomic skills/controllers, with task-specific skill sequences, grasp/contact and motion-plan sampling, success predicates, and phase-specific retry budgets (Appendix B.2–B.3). Each group executes a shared prefix, saves an anchor, and independently restores it for each continuation. Nominal records first-attempt success. NC completes and records a normal failed execution without retry. Recovery retains the failed attempt and adds corrective replanning and bounded retries; matched NC–Recovery branches share the failed prefix. These three modes have no explicit CF/DS intervention; controller sampling and physical interactions produce different outcomes. CF changes a task/skill parameter such as target pose or contact choice; DS injects action/motion noise, dropout, or delay. CF and DS may succeed or fail. The state-based experts use ground-truth object poses, robot/articulation states, and grasp/contact information. We record executed commands, observations, robot states, phase/attempt logs, intervention timestamps, and outcomes. Simulator state and task predicates provide success/progress/event labels for utilities and masks. The deployment context (Secs. 3 and 5.1) is $$c_t=(o_{t-L:t},q_t,g).$$ It excludes privileged object/contact states and outcome labels; neither action generation nor value selection calls the experts. We will bring the pipeline details into Sec. 4 and release the collection code, atomic skills/controllers, and task configurations **upon acceptance**, including initialization, sampling, interventions, retries, and success predicates.

Thank you again for your time and thoughtful review. As the discussion period draws to a close, we would appreciate hearing whether our responses and revisions have addressed your main concerns. If any questions remain, we would be happy to provide further clarification before the discussion closes.

Thank you for considering our responses.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. We summarize the shared concerns, clarifications, and new evidence below.

1. Contribution and data collection. Our contribution combines controllable data infrastructure, selective supervision, and lightweight candidate selection. The parallel pipeline produces PlayData and PlayBench; PlayWAM learns dynamics from all valid branches, imitates eligible successful actions, and selects among parallel video–action candidates using completed hidden features without RGB decoding or outer-loop refinement. Ranking/regression losses themselves are standard.

Collection is automatic after task configuration: scripted experts compose reusable atomic skills, restore shared anchors, and annotate outcomes/events. The collection covers 32 tasks, 32,000 groups, and 160,000 trajectories; PlayBench evaluates 12 tasks. Privileged simulator information is used offline; deployment uses images, proprioception, and instruction. NC records complete failure without retry; Recovery retains failure and adds bounded corrective retries. CF/DS may succeed or fail.

2. Shared-anchor ranking and supervision. “Same-state” means a shared initial simulator state; subsequent paired windows need not have identical current states. They remain counterfactual continuations from that anchor. We rank time-aligned windows from the first divergence, excluding identical prefixes and requiring a branch-utility gap >0.1. Each window inherits its branch utility. Ranking expresses branch-outcome preference, not superiority of every local action.

On PlayBench, G=(Y+0.5P)/1.5, where Y is terminal success and P is the completed fraction of equally weighted task phases. Ranking learns relative preferences; Huber calibration anchors scores to utility. Both train only the head on frozen WAM features. Failed intervals retain dynamics supervision but are excluded from action imitation. The revision specifies feature extraction, temporal masks, and update routes.

3. New mechanism controls. These are single-evaluation PlayBench controls with B=4 and the original Full reference. Robust Avg. averages Post-CF and Post-DS. Value-input/objective controls share the frozen generator; the failure-video ablation trains its head on its corresponding frozen generator.

ConfigurationRobust Avg. (%)FVD ↓
Full: Ranking + Huber53.7184.6
w/o Failure-video Loss49.9217.1
Action-only value input51.3—
Ranking-only52.2—
Huber-only50.4—
MSE-only49.4—

Removing failure-video loss costs 3.8 pp and worsens matched recorded-action FVD by 32.5, supporting failure-video supervision. FVD does not establish contact accuracy. Full gains 2.4 pp over Action-only, supporting the explicit video readout; action features can still contain visual information. The objective controls support combining ranking with calibration.

WAV+Action Mask raises Play-Full robustness from 10.2% to 29.0%, versus 30.3% on Play-Success. Thus, unmodified baseline degradation does not establish an inherent inability to learn from failures. Masking explains a substantial part of the drop; the remaining PlayWAM gap compares complete systems.

4. Repeated evaluation and candidate budget. Due to compute constraints, we report three fixed-checkpoint evaluation repeats, not independent training seeds. These statistics quantify evaluation uncertainty; the single-evaluation controls above remain distinct.

On LIBERO without PlayData, Direct/B4/B8 achieve 98.3±0.4% / 98.9±0.4% / 99.0±0.3% (mean±sample SD). B4−Direct is positive in all repeats: +0.6 pp [0.3,0.9] (paired 95% t-CI). B8−B4 is +0.1 pp [−0.5,0.7]. These results support the value architecture without PlayData. B4's 4.03 versus 2.35 Hz also supports our primary efficiency–performance configuration.

On PlayBench, Full achieves 53.3±1.8% Robust Avg., with mean 95% t-CI [48.8,57.8]%. Paired bootstrap comparisons give +6.0 pp [4.9,7.0] over Random and +1.7 pp [0.6,2.8] over Action-only. Full recovers 82.7% of observed Random-to-Oracle headroom; Oracle−Full is +1.3 pp [0.2,2.3]. Oracle is an evaluation-only candidate-set reference.

5. Revision and scope. We have incorporated these clarifications, results, protocols, and limitations. RoboTwin uses independently trained models on a second simulator; physical-robot experiments combine real demonstrations with simulated play. Task configuration, skill coverage, and simulator fidelity limit generality. Upon acceptance, we will release collection code, atomic skills, and task configurations, and continue expanding skill coverage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.