- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PlayWAM uses simulation to collect alternative continuations from shared states, including failures and successful recoveries. It learns dynamics from all recorded transitions and imitates only successful, executable action segments. A value head then selects among jointly generated future–action candidates in one round of sampling. The results show substantial robustness gains on PlayBench and RoboTwin-Play, smaller gains from selection on LIBERO, and better performance on eight real-world tasks when structured simulated data replace successful simulated data alongside fixed real demonstrations. I find the approach promising, but several details need clarification to establish why it works and how to reproduce the supervision.
Critical strengths
- The distinction between physical evidence and useful behavior is well motivated. Retaining a failed attempt for dynamics learning while imitating its successful correction is a sensible way to use recovery data (Section 5; Eqs. 25–28).
- The evaluation covers two simulators and physical robots, with meaningful controls within each model family.
- Scoring latent candidates avoids decoding videos and repeatedly refining proposals. Reporting both four and eight candidates makes the success/latency tradeoff visible (Table 3). The timing advantage over WAV is useful as a comparison of the evaluated systems.
Critical weaknesses
- The experiments do not yet clearly identify what is gained from learning failure dynamics. Full training adds both dynamics and value supervision, and the all-action ablation changes what the policy imitates. The random-selection result supports an improvement in generation, but does not show that better physical prediction is the reason. A useful control would remove only the failure-video loss while retaining successful action targets and failure-based value pairs. A focused prediction diagnostic and a value head without future-video features would help explain the mechanism. The collapse of WAV and DreamZero on Full data also needs careful interpretation, since they retain their original objectives. One comparison with basic executable-action masking would make the benefit of the full design clearer (Section 6.1; Tables 17–18).
- I could not fully reconstruct the value-training pairs. Appendix D.1.2 samples windows around events and uniformly along trajectories, but those trajectories no longer share the same state after divergence. Please explain how the current contexts are matched for preference learning, beyond retaining the original group identifier. It is also unclear how the whole-branch utility in Eq. 29 labels a short window when success depends on a later retry. An example with the two context windows, future intervals, and numerical targets would help. Please include the utility formula and explain how shared-prefix windows and outcomes outside the prediction horizon are handled (Appendices C.4 and D.2.3).
- The action masks need a more operational description. Eq. 26 supervises Recovery actions before the failure timestamp, so it matters whether that timestamp marks the unsuitable action or the later detection of failure. Please also define how the executable indicator in Eq. 27 is assigned. For a window that crosses a failure and recovery boundary, it is unclear how the time-indexed masks interact with clean versus noised action inputs and the freezing of action-specific parameters. A short pseudocode example would resolve this. Given the large effect of action supervision in Table 17, these details are central to understanding the result (pp. 22 and 27–28).
- The smaller gains are difficult to assess without uncertainty estimates. The paper gives evaluation-trial counts, but not the number of trained models or confidence intervals. This is particularly relevant to the 0.65-point LIBERO improvement with four candidates, the additional 0.1 point with eight, and the small gap to the oracle. Please report paired uncertainty using the shared evaluation seeds and distinguish it from training variability. Since WAV's LIBERO success rates are taken from its paper, small differences against that result should be interpreted cautiously. The large robustness gains are encouraging; the finer comparisons need more support (Tables 3 and 18; Appendix E.2).
Questions for authors
- What happens when failure-video supervision alone is removed? How much does selection depend on future-video features, and how does a baseline with executable-action masking compare?
- Could you show one complete Stage-2 preference pair, including the contexts, temporal alignment, utility calculation, and handling of outcomes beyond the window?
- How are failure onset and executable intervals annotated? Could you show the action inputs, loss masks, and gradient updates for a window spanning failure and recovery?
- How many independent training runs were used, and what uncertainty accompanies the within-PlayWAM improvements and oracle comparison? This would help determine which small gains support a firm conclusion.
6 comments
W2 + Q2: Time-aligned pairs and branch-outcome targets
Our comparison is defined over counterfactual continuations from a shared anchor, including their subsequent windows after divergence. Different actions or interventions can lead these continuations to different later states; this divergence does not remove their shared origin or their role as alternative continuations in our controlled-play construction. Subsequent windows remain part of these branches and inherit their respective branch-utility labels. We therefore apply the branch-level preference to time-aligned windows from the first divergence onward, without requiring their current states to remain identical.
Here “same-state” refers to the shared initial state. We exclude identical prefixes and require a utility gap >0.1 (Appendix C.4.2, Eq. (30)). We will clarify this in the manuscript.
The PlayBench implementation of Eq. (29) is
G_i^b=\frac{Y_i^b+0.5P_i^b}{1.5},$$ where Y is terminal success and P the completed fraction of equally weighted task phases; each phase counts once, including completion after retry. Windows inherit source-branch utilities. A failed branch may contain locally appropriate actions: lower rank denotes its recorded branch outcome, not that every local action is poor. A successful Recovery can have G = 1 despite earlier failure; value labels and action eligibility are distinct. **Outcomes can lie beyond the 36-action/9-frame horizon.** The target is complete-branch utility, not window-local return or a guarantee that the outcome is observable within the window. This is outcome-based weak supervision and can introduce temporal credit-assignment noise. We impose no additional current-state matching or horizon-local outcome filter. Huber calibration anchors scores to utility but cannot resolve unobservable events. The WAM stays frozen; Eq. (11) uses calibration weight 0.2. The shuffled-group control (**47.7% vs. Full's 53.7%**, Table 17) supports shared-origin grouping without proving every local preference noise-free. #### Table 6-W2 — A complete Stage-2 pair **Caption.** Schematic drawer example; fixed task goal, CF-altered controller target. Each context has four history frames, proprioception, and instruction; futures have 36 actions/nine frames. Both outcomes occur after the shown windows. Non-overlapping windows illustrate alignment, not sampler stride. Phases: approach, grasp, open. | Field | Nominal branch | CF branch | |---|---|---| | Initial anchor and task | Shared anchor; open the designated drawer | Same anchor and task | | Task-directed target | (1,1,1) | (1,1,1) | | Controller target after intervention | (1,1,1) | (2,2,1) | | Identical prefix | No ranking | No ranking | | First divergent window | Shared context c₀; future [0,36), moving toward the correct target | Shared context c₀; future [0,36), moving toward the perturbed target | | Next aligned window | Own context c₃₆ᴺ; illustrative position (1,0,1); future [36,72) | Own context c₃₆ᶜᶠ; illustrative position (2,1,0); future [36,72) | | Recorded terminal outcome | Success | Failure | | Completed phases | 3 of 3 | 1 of 3 | | Utility target | Y=1, P=1, G=1 | Y=0, P=1/3, G=1/9 | | Pair eligibility | Gᴺ−Gᶜᶠ=8/9>0.1 | Same utility gap | | Ranking at both shown windows | Preferred by branch outcome | Dispreferred by branch outcome |W3 + Q3: Failure boundaries, masks, and gradient routing
Failure onset marks the start of the unsuccessful attempt, not its later detection. Mixed windows mask action targets by time; parameter freezing applies to dynamics-only updates, not individual tokens.
In Eq. (26), onset is identified retrospectively from attempt outcomes/logs; recovery starts with the successful corrective attempt. The intervening failed interval is excluded from action imitation, while eligible preceding behavior and successful correction remain supervised. Repeated failures follow the same rule. NC records the complete failed execution without retry and provides no action-imitation targets.
For CF/DS, action eligibility follows Eq. (27); all valid transitions retain dynamics supervision (Eq. (25)):
m_{i,t}^{\mathrm{act},m} &= v_{i,t}^mY_i^m e_{i,t}^m,\\ m_{i,t}^{\mathrm{dyn},m} &= v_{i,t}^m. \end{aligned}$$ From phase/attempt logs and intervention timestamps, e = 1 for task-aligned successful execution and 0 for failure-causing intervals or externally injected disturbance commands. Failed branches have no action targets; successful branches contribute eligible actions, including task-directed actions under altered CF settings. These masks differ from value labels. In the joint route, eligible action positions are noised; others use clean detached actions, and action loss is masked. Video targets follow time-aligned supervision roles. In the dynamics-only route, all actions are clean/detached and encoded at action flow time zero; only video loss is optimized, freezing action-specific modules while updating the world path/adapters. The routes are separately normalized and interleaved (Eq. (7); Appendix D.3.1). Joint attention couples positions, so masking action loss does not imply complete gradient isolation of ineligible tokens. We will make this operational description explicit. #### Table 6-W3-A — A mixed 36-step Recovery window **Caption.** Illustrative indices; all transitions valid. Inputs refer to the joint route. Video roles use timestamp alignment, not direct copying of the action mask onto nine frames. | Action interval | Recorded behavior | Action mask | Joint-route action input | Action loss | Video supervision | |---|---|---|---|---|---| | [0,10) | Eligible approach | 1 | Noised | Yes | Yes | | [10,22) | Failed attempt and intervening replanning | 0 | Clean, detached | No | Yes | | [22,36) | Successful corrective retry | 1 | Noised | Yes | Yes | #### Table 6-W3-B — Parameter updates **Caption.** Freezing is route-specific; joint action loss uses eligible positions only. | Parameter group | Executable joint route | Dynamics-only route | |---|---|---| | World path | Trainable | Trainable | | Action-to-video adapters | Trainable | Trainable | | Action expert | Trainable | Frozen | | Action-specific input/output projections | Trainable | Frozen | #### Algorithm 6-W3 — Mixed-window supervision ```text Input: video/actions, context, attempt outcomes, timestamps, validity. Build action masks from Eqs. (26)-(27); keep valid dynamics (Eq. (25)). Align video supervision roles with action timestamps. Joint update: Noise eligible actions; keep others clean/detached. Predict video/action flows. Apply joint video loss and masked action loss; update all components. Dynamics-only update: Use clean/detached actions, encoded at action flow time zero. Noise video at sampled flow time; optimize dynamics video loss. Freeze action-specific modules; update world path and adapters. Normalize routes separately and interleave (Eq. (7)).
Overall
Thank you for your detailed review. We add targeted controls and paired uncertainty, and clarify branch-outcome supervision, time-aligned pairs, and action-mask routing.
W1 + Q1: Failure-video supervision, value features, and masking
Removing failure-video loss lowers robustness and worsens FVD; removing the explicit video readout weakens selection; executable-action masking substantially improves WAV on Play-Full. These controls use single evaluations, with the original Full reference of 53.7% Robust Avg.
Failure-video control. We remove video loss only on failure-designated dynamics targets, retaining trajectories, eligible action targets/masks, failure-based value labels/pairs, training budget, and B=4. Each generator's head is trained on its frozen features with the same Stage-2 protocol. Robust Avg. falls to 49.9%; both Post-CF and Post-DS fall 3.8 pp, versus 0.7 pp in Nominal. On identical held-out failure/intervention windows, Full's FVD is 184.6 versus 217.1, a 32.5 (15.0%) reduction. Predictions use recorded executed actions, matched contexts, nine-frame futures, and identical feature/sampling settings. This supports improved video prediction and robust execution, but FVD does not establish contact accuracy or a specific physical mechanism.
Video readout. With the same frozen generator, B=4 candidate sets, and ranking/calibration training, Action-only retains action hidden features but omits the explicit video readout. Full gains 2.4 pp (53.7 vs. 51.3). Action tokens may still carry visual information through stream interaction; this isolates the additional video readout, not all visual information.
Masked baseline. WAV+Mask changes only action-target eligibility, retaining WAV's remaining objective without PlayWAM's clean-action dynamics route or action-specific freezing. It raises Play-Full robustness from 10.2% to 29.0%, versus 30.3% on Play-Success. Thus, eligibility explains a substantial part of the degradation; the residual 1.3 pp is descriptive. We will clarify that unmodified WAV/DreamZero results do not show an inherent inability to learn from failures. The remaining PlayWAM gap compares complete systems and cannot be assigned to one routing component.
Table 6-W1-A — Failure-video control and prediction quality
Caption. Single evaluation, B=4; success rates (%). FVD uses the matched recorded-action prediction protocol above.
Table 6-W1-B — Value-head inputs
Caption. Same frozen generator and B=4 candidates; single-evaluation success rates (%).
Table 6-W1-C — Executable-action masking
Caption. Single-evaluation Robust Avg. (%); WAV+Mask changes only action-loss eligibility.