acceptodds
Under review as a conference paper at ICLR 2027

Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation

Abstract

World-Action Models (WAMs) couple action generation with predictions of how physical interactions unfold. However, current post-deployment learning paradigms typically improve behavior without requiring better world predictions. Especially in dexterous manipulation, small execution errors can compound in high-dimensional action spaces, hindering policy improvement and pushing interactions beyond the world model’s training distribution. Motivated by this, we propose Direct Experience World-Model Optimization (DEWO), a post-deployment learning paradigm for WAMs that, alongside action imitation, refines world representations through visual experience to better condition action generation. Specifically, it identifies interaction turning points and learns from successful and failed futures to support classifier-free guidance. An additional value head estimates task progress from video representations and activates guidance when progress stalls during inference. Across five DexJoCo tasks, DEWO improves average success across all three WAM formulations. Ablations show that visual supervision from successful and failed continuations improves both prediction and control beyond action supervision alone. On four real-world tasks across Wuji and Sharpa, 3 × 3 grid evaluations show that two rounds of deployment learning increase success from 51.0% to 71.7% in cells with at least one initial success, a gain of 20.7 percentage points. These findings support continued predictive learning for improving control through deployment experience, making world modeling an active part of WAM adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Action Model, post-deployment training, dexterous manipulation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_zF9YOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Experience World-Model Optimization (DEWO), a method for improving world-action models using deployment experience. It revisits states along failed rollouts to collect successful and failed continuations. Successful examples supervise both video and action prediction, while failed examples supervise video prediction without action imitation. The method also introduces success-conditioned action guidance, activated by a value head when progress stalls. Experiments cover simulated dexterous manipulation and real-world grasp-and-place tasks.

Critical strengths

s1. The paper addresses a meaningful question: how to learn from failed robot executions without imitating their unsuccessful actions. Using their observed visual outcomes as training targets is a sensible approach.

s2. The experiments cover multiple WAM formulations, five simulated dexterous tasks, and two real-world hand platforms. This provides useful evidence about the method’s applicability across different models and hardware.

s3. The paper includes helpful ablations on successful and failed visual supervision, experience selection, and guidance timing. Its detailed spatial evaluation also makes the limits of the real-world improvements visible.

Critical weaknesses

w1. The practical cost of collecting the proposed data is unclear. Real-world collection requires human inspection of failures, restoration of comparable scene and robot configurations, and repeated attempts from these configurations. This may require substantial human effort, particularly for dexterous manipulation. The paper does not report enough information to assess whether this procedure is more efficient than collecting additional demonstrations or human corrections.

w2. The evidence that better visual prediction leads to better action generation is limited. The ablations show that adding failed visual futures improves both prediction loss and task success, which is useful evidence. However, these improvements alone do not establish that better visual representations explain the control gains. It also remains unclear whether collecting more successful experience under the same collection budget would provide similar benefits.

w3. The method combines many components, but its main source of improvement remains unclear. These include replay exploration, critical-fragment selection, outcome labels, visual supervision, a value head, and selective CFG. Although the paper provides component ablations, the connection between its main claim about world-model learning and its reliance on inference-time guidance needs stronger evidence. On the two guidance-ablation tasks, DEWO without CFG performs slightly worse than the initial model on one task and slightly better on the other. The added complexity therefore needs clearer justification against simpler alternatives.

Questions for authors

q1. How much human time and how many robot attempts are required to collect the retained successful and failed continuations? Under the same collection budget, how does DEWO compare with collecting additional successful demonstrations or human corrections?

q2. Could the authors isolate the contribution of failed-video training by keeping successful-action data and inference settings fixed, including a comparison with CFG disabled? How does adding failed videos compare with adding more successful videos under a matched budget?

q3. Which components are necessary to obtain the reported gains? In particular, how much improvement comes from training on the selected experience, and how much comes from selective CFG? How was the CFG scale of 0.2 selected, and how sensitive are the results to this choice?

4 comments
AuthorsOpenReview ↗

Response to Reviewer zF9Y (1/3)

W1&Q1. Human effort and comparison with additional demonstrations

We appreciate the reviewer's concern regarding interaction cost and human effort. DEWO integrates collection with deployment-time capability assessment: the same recovery trials assess the policy's capabilities and provide successful and failed continuations for adaptation.

Simulation collection budget. Task-stage cues help localize candidate replay regions. Each window allows up to 20 anchors with ten continuations each, and querying stops at the first zero-success anchor. Across 116 trajectories with identifiable ITPs, the selected windows average 5.28 queried anchors (median: 3), corresponding to 52.8 continuation attempts per window, compared with the maximum of 200.

Human effort on physical robots. Our grasping sequences are short, with recognizable alignment, contact, and stabilization stages. Reviewing these stages to diagnose failures also identifies candidate replay anchors, allowing anchor selection to share the existing inspection workflow. Scene restoration is quick in this tabletop setup, and the policy generates four continuations per anchor without additional expert demonstrations.

We will report human inspection and restoration time, together with robot interaction costs, in the revised appendix.

Learning from the same deployment experience. We have also completed a complementary control comparing SFT and DEWO on the same newly collected experience, using Tennis Ball with the Sharpa hand. Within each backbone, both methods start from the same initial checkpoint. Evaluation uses a 3 × 3 grid with ten trials per cell. R0 is the initial policy; R1 and R2 follow one and two adaptation rounds.

Table 1. OpenWAM: grasping success, % (successes/90).

MethodR0R1R2
SFT34.4 (31/90)50.0 (45/90)17.8 (16/90)
DEWO34.4 (31/90)58.9 (53/90)100.0 (90/90)

The original real-world backbone shows a similar pattern under its complete grasp-and-place metric: from 13.3% (12/90) initially, SFT reaches 15.6% (14/90) and 8.9% (8/90), while DEWO reaches 23.3% (21/90) after each adaptation round.

SFT improves after the first round but declines after the second. In the OpenWAM control, this decline accompanies less stable grasping. Our interpretation is that repeated SFT may overfit the limited deployment data and disturb the pretrained action distribution. These results support DEWO's use of the same collected experience; they do not measure the relative acquisition efficiency of recovery trials and additional demonstrations or human corrections.

AuthorsOpenReview ↗

Response to Reviewer zF9Y (2/3)

W2&Q2. Isolating the contribution of failed visual supervision

The original ablation retains successful-action supervision while adding visual targets. Adding failed futures to successful visual supervision raises success from 88.7% to 90.7% on Water Plant and 74.0% to 77.3% on Fold Glasses. Held-out prediction loss also decreases. These results support improved predictive learning alongside improved control, but do not by themselves establish that a particular change in visual representations causes the control gains.

Matched successful-video and failed-video controls with CFG disabled. We additionally evaluated OpenWAM, providing a complementary backbone with additional large-scale pretraining. The two SFT controls share the same common training data, initialization, successful-action supervision, visual-supervision quantity, and training-update budget. They differ only in whether the additional visual targets come from successful or failed trajectories. These additional videos introduce no extra action targets, and both SFT controls are evaluated without CFG under the same inference settings. Full DEWO is included as a separate reference for the complete method.

Table 2. Success rates (%, ↑) with OpenWAM; 50 trials per task and condition.

Training variantInferenceFold GlassesPick BucketPinch TongsWater PlantFour-task avg.
SFT + additional successful videosNo CFG7694848885.5
SFT + additional failed videosNo CFG8094947485.5
DEWOValue-gated CFG8496949291.5

With CFG disabled, additional failed videos improve Fold Glasses and Pinch Tongs relative to additional successful videos, match Pick Bucket, and reduce Water Plant. Both controls average 85.5%. Thus, failed visual experience is useful on some tasks, but does not uniformly outperform an equal amount of successful visual experience.

Full DEWO reaches 91.5%. This six-point advantage reflects the combined experience-selection, learning, and guidance approach, rather than failed visual supervision alone. The matched budget here refers to training-data volume and updates. We will make the CFG-disabled control setting explicit and clarify the scope of the mechanism claims in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer zF9Y (3/3)

W3&Q3. Contributions of experience learning and selective guidance

The components follow one design logic: learning from ITP experience and selectively using that learning during execution. Successful and failed continuations supervise physical consequences; successful actions make feasible behavior accessible through a success condition. The same replay outcomes supervise progress estimation. The value gate determines when to intervene, while the learned success condition supplies the action direction.

Training and inference contributions. The original guidance ablation evaluates the same adapted FastWAM checkpoint under different inference settings.

Table 3. Success rates (%); three 50-scene sets per task and condition. Initial-policy results are included for reference.

TaskInitial policyDEWO without CFGContinuous CFGValue-gated CFG
Water Plant88.786.778.090.7
Fold Glasses72.073.352.077.3

Unguided performance remains close to the initial policy. Selective guidance adds 4.0 percentage points on both tasks relative to unguided DEWO, while continuous guidance reduces performance. These results show that the control benefit depends on how the learned success condition is applied. The visual-supervision comparisons in W2&Q2 address the contribution of predictive learning under controlled action data and inference settings.

Experience selection and simpler alternatives. Figure 4c compares critical experience with full trajectories at matched data sizes and training budgets: success increases from 89.3% to 90.7% on Water Plant and 74.0% to 77.3% on Fold Glasses. In an additional OpenWAM comparison on Fold Glasses, DEWO with ITP anchors achieves 84.0%, versus 80.0% when anchors are placed two seconds before GPT-annotated visible failures, under the same replay-budget limit and training budget.

We also compare simpler learning and intervention choices on Fold Glasses. SFT on ITP experience and unguided DEWO both achieve 80.0%. With the same DEWO model and value gate, an equal-magnitude random perturbation achieves 80.0%, compared with 84.0% for success-conditioned CFG. This supports the learned direction beyond an arbitrary intervention.

Table 4. Guidance timing on Fold Glasses; success rates (%), 50 trials per condition, and a single-trigger budget for all guided variants.

No guidanceRandom gatingFixed-schedule gatingProgress-based gating
80.080.082.084.0

Together, these comparisons support experience selection, the learned success direction, and selective activation against simpler alternatives.

CFG-scale selection and sensitivity. We empirically selected w = 0.2 using collection scenes whose spatial configurations differ from the evaluation scenes. The three 50-scene evaluation sets were not used for hyperparameter selection. The same scale is used across all five FastWAM simulation tasks.

We evaluated additional CFG scales on Fold Glasses using OpenWAM, holding the trained checkpoint, value-gating rule, and evaluation protocol fixed.

Table 5. CFG-scale sensitivity on Fold Glasses using OpenWAM; 50 trials per condition.

CFG scale wSuccess rate (%)
080.0
0.180.0
0.284.0
0.384.0

We will include more results and the parameter-selection protocol in the revised appendix.

Thanks for the additional experiments. The video comparisons and CFG ablations addressed my main concerns. If time permits, I’d appreciate a brief discussion of how DEWO’s recovery trials compare with additional demonstrations or human corrections under the same collection budget. I’m raising my score from 2 to 3.

Reviewer_jMQsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes DEWO, a post-deployment learning method for World-Action Models. It identifies “interaction turning points”, then samples successful/failed continuations from them to train the model under success/failure text conditions. At inference, a value head is used to detect stalled progress and trigger the guidance.

Critical strengths
  1. Well-motivated with robust gains in some settings. Using matched failed-successful continuations as visual supervision is a reasonable way to exploit deployment experience that action-only methods ignore. Improvements on FACT and real-world full-grid success are notable.
  2. Selective, value-gated guidance is well-designed. Using a value head on video features to decide when to apply guidance appears new for WAMs.
Critical weaknesses
  1. The evidence only partially supports the claimed mechanism. The paper attributes the improvements to refined world representations, while its own ablation shows that the DEWO-trained model without guidance performs on par with the initial checkpoint (86.7% vs. 88.7% initial on Water Plant, 73.3% vs. 72.0% initial on Fold Glasses; Fig. 4d vs. Table 1).

  2. Limited methodological novelty. The approach of outcome-conditioned training with condition dropout, as well as classifier-free guidance toward the desired outcome, follows Decision Diffuser (Ajay et al., ICLR 2023), which is not cited. RECAP has likewise carried this idea over to VLA policies. In addition, value supervision on WAM features is already present in the paper’s FACT-style baseline, so the novelty reduces to two components: the selection of turning points and the value-gated activation of guidance. The former is not compared against simpler anchor choices (Q1).

DEWO is a sensible approach to post-deployment learning for WAMs, and its full pipeline yields clear gains on FACT and in real-world evaluation. However, the evidence does not yet show that these gains come from improved world modeling rather than from gated test-time guidance. In terms of novelty, outcome-conditioned training with classifier-free guidance follows existing methods, so the contribution rests mainly on turning-point selection and value-gated guidance; the latter is supported by the ablation in Fig. 4d, while the former has not been compared against simpler alternatives. Addressing Q1 and Q3 would strengthen the paper’s conclusions.

Questions for authors
  1. Could the authors compare turning-point anchors with failure-point anchors (e.g., a fixed number of steps before the visible failure) under the same replay and training budget? On the real robots, can you add an SFT control trained on the same newly collected data?

2.The guidance window (replans 10–24, roughly 8–20 s of a 33 s episode) and the scale w = 0.2 are not motivated. How were they chosen, and were they tuned on the 3 × 50 evaluation sets?

3.Fig. 4d shows that guidance helps only when gated, supporting selective activation. To attribute the gain between the gate and the learned success direction, could the authors compare against SFT with the same gate (e.g., resampling the action chunk) and against a random perturbation of equal magnitude under the same gate?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jMQs (1/2)

W1&Q3. Predictive learning and selective guidance

The original no-CFG results show that unguided execution does not consistently improve over the initial checkpoint. DEWO's post-training also learns a success-conditioned prediction that CFG selectively emphasizes during execution. The gate determines when to intervene; the success-conditioned model supplies the learned direction.

Figure 4a–b provides evidence for predictive learning: with successful-action supervision retained, adding visual supervision reduces held-out video prediction loss by 32.7% on Water Plant and 17.8% on Fold Glasses, while task success increases from 86.7% to 90.7% and 72.0% to 77.3%, respectively. These results support improved predictive learning, while its effect on control also depends on how the learned condition is used at inference.

We additionally compare adaptation and intervention choices on Fold Glasses using OpenWAM, with 50 trials per condition.

Table 1. Success rates (%, ↑) on Fold Glasses.

Policy / training methodInterventionSuccess rate (%)
SFT on ITP dataNone80.0
SFT on ITP dataReplay and resampling at the first gate activation82.0
DEWONone80.0
DEWOEqual-magnitude random perturbation80.0
DEWOSuccess-conditioned CFG84.0

Resampling protocol.

Our gate detects stalled value progress rather than evaluating a candidate action. Simply drawing another action chunk during ordinary inference samples from the same conditional distribution. We implement the resampling control through replay: each evaluation trajectory with a gate activation is replayed to its first activation, where a new action chunk is sampled and execution continues with SFT. Successful trajectories without a gate activation remain counted as successes; the reported rate uses the full evaluation set.

For the same DEWO model and gating rule, success-conditioned CFG reaches 84.0%, compared with 80.0% for an equal-magnitude random perturbation. This supports the value of the learned direction beyond triggering an arbitrary intervention. Together, the comparisons support predictive learning and selective use of the learned success condition; they do not establish that visual representation refinement alone explains all control gains. We will clarify this distinction in the revised manuscript.

W2. Methodological contribution and relationship to prior work

We appreciate the reviewer's recognition of DEWO's motivation and selective guidance design. We thank the reviewer for pointing out Decision Diffuser (Ajay et al., ICLR 2023) and will add the missing citation. Its conditional generation, condition dropout, and CFG, together with related conditioning in RECAP and value supervision in our FACT-style baseline, provide foundations for our technical choices.

DEWO's central methodological contribution lies in connecting the probing of the current policy's local capability boundary with the acquisition and learning of ITP experience. The components are connected through the shared outcome-based learning signal: successful and failed continuations teach physical consequences, successful actions make feasible behavior accessible through a success condition, and replay outcomes supervise progress estimation. This estimate then determines when guidance strengthens the learned success condition. The design logic is learning from ITP experience and selectively using that learning during execution. The additional anchor comparison in Q1 and intervention controls in Q3 directly examine these two aspects.

Reference: Ajay et al., Is Conditional Generative Modeling All You Need for Decision-Making?, ICLR 2023.

AuthorsOpenReview ↗

Response to Reviewer jMQs (2/2)

Q1. Simpler anchors and real-robot SFT

Failure-point anchors.

Following the reviewer's suggestion, we use GPT to annotate visible failure times and place control anchors 2 seconds before those times. We sample continuations from these anchors and apply the same DEWO training procedure, under the same replay-budget limit and training budget.

Table 2. Success rates (%, ↑) on Fold Glasses using OpenWAM; 50 trials per condition.

Anchor selection for DEWOFold Glasses
2 s before GPT-annotated failure80.0
ITP anchors84.0

ITP anchors improve success by 4.0 percentage points over the failure-point control, supporting recoverability-based experience selection beyond a fixed offset from visible failure.

Real-robot SFT on the same data.

We added SFT controls on Tennis Ball with the Sharpa hand, using the original real-world backbone and OpenWAM. Within each backbone, SFT and DEWO start from the same initial checkpoint and use the same collected deployment experience for adaptation. Evaluation covers the 3 × 3 grid with ten trials per cell. R0 denotes the initial policy; R1 and R2 follow one and two adaptation rounds, respectively.

Table 3. Original real-world backbone: complete grasp-and-place success, % (successes/90).

MethodR0R1R2
SFT13.3 (12/90)15.6 (14/90)8.9 (8/90)
DEWO13.3 (12/90)23.3 (21/90)23.3 (21/90)

Table 4. OpenWAM backbone: grasping success, % (successes/90).

MethodR0R1R2
SFT34.4 (31/90)50.0 (45/90)17.8 (16/90)
DEWO34.4 (31/90)58.9 (53/90)100.0 (90/90)

SFT improves after the first adaptation round but declines after the second on both backbones. In the OpenWAM control, this decline was accompanied by less stable grasping. Our interpretation is that repeated SFT may overfit the limited real-world deployment data and disturb the pretrained action distribution. DEWO retains its first-round gains with the original backbone and improves further with OpenWAM. These controls support the benefit of DEWO beyond simply fine-tuning on the additional deployment experience.

Q2. Guidance-window and scale selection

We empirically selected the guidance window (replans 10–24) and scale (w = 0.2) using the collection scenes, whose spatial configurations differ from the evaluation scenes. The three 50-scene evaluation sets were not used for hyperparameter selection. The same window and scale are used across all five FastWAM simulation tasks.

The window defines when guidance is eligible; activation within it still depends on the value-based progress criterion. It therefore does not prescribe continuous guidance throughout this period.

Under our parameterization, active guidance uses f_guided = 0.8 f_base + 0.2 f_success. Thus, this setting retains most of the base prediction while incorporating the learned success condition.

We will clarify the selection protocol and the separation between collection and evaluation scenes in the revised appendix.

Reviewer_51QVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Experience World-Model Optimization (DEWO), a framework for post-deployment adaptation of world-action models. It uses replay-based exploration in simulation and human-assisted selection on physical robots to collect successful and failed continuations around interaction turning points. Both outcomes provide outcome-conditioned visual prediction supervision, while action imitation is restricted to successful trajectories. At inference, a value head monitors task progress and selectively activates success-conditioned classifier-free guidance. Experiments on five simulation tasks and four real-world manipulation tasks report improved success rates, with real-world gains concentrated at positions where the initial model already occasionally succeeds. Ablations examine the contributions of visual supervision, experience selection, and guidance timing.

The paper is well presented, the motivation is appealing, and the experimental coverage is broad. However, I am not yet convinced that the analysis isolates the specific benefit of failed experience or sufficiently justifies the complexity of the proposed framework. Several gains are modest without clear evidence of statistical robustness, while the real-world results show limited improvement beyond the first adaptation round. The replay and human intervention costs also need clearer quantification. Clarifying these issues and providing stronger controlled comparisons could change my assessment.

Critical strengths
  1. Clear organization and presentation. The paper is well organized, with a clear progression from motivation to method and experimental analysis. The figures and tables are visually appealing and easy to follow. In particular, the framework overview and component analysis help readers understand the different stages of the method and their contributions.

  2. Intuitive and insightful motivation. The paper builds on the observation that failed executions still contain valid information about physical interactions, even when their actions should not be imitated. Using this experience to improve world modeling, and thereby support better action generation, is an intuitive and insightful motivation. The distinction between learning from an execution’s physical consequences and imitating its behavior provides a useful perspective on deployment learning.

  3. Broad experimental evaluation. The experiments cover five simulation tasks, three prediction–action formulations, and four real-world manipulation tasks on two dexterous-hand platforms. This breadth makes the empirical evaluation substantial and strengthens the evidence that the approach is applicable across different model formulations and physical setups. The combination of baseline comparisons, component ablations, and spatial evaluations also provides several complementary views of its performance.

Critical weaknesses
  1. The analysis does not establish the specific benefit of failed experience. Appendix D.1 states that the success-only visual supervision variant uses successful futures from D0 and D+, whereas the full method additionally uses failed visual futures from D_fail. My concern is that this comparison adds visual training data while also introducing failure-specific content. I suspect that the small gains after adding D_fail could come from the additional visual training data, rather than the specific value of failed experience.

  2. The roles and connections of the components are unclear. I found the combination of replay-based turning-point selection, outcome-conditioned learning, a value head, and progress-gated guidance difficult to follow. The initial intuition is straightforward, but I struggled to understand how these components fit together and why all of them are needed. I am also not convinced that each component provides a distinct benefit that justifies the added complexity. The component ablations are helpful, but they do not fully address these questions or explain why these particular designs are necessary.

  3. Several improvements are modest, and their statistical robustness is unclear. Average success improves by 1.6 percentage points on FastWAM and 1.9 points on IDM, although the gain on FACT is substantially larger. Adding failed visual supervision improves success by only 2.0 and 3.3 points on the two ablation tasks. Without uncertainty estimates or results from independent training runs, it is difficult to judge how reliable these gains are.

  4. The real-world results show limited expansion of capability. Full-grid success increases from 20.6% to 29.0% after the first adaptation round, but does not improve further in the second round, reaching 28.9%. All 43 grid cells with zero initial successes remain unsuccessful. These results suggest that DEWO mainly makes existing capabilities more reliable. While this is useful and acknowledged by the authors, I see limited evidence that the method can sustain improvement over further rounds or extend performance to previously unsuccessful regions.

  5. The cost and sensitivity of experience collection need more attention. Appendix B.1 specifies that simulation queries start at step 96, occur every 24 steps, and sample ten continuations per anchor, with up to 20 anchors per failed trajectory. It is unclear why these settings were chosen or how sensitive the results are to them. The procedure allows up to 200 additional attempts per failed trajectory, although early stopping may reduce this number. This is a potentially substantial interaction cost that should be considered alongside the reported gains. On physical robots, selecting replay anchors requires human inspection, and collecting alternative continuations requires scene restoration and repeated trials.

Questions for authors
  1. In the visual-supervision ablation, how is the training-data budget matched between the success-only variant and the full method? Are failed visual targets replaced with successful ones, or is their loss simply disabled? Would the gains remain when compared with an equivalent amount of additional successful visual supervision?

  2. Could the authors explain why the different components are needed together? In particular, what connects replay-defined turning points during data collection to progress-based guidance during inference? Does the learned gate outperform simpler alternatives, such as a fixed schedule or random gating with the same activation budget?

  3. How consistent are the reported improvements across independent training runs? Could the authors provide uncertainty estimates for the main comparisons and Figure 4, particularly for the smaller gains on FastWAM, IDM, and the failed-visual-supervision ablation?

  4. Why does performance plateau after the first real-world adaptation round, and why do all 43 initially unsuccessful cells remain unsuccessful? Does the collection procedure obtain useful successful continuations for these cells, or does it mainly reinforce regions where the initial policy already succeeds?

  5. How were the replay start step, query interval, and number of continuations chosen, and how sensitive are the results to these choices? What are the actual additional environment interactions after early stopping, and how much human time is required for anchor selection and scene restoration on physical robots?


Post rebuttal: the rebuttal has moved my assessment in a positive direction, but the remaining concern about the evidence supporting the central insight prevents me from moving to a higher rating. My assessment is closer to 2.5; since this option is unavailable, I lean toward retaining my current score of 2.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 51QV (1/3)

W1&Q1. The value of interaction-turning-point experience

We agree that the original ablation does not fully separate failed experience from additional visual data, as it lacks an equal-volume replacement with successful videos. We therefore conducted a controlled comparison on OpenWAM, which has undergone additional large-scale pretraining. This may make successful alternatives accessible in more states, providing more opportunities to learn from interaction-turning-point experience.

The four augmented-data conditions share the same initialization, successful-action supervision, visual-supervision quantity, and training-update budget. Additional successful or failed trajectories provide visual supervision only, without extra action targets.

Table: Success rates (%, ↑) with OpenWAM, evaluated over 50 trials per task.

Training variantFold GlassesPick BucketPinch TongsWater PlantFour-task avg.
Original-data OpenWAM (reference)7486809283.0
Additional successful videos7694848885.5
Additional failed videos8094947485.5
Short ITP windows8096888086.0
DEWO (OpenWAM)8496949291.5

Hammer Nail is omitted because performance varies substantially across nearby checkpoints; the comparisons below concern the four reported tasks.

Additional failed videos have mixed effects: relative to additional successful videos, they improve Fold Glasses and Pinch Tongs but reduce Water Plant, leaving the average unchanged at 85.5%. DEWO improves over the successful-video control on all four tasks, by 8, 2, 10, and 4 percentage points, respectively.

DEWO's 91.5% average exceeds both video controls by 6.0 percentage points and short ITP windows by 5.5 points. Under matched budgets, these results support the complete approach to selecting and learning from interaction-turning-point experience, beyond adding successful or failed videos alone. We will include these controlled comparisons in the revised manuscript.

W2&Q2. Roles and connections of the components

DEWO learns from interactions near the current policy's local capability boundary, where successful alternatives become less accessible as execution evolves. Replay identifies interaction turning points (ITPs) through changes in empirical recoverability, providing an empirical approximation to this boundary. The components follow one design logic: learning from this experience and selectively using that learning during execution.

Replay collects successful and failed continuations from comparable states. Outcome-conditioned learning associates their physical futures with their outcomes: both provide visual supervision, while only successes provide action supervision. This teaches failure consequences without imitating failed actions and makes successful behavior accessible through an explicit success condition. The replay exploration also supplies discounted terminal-success targets for the value head, linking collected experience to an online progress estimate.

CFG uses the same WAM's base and success-conditioned action predictions to adjust the strength of the learned success condition. A fixed rule monitors changes in predicted value and activates guidance when progress stalls. Thus, the connection from replay to inference is the shared outcome-based learning signal: replay identifies informative experience, and value learning provides feedback for selectively applying what was learned. The gate does not directly classify states as ITPs. This design reuses collected outcomes and WAM representations, with shared parameters for conditional prediction and no separate policy or learned gating network.

We compared progress-based gating with random and fixed-schedule alternatives on Fold Glasses in simulation, with 50 trials per condition and a single-trigger budget for all guided variants.

Table: Success rates (%, ↑) on Fold Glasses in simulation, with 50 trials per condition.

Gating strategyNo guidanceRandom gatingFixed-schedule gatingProgress-based gating (DEWO)
Success rate (%)80.080.082.084.0

Progress-based gating achieves the highest observed success rate: 4.0 percentage points above no guidance and random gating, and 2.0 points above fixed scheduling. This comparison provides evidence in favor of execution-dependent feedback for timing guidance under the same trigger budget. We will add these comparisons and clarify the unified design in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 51QV (2/3)

W3&Q3. Modest improvements and statistical robustness

We agree that the average gains on FastWAM and IDM are modest. The real-world analysis in W4/Q4 suggests that access to successful alternatives matters for learning from failures around interaction turning points.

Task-level gains vary: DEWO improves Fold Glasses by 5.3 and 4.0 percentage points on FastWAM and IDM, respectively, and Water Plant by 2.0 and 10.7 points. Pick Bucket offers less headroom on FastWAM (92.0% → 93.3%). In Hammer Nail, the policy can repeatedly strike and retry after grasping the hammer. Success may therefore depend less on a single decisive interaction, potentially making turning-point experience less informative. This is a possible explanation for the lack of improvement on FastWAM and IDM.

Regarding uncertainty, the original simulation comparisons, including Figure 4, evaluate a fixed checkpoint per condition on three 50-scene sets per task. The new simulation comparisons in W1/Q1 and W2/Q2 each use 50 trials per task and condition. From the per-set results in Appendix F, five-task success changes from 80.7 ± 1.7% to 82.3 ± 3.2% for FastWAM and 82.1 ± 1.2% to 84.0 ± 1.1% for IDM. Here, ± denotes the sample standard deviation across evaluation sets, not independent training runs.

For Figure 4's failed-visual-supervision comparison, approximate 95% binomial Wilson intervals based on 150 trials per condition are 82.6–92.8% versus 84.9–94.4% for Water Plant (88.7% → 90.7%), and 66.4–80.4% versus 70.0–83.3% for Fold Glasses (74.0% → 77.3%). These are marginal success-rate intervals; they do not quantify uncertainty in the paired differences or variation across training runs.

We have not evaluated independent training seeds for these comparisons. The matched-budget OpenWAM results in W1/Q1 provide additional evidence across four tasks, but do not replace that assessment. We will report evaluation uncertainty and clarify this limitation alongside the task-level analysis.

W4&Q4. Real-world capability expansion

We agree that the original real-world results primarily demonstrate improved reliability: full-grid success rises from 20.6% to 29.0% after the first adaptation round and remains at 28.9% after the second, with all 43 initially unsuccessful cells remaining unsuccessful.

DEWO turns experience obtained while evaluating the policy's capabilities into post-training supervision. For the 43 initially unsuccessful cells, collection did not obtain successful alternatives to support success–failure comparisons around ITPs. Failed continuations provide visual supervision about physical consequences, but no successful-action targets. Our interpretation is that the first round strengthens behaviors with accessible recovery paths, while further improvement is constrained by the availability of informative new experience. The plateau is consistent with this limitation.

Initial task demonstrations were collected only in the central grid cell. Additional successful demonstrations across the workspace could address this coverage limitation, but would change the data setting and make it harder to distinguish gains from deployment experience from gains due to broader demonstration coverage.

We therefore added an OpenWAM evaluation on the Sharpa hand with the same four objects as in the original study, retaining initial task fine-tuning on central-cell demonstrations only. Following the original protocol, each object is tested ten times per cell across the 3 × 3 grid, giving 360 trials per round.

The initial OpenWAM policy already achieves nonzero grasping success throughout the grid, and DEWO reaches 100% grasping success after two adaptation rounds. This demonstrates improved spatial performance beyond the demonstration region on the evaluated hand and objects. It is consistent with broader pretrained capabilities making successful alternatives accessible in more states, providing more opportunities to collect informative ITP experience and achieve further post-training gains.

We will include the OpenWAM results and clarify how improvement depends on the initial policy and collected experience. The original experiment supports reliability gains within a limited spatial range; the supplementary experiment demonstrates broader spatial performance with a stronger initial policy.

AuthorsOpenReview ↗

Response to Reviewer 51QV (3/3)

W5&Q5. Cost and sensitivity of experience collection

We appreciate the reviewer’s concern regarding interaction cost and human effort. DEWO integrates collection with deployment-time capability assessment: the same recovery trials assess the policy’s capabilities and provide successful and failed continuations for adaptation.

Simulation collection budget. Task-stage cues help localize candidate replay regions. Each window allows up to 20 anchors with ten continuations each, and querying stops at the first zero-success anchor. Across 116 trajectories with identifiable ITPs, the selected windows average 5.28 queried anchors (median: 3), corresponding to 52.8 continuation attempts per window, compared with the maximum of 200.

Parameter choices and sensitivity. We empirically chose step 96 (3.2 s at 30 Hz) to focus probing around the transition from the initial approach to physical interaction. The 24-step interval matches one executed action chunk (0.8 s), controlling temporal resolution. Ten continuations per anchor provide empirical recoverability estimates in increments of 0.1, balancing sampling precision against interaction cost. We used the same configuration across all five simulation tasks.

Human effort on physical robots. Our grasping sequences are short, with recognizable alignment, contact, and stabilization stages. Reviewing these stages to diagnose failures also identifies candidate replay anchors, allowing anchor selection to share the existing inspection workflow. Scene restoration is quick in this tabletop setup, and the policy generates four continuations per anchor without additional expert demonstrations.

We will add a sensitivity analysis of the query configuration and measurements of human time spent on real-world experience collection to the appendix of the revised manuscript.

Reviewer_djU2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new post-training method for WAMs based on continued world modeling. It proposes to revisit critical states in failed policy deployment trajectories and resamples alternative continuations to collect successful and failed experiences. It then performs outcome-conditioned SFT on these data using successful actions for imitation and both outcomes for video prediction. A value head trained on video-model features monitors task progress and activates success-conditioned CFG when progress stalls during inference. Experiments on five DexJoCo tasks and two real-world dexterous-hand platforms show improved manipulation performance over other post-training approaches, with ablations examining experience selection, visual supervision, and guidance timing.

Critical strengths
  1. The proposed framework offers a pragmatic and well-motivated approach to improving WAMs through deployment experience.
  2. The experiment results are comprehensive. The paper provides extensive comparisons with different post-training methods, and demonstrates the effectiveness of DEWO across multiple WAM formulations. Real-world experiments on two dexterous-hand platforms further support its practical applicability.
  3. The ablation studies provide empirical support for the paper's method design and main claims.
Critical weaknesses
  1. The proposed framework mainly combines existing techniques, including failure-aware visual supervision, outcome-conditioned learning, value estimation, and CFG. While the overall integration is practical, the methodological novelty appears relatively limited.
  2. Several key components, such as turning-point identification, continuation selection, and value-gated guidance, rely on heuristic design choices without sufficient justification or sensitivity analysis.
  3. The ablation studies are conducted on only two of the five simulation tasks. Although they provide useful evidence for the proposed components, broader evaluations would be needed to make the claims more convincing.
  4. Several reported improvements are relatively modest in Table 1, such as FastWAM (80.7% to 82.3%) and IDM (82.1% to 84.0%). It is difficult to determine whether these smaller gains reflect consistent improvements rather than experimental variability.
  5. The proposed critical-experience collection relies on repeated state restoration and continuation sampling, which are straightforward in simulation but considerably more challenging on physical robots. In the real-world experiments, your turning points are identified through human inspection, and task-relevant states are manually restored approximately. This differs substantially from the automated procedure in simulation and raises concerns about the scalability of the framework to more complex manipulation tasks and autonomous deployment. For example, in scenarios involving deformable objects, contact-rich tasks or long-horizon tasks, it is hard for human to inspect turning points or recover the states. The paper also provides limited analysis of the additional interaction and human intervention costs involved.
Questions for authors
  1. How exactly does continued video prediction learning benefit action generation in FastWAM? My understanding is that FastWAM does not explicitly predict future videos during inference. Instead, ActionDiT attends to only the current observation and task instruction's VideoDiT features through layer-wise K/V caches. Moreover, DEWO introduces success and failure conditions simply by appending outcome labels to the original task prompt. Since the VideoDiT, ActionDiT, and MoT backbones are frozen, with only text cross-attention K/V LoRA adapters updated, I hypothesize that continued video prediction learning improves action generation only by modifying the outcome-conditioned visual representations passed to the ActionDiT, rather than directly updating the underlying visual encoder or its dynamics modeling capabilities. Is this interpretation correct?
  2. Could the authors provide more implementation details on how RECAP and DSRL are adapted to WAMs?
4 comments
AuthorsOpenReview ↗

Response to Reviewer djU2 (1/3)

W1. Methodological contribution and novelty

We appreciate the reviewer's recognition of DEWO's motivation and practical value. Prior research on conditional learning, value estimation, and CFG provides the foundations for our technical choices. DEWO's central methodological contribution lies in connecting the probing of the current policy's local capability boundary with the acquisition and learning of ITP experience.

The components are connected through the shared outcome-based learning signal from ITP experience. Successful and failed continuations provide supervision for learning their physical consequences, while successful actions make feasible behavior accessible through an explicit success condition. The same replay outcomes supervise the value head, linking this experience to an online progress estimate that determines when CFG strengthens the learned success condition. The components follow one design logic: learning from ITP experience and selectively using that learning during execution.

W2&W3. Design choices and broader experimental validation

Selection settings and sensitivity. We empirically selected step 96 (3.2 s at 30 Hz) to focus probing around the transition from initial approach to physical interaction. The 24-step interval matches one executed action chunk (0.8 s), while ten continuations per anchor provide recoverability estimates in increments of 0.1, balancing sampling precision against interaction cost. We use the same configuration across all five simulation tasks. Among successful continuations, we select the shortest to favor compact task completion. We will add a sensitivity analysis of the query configuration and clarify continuation selection in the revised appendix.

Experience selection and broader task coverage. Figure 4c compares critical experience with full trajectories under matched data sizes and training budgets. We extend the evaluation to four simulation tasks using OpenWAM.

Post-training depends partly on the starting policy's ability to produce successful alternatives. OpenWAM's additional large-scale pretraining may make these alternatives accessible in more states, making it a useful complementary backbone beyond FACT and FastWAM.

The four augmented-data conditions share the same initialization, successful-action supervision, visual-supervision quantity, and training-update budget. Additional successful or failed trajectories contribute visual supervision only, without introducing extra action targets.

Table 1. Success rates (%, ↑) with OpenWAM; 50 trials per task and condition.

Training variantFold GlassesPick BucketPinch TongsWater PlantFour-task avg.
Original-data OpenWAM (reference)7486809283.0
Additional successful videos7694848885.5
Additional failed videos8094947485.5
Short ITP windows8096888086.0
DEWO (OpenWAM)8496949291.5

This supplementary comparison covers four tasks; Hammer Nail is omitted because its results vary across nearby checkpoints.

DEWO achieves 91.5% average success, exceeding both additional-video controls by 6.0 percentage points and short ITP windows by 5.5 points. These matched-budget comparisons support the complete ITP-centered experience selection and learning approach beyond adding successful or failed videos alone.

AuthorsOpenReview ↗

Response to Reviewer djU2 (2/3)

W2&W3. Design choices and broader experimental validation (continued)

Failure-localized experience and anchor selection. We additionally compare ITP-based DEWO with two controls using GPT-annotated failure times: fine-tuning on longer failed segments around those times, and applying DEWO to continuations sampled from anchors 2 seconds before the annotated failures.

Table 2. Success rates (%, ↑) on Fold Glasses using OpenWAM; 50 trials per condition.

Training variantSuccess rate (%)
Fine-tuning on GPT-localized failure segments80.0
DEWO with GPT-localized failure-point anchors80.0
DEWO with ITP anchors84.0

ITP-based DEWO achieves a 4.0-percentage-point improvement over DEWO with failure-point anchors, providing additional support for recoverability-based experience selection.

Value-gated guidance. Discounted terminal-success targets motivate tracking increases in predicted value as execution progresses, with stalled progress triggering guidance. Figure 4d already compares no guidance, continuous guidance, and value-gated guidance.

We additionally compare progress-based gating with random and fixed-schedule alternatives on Fold Glasses, using 50 trials per condition and a single-trigger budget for all guided variants.

Table 3. Guidance timing on Fold Glasses; success rates (%, ↑).

Gating strategyNo guidanceRandom gatingFixed-schedule gatingProgress-based gating (DEWO)
Success rate (%)80.080.082.084.0

W4. Modest improvements and experimental variability

We agree that the average gains on FastWAM and IDM are modest. Task-level gains vary: DEWO improves Fold Glasses by 5.3 and 4.0 percentage points on FastWAM and IDM, respectively, and Water Plant by 2.0 and 10.7 points. Pick Bucket offers less headroom on FastWAM (92.0% → 93.3%). In Hammer Nail, the policy can repeatedly strike and retry after grasping the hammer. Success may therefore depend less on a single decisive interaction, potentially making turning-point experience less informative. This is a possible explanation for the lack of improvement on FastWAM and IDM.

AuthorsOpenReview ↗

Response to Reviewer djU2 (3/3)

W5. Physical experience acquisition, scalability, and human effort

For long-horizon tasks, deformable objects, and complex contacts, final success alone does not reveal where progress breaks down. Assessing intermediate stages and recording their outcomes are also important for understanding the policy's capabilities. Identifying and documenting these stages is itself a challenge for such assessment. DEWO connects this assessment effort to post-training by using the resulting successful and failed experience to learn physical consequences and the success condition used for guidance.

Learning from these records does not require successful and failed continuations to start from an exactly identical physical state. Repeated trials with comparable task-relevant layouts and interaction stages can supply both outcomes, with each continuation supervised using its own observed context. Thus, approximately restored or repeated evaluation trials can provide training experience even when exact physical replay is impractical. We will report human inspection and restoration time, together with robot interaction costs, in the revised appendix.

Q1. How continued video prediction benefits FastWAM action generation

Your interpretation of FastWAM's inference pathway is correct: ActionDiT attends to current-observation VideoDiT features through layer-wise K/V caches, without generating future videos at inference. Video supervision affects action generation through adapted representations.

However, frozen backbone parameters do not imply an unchanged predictive function. The video prediction loss updates VideoDiT's text cross-attention K/V LoRA adapters, changing its visual hidden representations and future-video predictions. When these adapters are active at inference, they also affect the current-observation representations supplied to ActionDiT.

“Continued world-model learning” refers to optimizing this adapted predictive function against observed physical futures. Figure 4 reports reduced held-out video prediction loss alongside improved control as visual supervision is added. We will clarify this scope in the revised manuscript.

Q2. Adapting RECAP and DSRL to WAMs

The reported FastWAM comparisons share the same initial checkpoint. RECAP uses the same collected deployment data as DEWO, while DSRL obtains experience through online interaction. To make the adaptation explicit, we summarize our RLinf-based reference reimplementation:

RECAP. We retain the return-estimation, value-learning, advantage-labeling, and advantage-conditioned policy-learning pipeline. A value head estimates returns from frozen current-observation WAM features, and N-step advantages determine positive-advantage labels. We optimize the WAM's native action flow-matching objective by appending the positive-advantage condition to the task text, with condition dropout to train the base branch. The pretrained backbone remains frozen, while text cross-attention K/V LoRA adapters are updated. At inference, CFG combines base and positive-advantage predictions at each denoising step. Advantage labels are distinct from terminal success labels, so actions from failed trajectories are not categorically excluded. This baseline does not include DEWO's future-video supervision or value-based guidance gate.

DSRL. We freeze the WAM and train a SAC policy in its initial action-noise space. The noise policy generates initial noise from current-observation features, and the WAM converts it into an executable action chunk through its existing denoising process. Replay stores sampled noise, resulting rewards, and next observations; the critic evaluates the noise. Only the noise policy, critics, and entropy temperature are optimized, with no gradient through the WAM, preserving DSRL's principle of adjusting input noise for a frozen generative policy.

Thank you for the detailed responses. I raised my score to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal Period

Dear Area Chair and Reviewers,

Thank you for your time, careful reviews, and constructive suggestions. Your feedback helped us clarify DEWO’s contribution and identify the comparisons that would make the evaluation stronger.

Contribution and initial feedback

DEWO probes the current policy’s local capability boundary to identify interaction turning points (ITPs) and collect experience around them. The collected continuations provide post-training supervision, and their outcomes train the value head that gates guidance. This design lets DEWO learn from ITP experience and selectively apply what it learns during execution.

In the initial reviews, djU2 highlighted the practical value of adapting from deployment experience, 51QV and zF9Y highlighted the motivation for learning from failed executions, and jMQs noted the gains on FACT and in real-robot evaluation.

Experiments and clarifications during rebuttal

  • Visual supervision and attribution. We added video-augmentation controls under matched training budgets across four OpenWAM tasks. Full DEWO with selective guidance averaged 91.5% success, compared with 85.5% for each unguided video control. The two video controls differed by task. These results strengthen the evidence for the full method and leave open how much of the gain comes from failed-video supervision alone. (51QV Q1, zF9Y Q2)

  • Experience selection and guidance. We compared ITP anchors with anchors placed before visible failures and tested progress-gated guidance against simpler timing rules, SFT resampling, and random perturbation. The results favored ITP selection and progress-gated guidance on Fold Glasses. We also clarified hyperparameter selection and how RECAP and DSRL were adapted. (jMQs Q1–Q3, 51QV Q2, zF9Y Q3, djU2 Q2)

  • Real-robot adaptation. We added SFT controls trained on the same collected data on the Sharpa Tennis Ball task. DEWO outperformed SFT after two rounds on both tested backbones. We also clarified that the original full-grid study primarily showed reliability gains in regions where the initial policy sometimes succeeded. (jMQs Q1, 51QV Q4)

  • Robustness and collection cost. We reported evaluation-set uncertainty and simulation replay costs. The smaller gains still need validation across independent training seeds. (djU2 W4–W5, 51QV Q3 and Q5, zF9Y Q1)

Discussion and reviewer feedback

djU2 and zF9Y raised their ratings to 3. 51QV retained a rating of 2 and said the rebuttal moved their assessment in a positive direction, describing it as closer to 2.5. jMQs remains at 2 and has not posted a follow-up. We added targeted experiments for Q1 and Q3, which jMQs highlighted as important to strengthening the conclusions. Several of their other concerns also arose in other reviews, and follow-up comments indicated that the new experiments and clarifications addressed some of those shared concerns.

We will incorporate the new results and add the Decision Diffuser citation in the revision. The appendix will document human effort and robot interaction costs and, at zF9Y’s suggestion, discuss how recovery trials compare with collecting additional demonstrations or human corrections under the same collection budget.

Thank you for considering our submission.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.