acceptodds
Under review as a conference paper at ICLR 2027

Learning from What Goes Wrong: Failure-Aware World Models for Faithful Robot Policy Evaluation

Abstract

Video world models offer a scalable proxy for robot policy evaluation, but successonly training can overestimate policy success when the supplied actions lead to failure. Simply adding failure examples is still insufficient, because the failure space is broad and post-failure object dynamics are difficult to model plausibly, often leading to unrealistic motion or appearance changes that can mislead downstream VLM-based judges. We present a failure-aware framework that combines simulator-verified, scene-matched success–failure trajectories with Semantic-Aware Feedback Forcing (SAFF). SAFF uses feedback and semanticmotion signals to prioritize interaction-relevant prediction errors without changing the flow-matching target or adding auxiliary inference models. We also introduce a six-dimensional benchmark with confidence-aware trajectory evaluation. On simulation and real-world video benchmarks, our model scores 73.17 and 73.31 out of 100, surpassing the strongest baseline by 5.32 and 21.21 points, respectively. Across five policies in each setting, world-model–VLM success-rate estimates achieve Pearson correlations of 0.928 and 0.942 with execution-based rates, while reducing mean absolute error relative to Ctrl-World from 13.0 to 7.0 and from 16.16 to 8.26 percentage points, respectively. These results demonstrate improved prediction fidelity and closer agreement with execution-based policy performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
world model, policy evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KPTVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies video world models for robot policy evaluation when supplied actions can lead to failure. It constructs simulator-verified, scene-matched success-failure trajectories through controlled action perturbations and trains a Wan2.2 video model with the KVAF action representation from EA-WM. Its main modeling contribution, SAFF, uses feedback-derived correction signals and semantic-motion gradients to construct detached weights for the flow-matching loss. It also introduces a six-dimensional fidelity benchmark with confidence and missing-data rules. Experiments report improved video prediction and lower policy success-rate estimation errors in simulation and on real robots.

Critical strengths
  1. The simulator-verified failure corpus is a useful practical contribution. Same-scene success–failure pairs make controlled action deviations and their consequences traceable (Section 3.1; Appendix A.1).

  2. The experiments support improved video prediction and policy success-rate estimation in simulation and on real robots. The simulation ablation supports SAFF over standard flow matching, with auxiliary computations used only during training (Table 5; Tables 15-16).

Critical weaknesses
  1. The evidence for predicting specific failure consequences needs strengthening. The controlled corpus and aggregate results are useful, and Figures 6-9 illustrate improved appearance and post-contact motion. However, these examples do not clearly connect a named action perturbation, its observed failure consequences, and the model's prediction. Representative simulation case studies would clarify which failure modes are reproduced faithfully and where prediction errors remain. This connection is central to the claim of faithful policy evaluation from controlled failure data.

  2. The collection and use of real-world failure data are unclear. Section 4.1 describes teleoperation on three tasks and synchronized video/state recording, but does not specify how failed trajectories were obtained or their composition in world-model training and evaluation. The real-world results are encouraging, yet this missing protocol makes it difficult to assess failure coverage and the relationship between collected failures and evaluated policy rollouts. The detailed simulation failure-construction protocol does not resolve this real-world data question.

  3. The confidence-aware trajectory score needs independent validation. The appendix provides detailed support and missing-data rules, including abstention when GT evidence is insufficient. For supported track pairs, Equation (19) adds a penalty when mean paired SAM 3 confidence is below 0.600.60. Such low confidence could reflect a distorted prediction or difficulty tracking an otherwise faithful prediction, for example under partial occlusion. The penalty may therefore mix prediction error with tracking difficulty. The paper does not show whether it improves agreement with independently assessed trajectory fidelity. Better policy success-rate estimates support the full system, but do not validate this specific scoring choice.

Questions for authors
  1. Please provide representative simulation cases connecting a specific action perturbation, its observed failure consequences, and the model's prediction. Which failure modes are reproduced faithfully, and where do prediction errors remain? Existing rollouts could help clarify what is learned beyond the aggregate gains.

  2. How were real-world failed trajectories collected: deliberate teleoperation errors, naturally occurring failures, policy rollouts, or another procedure? Please clarify the success/failure composition of world-model training and evaluation data and its relationship to the evaluated policy rollouts.

  3. For supported track pairs, do the low-confidence penalties improve agreement with independently assessed trajectory fidelity? How do you distinguish low confidence caused by prediction distortions from difficult tracking of an otherwise faithful prediction? Existing cases or other validation evidence would help justify the scoring choice.

  4. Please provide representative videos of successful and failed executions, showing GT, your method, and Ctrl-World side by side. The sampled frames in Figures 6-9 are useful; videos would help assess the continuity and plausibility of contact, release, and subsequent object motion. If videos were uploaded separately, please indicate where they can be accessed.

4 comments

We thank the reviewer for recognizing the practical value of our simulator-verified failure corpus and the improvements in video prediction and policy success-rate estimation. We appreciate the suggestions to connect specific action perturbations to their predicted consequences, clarify the real-world failure-data protocol, validate the confidence-aware trajectory score, and provide comparative videos. Below, we address these points individually with annotated case studies, protocol details, scoring analyses, and side-by-side videos.

1. Perturbations and their predicted consequences

Thank you for the suggestion. We randomly selected three simulation case studies from existing rollouts, explicitly connecting the executed action perturbation, its observed consequence in GT, and the corresponding model prediction. The cases use the epoch-13 SAFF checkpoint. Intervention frames below refer to the aligned 81-frame sequences, indexed from 0 to 80.

1. Grasp-position offset: unsuccessful cube pickup (intervention frame 28). We offset the grasp target when placing a cube onto a stand. In GT, the offset contact tilts the cube without securing a grasp; the empty gripper moves away, leaving the cube on the table and the stand empty. Our prediction reproduces this sequence. The remaining errors concern the cube's contact pose, rotation, and local geometry. Ctrl-World exhibits cube deformation and motion toward the image boundary, with the cube no longer visible at the end.

2. Premature release: the object separates before cabinet placement (intervention frame 56). We open the gripper early while transporting a toy car. In GT, separation becomes visible around frame 59: the car returns to the table while the empty gripper continues toward the drawer. Our prediction reproduces this separation and subsequent object–gripper motion. Ctrl-World also captures the broad release outcome.

3. Suppressed release: failed block handover (intervention frame 54). We override the planned release so that the source-side gripper remains closed. In GT, the block returns with that gripper, leaving the receiving gripper empty. Our prediction preserves this return direction and object–gripper association, whereas Ctrl-World associates the block with the receiving side in the later frames.

The anonymous video page provides side-by-side GT/Ours/Ctrl-World playback for all three cases, including annotated full clips, half-speed clips around and after the intervention, and versions without event annotations.

2. Real-world failure collection and composition

Thank you for pointing out the missing collection details. We clarify the sources of real-world failures, the training and evaluation composition, and their relationship to policy evaluation.

Collection procedure. Successful trajectories come from teleoperated demonstrations, while failed trajectories come from naturally occurring failures during autonomous executions of a fully trained π0.5 policy on three tasks: fruit placement into a black box, mug hanging, and duck placement onto a shelf. A human initializes the environment; execution then proceeds autonomously without takeover or injected errors. Success/failure labels are assigned manually. Videos and robot states are recorded synchronously at 30 Hz.

World-model training and video evaluation. The unique training set contains 1,076 successful and 527 failed recordings. We balance the training mixture across both tasks and outcomes through stratified repeated sampling: each task contributes 618 successful and 618 failed entries per epoch, giving 3,708 training entries with a 1:1 success/failure ratio. The composition is:

TaskUnique training videos (success / failure)Training entries per epoch (success / failure)Held-out video-test recordings (success / failure)
Fruit placement171 / 271618 / 61829 / 30
Mug hanging287 / 190618 / 61842 / 21
Duck placement618 / 66618 / 61874 / 10
Total1,076 / 5271,854 / 1,854145 / 61

The 206-video prediction test set uses unique recordings without repeated sampling. Splitting is performed by collection groups before training resampling, and we verified that training and this test set share neither source recordings nor collection groups. RGB and KVAF use the same temporal sampling.

Relationship to policy evaluation. Failure collection uses a fully trained π0.5 policy, while the policy-evaluation experiment assesses five π0.5 variants trained on 10%, 20%, 30%, 50%, and 100% of the policy-training data. These percentages refer to policy training; the world-model training mixture follows the task and outcome balance above. The video-prediction test evaluates fidelity against recorded futures, while policy evaluation compares world-model–VLM success-rate estimates with execution-based rates.

3. Independent validation of the confidence penalty

Thank you for raising this concern. We added independent trajectory-fidelity ratings and an ablation that isolates the explicit low-confidence penalty.

Independent reference and controlled comparison. Three additional evaluators independently rated 100 existing GT–SAFF video pairs on a 1–5 trajectory-fidelity scale, without prior access to automatic scores or other evaluators' ratings. Three videos have no available automatic trajectory score for either role under the original support rules; the remaining 97 videos cover 50 tasks. Pairwise inter-rater Spearman correlations are 0.9024–0.9601.

We compare the original setting, βconf=0.06\beta_{\mathrm{conf}}=0.06, with βconf=0\beta_{\mathrm{conf}}=0, keeping tracking, instance matching, support rules, confidence-weighted localization error, and missing-observation penalties fixed. The Spearman correlations with human ratings are:

Human referenceWithout explicit penaltyWith explicit penaltyΔρ (with − without)95% CI for Δρ
Evaluator 10.83500.8355+0.00046[−0.00656, 0.00692]
Evaluator 20.79940.7964−0.00303[−0.01214, 0.00448]
Evaluator 30.75150.7485−0.00299[−0.01204, 0.00444]
Per-video mean rating (supplementary)0.80560.8036−0.00208[−0.01034, 0.00498]

Both score variants correlate strongly with human ratings, with only small changes after adding the explicit term. Confidence intervals use 10,000 paired task-cluster bootstrap resamples.

Why the two score variants are close. Both variants retain confidence-weighted localization error and missing-observation penalties; the ablation removes only the additional low-confidence term. This term affects 18 of the 97 videos, and the Spearman correlation between the two automatic score variants is 0.9984. Their nearly identical rankings leave limited room for differences in correlation with the same human ratings: for these fixed scores, the absolute difference is bounded by 0.0571. In the 45 videos with supported arm and object tracks, the rankings are identical, so the human correlations are identical as well.

Distinguishing prediction error from tracking difficulty. The paired confidence uses the lower GT/prediction detection confidence, so a low value can also originate on the GT side. We inspected selected GT/prediction frames and their segmentation masks to check whether trajectory discrepancies arose from the visible motion or from track extraction. This identified a cross-arm instance mismatch and a segmentation-drift case. Both cases received no explicit confidence penalty, and their original scores remain in the analysis. These checks show why visible motion, instance identity, and segmentation masks need to be considered together when interpreting low confidence.

Effect on the reported scores. Across all 315 SAFF predictions, the explicit penalty affects 58 videos. Arm and object scores are first averaged separately and combined with weights of 0.4/0.6; the resulting trajectory dimension contributes one sixth of the final benchmark score. Removing only the explicit penalty changes the trajectory dimension from 65.62 to 65.81 and the overall score from 73.17 to 73.20, a difference of about 0.03 points on the 0–100 scale.

4. Full videos of successful and failed executions

Thank you for the suggestion. We provide complete videos for the cube-placement scene shown in Figure 6, covering successful and failed GT executions, each with side-by-side GT/Ours/Ctrl-World comparisons:

The full videos retain all 81 aligned frames, allowing inspection of contact, separation, and subsequent object motion throughout the execution. For the failed execution, we also provide an annotated version marking the intervention and subsequent events, together with half-speed clips around and after the intervention.

Both pages support direct browser playback. Select “Without event annotations” to view the full comparison without event annotations; no ZIP download or login is required. The video index also includes the premature-release and suppressed-release cases discussed in our response to Question 1.

Reviewer_pNYsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies success hallucinations in video world models used for robot policy evaluation. It constructs paired successful and failed trajectories from identical initial scenes through action perturbations verified in simulation. SAFF uses feedback errors and semantic motion signals to reweight prediction losses during training, without auxiliary models at inference. Evaluation combines a six dimensional video benchmark with VLM judgments of generated rollouts. Experiments in simulation and three physical tasks report improved video fidelity and closer agreement with actual policy success rates.

Critical strengths
  1. Successful and failed trajectories start from identical scenes, with action interventions and resulting failures verified in simulation. This provides controlled examples for learning how changes in actions affect task outcomes.
  2. Across five policies, the full method reduces simulation success rate MAE from 43.0 to 7.0 percentage points relative to training on successes alone. Its mean estimate of 47.6% is closer to the actual 43%. SAFF achieves these improvements through training loss reweighting without auxiliary models at inference.
Critical weaknesses
  1. Faithful failure prediction remains insufficiently validated. Aggregate success rates can hide false successes and false failures that cancel out, yet no confusion matrix is reported. The lenient VLM judge also lacks reported independent validation. Simulation Spearman correlation remains 0.90 across configurations, and raw object trajectory errors do not consistently improve, limiting evidence for better ranking and interaction dynamics.
  2. The separate contributions of scene pairing and SAFF remain unclear. Controls using ordinary failure data of equal size, individual SAFF components, and simple error weighting are missing. Evaluation covers five π0.5 variants trained on different amounts of data and three physical tasks. Unseen failure types or combinations are not explicitly tested, and the physical experiments lack a SAFF ablation.
Questions for authors
  1. How often are actual failures predicted as successes? Could you validate the VLM judge on actual execution videos against execution labels to help distinguish generation errors from judging errors?
  2. Can you compare paired data with ordinary failure data of equal size, ablate individual SAFF components, and include simple error weighting? An experiment without SAFF on physical tasks would also clarify its independent contribution.
  3. Do success rate estimation and ranking remain reliable for unseen failure types, failure combinations, and different policy architectures? Can you quantify interaction errors after the action intervention, beyond aggregate video scores?
4 comments

We thank the reviewer for the constructive assessment and for recognizing the value of our controlled success–failure data and training-time loss reweighting. We appreciate the suggestions on evaluating individual failures, validating the visual judge, separating the contributions of data construction and SAFF, and testing generalization. Below, we address these points individually and present the additional analyses and experiments.

1. False successes and matched judge validation

Thank you for highlighting the need for evaluation at the level of individual executions. We added a confusion matrix for generated videos and tested the same visual judge on original execution videos.

False successes and false failures. We evaluated all 315 existing SAFF generations, corresponding to 50 successful and 265 failed executions. GPT-5.6 judged each video using the task description and 17 fixed frames, with execution labels and failure annotations withheld from the judge. Taking success as the positive class, the confusion matrix is:

Execution labelGenerated video judged successfulGenerated video judged failed
Success47 (TP)3 (FN)
Failure68 (FP)197 (TN)

The judge-based false-success rate is 68/265 = 25.66%, and the false-failure rate is 3/50 = 6.00%. Overall outcome agreement is 244/315 = 77.46%, with balanced accuracy of 84.17%. Reporting both error directions makes individual failures visible alongside the aggregate success-rate estimates.

Judge validation on execution videos. We selected one successful and one randomly sampled failed execution from each of the 50 tasks, giving 100 GT videos with balanced outcomes. We applied the same judge, prompt, and 17-frame protocol to these original recordings:

Execution labelGT video judged successfulGT video judged failed
Success48 (TP)2 (FN)
Failure18 (FP)32 (TN)

Agreement with execution labels is 80%, with a false-success rate of 36% and a false-failure rate of 4%. Of the 20 disagreements, 18 are failed executions judged successful. This GT control reveals an optimistic bias relative to the simulator labels under the lenient visual criterion, which is an important source of disagreement in world-model-based success estimates.

2. Attribution to paired data and SAFF

Thank you for the suggestion. We distinguish the construction of the supervision data from the weighting applied during training.

Contribution of the paired data. Each failed trajectory retains its successful source episode and scene seed, together with the executed intervention and outcome checks. This organization provides successful and failed futures associated with the same initial scene. Our audit of 532 examples confirmed matching recorded initial robot states and camera parameters, with all execution markers consistent with the logged interventions. These properties make the action–outcome supervision controlled and traceable.

SAFF with the training data held fixed. The Standard FM and SAFF comparison uses the same training corpus and retains KVAF conditioning. Changing the training objective from uniform FM to SAFF raises the simulation benchmark score from 70.89 to 73.17. This comparison evaluates the weighting strategy on the same data.

Simple error-weighting control. We additionally trained a control with the same initialization, training split, KVAF conditioning, optimization schedule, and epoch-13 budget. Its detached weights are derived from FM residual magnitudes, using the same smoothing, percentile normalization, and weight range [1,2], without feedback or DINO features. We evaluate all three methods on the same 315-video test set, using the same six normalized dimensions and aggregation as Table 10. Dimension scores are on [0,1], and the two aggregate scores are on 0–100; higher is better.

MetricStandard FM (paper)Simple error weightingSAFF
Visual fidelity0.9012330.9081420.918170
Distributional quality (mapped FVD score)0.4912910.5144280.571555
Temporal consistency0.8527460.8520970.862126
Semantic consistency0.7778350.7781370.779410
Motion consistency0.5913470.5909710.602476
Trajectory consistency0.6390200.6302480.656227
Overall six-dimension score (0–100)70.891271.233773.1661
Five-dimension mean excluding FVD (0–100)75.243675.191976.3682

SAFF outperforms simple error weighting in all six dimensions, with a 1.93-point overall gain and a 1.18-point gain after excluding FVD. With the training corpus and conditioning held fixed, this comparison supports the added value of SAFF's weighting design over weighting by error magnitude alone.

3. Local interaction fidelity after intervention

Thank you for the suggestion. We added a temporally localized analysis of the motion following each action intervention.

Post-intervention evaluation. We evaluated all 265 failed executions in the video benchmark, covering 50 tasks and 13 failure modes. For each execution, let kk denote the recorded perturbation-execution frame. We first extract and match arm/object tracks using the original evaluation protocol, then restrict the matched trajectories to frames t>kt>k. We retain the original 2D trajectory metric, confidence and missing-track rules, and arm/object aggregation weights of 0.4/0.6. All methods are evaluated against the same GT instances and post-intervention windows.

MethodPost-intervention trajectory component ↑ (0–100)
Standard FM60.70
Ctrl-World57.51
SAFF62.92

The arm and object components are aggregated over 242 and 173 scored cases, respectively. SAFF improves the post-intervention trajectory component by 2.22 points over Standard FM and 5.41 points over Ctrl-World. This analysis focuses on motion after the recorded perturbation, preventing the pre-intervention portion from diluting the comparison.

We also provide annotated GT/Ours/Ctrl-World comparisons for grasp-position offset, premature release, and suppressed release, showing the intervention frame and subsequent object motion.

4. New policy outputs and an unseen goal-swap failure

Thank you for the suggestion. We added tests on outputs from two additional policies and on an unseen goal-swap failure.

New policy outputs. We evaluated published FastWAM and OpenVLA-OFT checkpoints on handover_block, which lies outside the five-task posttraining set. The policies execute naturally, without injected faults. From fixed pools of 116 executions per policy, we selected 20 cases using a predeclared outcome-stratified rule and froze the selections before world-model generation.

PolicyExecution successes / failuresSuccess recall ↑Failure recall ↑Agreement ↑Balanced accuracy ↑
FastWAM13 / 79/13 (69.23%)7/7 (100%)80%84.62%
OpenVLA-OFT5 / 152/5 (40%)15/15 (100%)85%70.00%

Both policies show partial outcome transfer, with lower success recall for OpenVLA-OFT. Matched execution-video controls, evaluated with the same judge and 17-frame protocol, achieve 85% agreement with simulator labels for each policy. All seven successes missed in generation are correctly recognized in the corresponding execution videos, identifying a generation limitation.

A separate pick_dual_bottles test with 20 rollouts per policy also reveals OpenVLA-OFT overestimation: simulator/generated-video success rates are 100%/100% for FastWAM and 5%/95% for OpenVLA-OFT. The matched OpenVLA-OFT execution videos are judged successful in 65% of cases, showing that judge disagreement also contributes in this task. These results leave reliable cross-policy success-rate calibration and ranking unresolved. The policies additionally differ in their native embodiments and camera inputs.

An unseen goal-swap failure. To test transfer to an unseen failure combination, we use RGB block ranking. The task still requires red–green–blue, but the executed actions swap the destinations of the red and blue blocks. The pick order, simulator dynamics, and original success criterion are unchanged. Training-manifest and constructor audits confirm that this coordinated goal swap is absent from the audited failure-training construction. We assess whether the model reproduces the specific wrong ordering caused by this new combination of action errors.

All 20 predetermined scenes succeed in paired clean controls and fail under the swap, ending in blue–green–red rather than the required red–green–blue order. We generated videos for the 20 swapped executions using the recorded actions. All 20 generated videos are judged failed. Review of the sampled frames finds discernible blue–green–red order in 17/20 cases; one has uncertain terminal visibility, and two no longer show the red block at the end. These results support conditional reproduction of this specific unseen failure combination.

Thank you for the additional experiments and clarifications. The new results strengthen the empirical evidence, and I am keeping my rating.

Reviewer_T2RWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The author propose a failure-aware training approach for video world models, with a failure data generation pipeline in simulation and a Semantic-Aware Feedback Forcing (SAFF) framework for training, along with a six-dimensional benchmark. They evaluate video-prediction fidelity in simulation and the real world against prior world models, and policy-evaluation accuracy on five π0.5 policies per setting, with ablations on SAFF and failure data.

Critical strengths
  • The problem of sensitivity of failed interactions in the world model is an important problem to tackle.
  • Real world experiment results on the whole system show an advantage than baseline approach, which contributes empirically to adapting pretrained world models to downstream use.
Critical weaknesses
  • W1: I have concerns about the novelty and validity of the proposed contributions. The claim to “couple mechanism-specific failure construction with training signals that prioritize learning the resulting interaction dynamics” is poorly established: SAFF uses none of the mechanism-specific information of each training trajectory (failure type, perturbation timestamp, or scene-matched pair etc.), even though the pipeline labels them, and only reweights the loss by generic DINOv3 appearance and motion discrepancies between predicted and ground-truth frames (plus a minor feedback term). The other two contributions, the controlled failure construction and the six-dimensional benchmark, lack experiments showing that they improve on very similar established work (AHA/FailGen; EWMBench, which is not cited) or on any other alternatives.

  • W2: SAFF’s own contribution is small. In the ablation, SAFF adds only 2.28 points (70.89 to 73.17), whereas adding KVAF, coming from a previous work, contributes about 93% of the gain over the variant without both (40.78 to 70.89). Most of SAFF’s improvement also comes from FVD, a video-quality metric, while the semantic score, which SAFF explicitly targets through DINOv3 features, is essentially unchanged (0.778 to 0.779). The usefulness of SAFF therefore needs further validation through more experiments.

Questions for authors
  • Q1: For W1, can you distinguish your failure construction approach with others by showing how it is essential for world model improvement?

  • Q2: Can you validate and compare with others your proposed benchmark using either world model's ranking's correlation between real world policy rankings, or human judgements as evaluation for generated videos?

7 comments

We thank the reviewer for recognizing the importance of modeling failed robot interactions and the empirical value of our real-world results. We appreciate the concerns about the novelty of our failure construction, the independent contribution of SAFF, and the validity of our benchmark. Below, we address these points individually, clarify the scope of our claims, and report additional experimental comparisons and human evaluations.

1. Connection between failure construction and SAFF

Thank you for raising this point. We clarify how the mechanism annotations enter training and how feedback participates in SAFF.

Mechanism information is supplied through text conditioning. In our simulation training, the prompt includes the task description and failure type. For example, an excerpt from a failure-training prompt is:

In a fixed robotic workspace, generate a rigid, physically consistent embodied robotic arm. The arm maintains high stability with no deformation and enters the frame to position large block, medium block, and small block centrally, ordered from largest to smallest.

[Outcome]: failure.
[Failure Type]: target_position_offset.
[Generation Requirement]: generate the failed execution and keep the failure visible.

Successful training entries use [Outcome]: success. and [Failure Type]: none. These fields are passed to the prediction network as text conditions, together with the initial observation and action conditions. Each execution provides its observed future video as the supervision target.

The connection to SAFF is through this conditional prediction problem. The construction pipeline supplies actions and their observed successful or failed consequences. SAFF evaluates the current conditioned prediction against its ground-truth future and allocates training weight using feedback-derived correction demand and semantic-motion sensitivity. The mechanism annotations therefore specify the conditional prediction task, while SAFF determines the relative training emphasis across spatiotemporal positions. This is the scope of the connection we intend; the annotations are not used as direct inputs to a category-specific weighting rule.

Feedback gates all additional weighting in SAFF. Regarding the description “a minor feedback term,” the feedback signal is not a small additive auxiliary loss. Under the reported setting, Eq. (5) becomes

Wi=sg⁡ ⁣[1+Ci(0.1+0.9Di)],W_i=\operatorname{sg}\!\left[1+C_i\left(0.1+0.9D_i\right)\right],

where Ci∈[0,1]C_i\in[0,1] is correction demand obtained from the endpoint vector–Jacobian product in Eq. (1), and Di∈[0,1]D_i\in[0,1] is semantic-motion gradient sensitivity projected into latent space. Thus, the feedback map CiC_i multiplies the entire additional weight:

Feedback CiC_iSemantic-motion DiD_iUnnormalized weight WiW_i
0Any value1
101.1
112

When Ci=0C_i=0, semantic sensitivity alone cannot increase the weight above one. The coefficient 0.1 is a semantic-gate floor that preserves correction evidence when semantic sensitivity is weak; it does not mean that feedback contributes only 10% of the loss. The final objective is a normalized weighted FM loss with the original velocity target unchanged. Neither feedback nor the DINO objective is added as a separate training loss, and neither is required at inference. These equations clarify the structural role of feedback; they do not by themselves quantify its empirical contribution.

2. Failure construction relative to AHA and FailGen

Thank you for the question. As acknowledged in Appendix A.1, our pipeline builds on AHA/FailGen's task-specific interventions, parameter sweeps, and simulator outcome checks. Our dataset contribution is a systematic and auditable construction protocol for action-conditioned video learning, with three concrete extensions.

1. Broader interaction coverage, including bimanual coordination. We organize failures by manipulation stage, covering grasp, transport, contact, placement, release, and inter-arm coordination. Relative to AHA/FailGen's published single-arm RLBench setting, our corpus explicitly includes bimanual_phase_shift and relative_pose_offset, with 2,000 retained examples of each in the release audit. For example, in a recorded block-handover execution, a 6-cm offset to the receiving gripper's move leaves the block on the table, whereas the matched-scene successful execution transports it toward the target. These examples extend the corpus to failures involving the relative actions of two manipulators.

2. Explicit construction and acceptance checks. We determine task–fault applicability before injecting a perturbation into the corresponding manipulation primitive. A rollout is retained only after checking that the intervention hook was activated, the fault was applied, the simulation remained valid, and the execution failed under the original task checker. Pre-intervention failures and injection artifacts are rejected. These checks distinguish an intended perturbation from an intervention that actually executed and produced a valid failed rollout.

3. Same-source branches with verified scene correspondence and execution records. Each retained failure is linked to its clean source episode and scene seed. Across 25,000 clean source identities, the retained release contains an average of 5.32 failure branches per source, organized with continuous video, robot states, and intervention-execution records.

We audited 532 examples spanning 50 tasks and 266 task–fault combinations. Their recorded initial robot states and camera parameters matched the linked clean executions. In a four-way first-frame retrieval test against three different-scene controls from the same task, the correct paired-source retrieval rates were:

CameraCorrect paired-source retrieval
Head532/532 (100%)
Front532/532 (100%)
Left87.78%
Right95.68%

All 532 execution markers also matched the logged move-end or gripper-onset events, linking each recorded intervention to its trajectory.

The world model retains the same input interface. Our contribution lies in how its action–video supervision is constructed, validated, and organized. Table 1 documents these construction choices, and the additional audits substantiate their implementation.

3. Incremental SAFF benefit and a simple weighting control

SAFF's incremental contribution. We compare Standard FM and SAFF with KVAF conditioning retained in both. SAFF raises the simulation benchmark score from 70.8912 to 73.1661, improving all six normalized dimensions. The FVD-based dimension accounts for 58.8% of this gain. Excluding it and equally averaging the remaining five dimensions gives 75.2436 for Standard FM and 76.3682 for SAFF, showing improvement across the other fidelity dimensions as well.

Comparison with simple error weighting. We added a control using the same initialization, training data, KVAF conditioning, optimization schedule, and epoch-13 training budget. Its detached weights are constructed from FM residual magnitudes using the same smoothing, percentile normalization, and weight range [1,2], without feedback or DINO features. We evaluate all three methods on the same 315-video test set, using the same six normalized dimensions and aggregation as Table 10. Dimension scores are on [0,1], and the two aggregate scores are on 0–100; higher is better.

MetricStandard FM (paper)Simple error weightingSAFF
Visual fidelity0.9012330.9081420.918170
Distributional quality (mapped FVD score)0.4912910.5144280.571555
Temporal consistency0.8527460.8520970.862126
Semantic consistency0.7778350.7781370.779410
Motion consistency0.5913470.5909710.602476
Trajectory consistency0.6390200.6302480.656227
Overall six-dimension score (0–100)70.891271.233773.1661
Five-dimension mean excluding FVD (0–100)75.243675.191976.3682

Simple error weighting raises the overall score to 71.2337, while its five-dimension mean excluding FVD is 75.1919, slightly below Standard FM's 75.2436. SAFF outperforms this control in all six dimensions, with a 1.93-point overall gain and a 1.18-point gain after excluding FVD. This comparison supports the added value of SAFF's weighting design beyond weighting by FM residual magnitude alone.

Interpretation of the semantic score. The mean semantic component changes only slightly, from 0.777835 to 0.779410. SAFF uses DINOv3 feature gradients to allocate training emphasis while retaining the original FM target; the benchmark measures semantic similarity with DINOv2. The observed gains span visual fidelity, motion, and trajectory consistency, alongside the small change in average semantic similarity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.