- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ForeAlign uses an action-conditioned world model to predict safety outcomes of candidate actions and update the action policy using DiffusionNFT. The predictor is frozen during policy training to prevent the policy from improving its scores by mis-predicting unsafe outcomes as safe. Experiments show improved safety and task performance across several WAMs on simulated and real robot tasks.
Critical strengths
- Separating the action policy from the frozen outcome predictor addresses the alignment faking problem.
- Experimental results show improved safety and task completion across various WAMs and tasks.
Critical weaknesses
- The safety probabilities do not appear to be calibrated. Figure 4(b) reports binary classification accuracy, but does not verify whether the assigned probabilities match observed safety event frequencies. Perhaps the safety probabilities can be calibrated with some held out safe and unsafe events using conformal prediction?
- It is unclear how reliable the world model prediction of unsafe outcomes is. Also, it is unclear how the WAMs were trained to enable them to imagine unsafe situations. Prior work, like "StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement", explicitly required training the base WAMs on unsafe data. How without being exposed to unsafe data related to a new task, the WAM can imagine unsafe rollouts reliably in this paper?
Questions for authors
- Clarification: What unsafe events occur in the AC-WM training data, and how frequently? Is failure coverage deliberately ensured?
- Validation: (a) Are the safety probabilities calibrated against actual execution outcomes? (b) How sensitive are results to failure coverage in world-model training?
5 comments
Thank you for the valuable review (2/3)
-
W2, Q1 World-model reliability and unsafe-data coverage
Reliability of predicted safety feedback. We would like to humbly clarify that Fig. 4(b), left, evaluates VLM safety judgments on both actual and AC-WM-predicted outcomes against ground-truth safety labels, assessing the evaluator and the combined prediction–evaluation pipeline. We further evaluate predicted futures on the unsafe subset of the same test dataset to testify the evaluation reliability on unsafe samples.
Table. Safety-classification accuracy (%). Unsafe-case accuracy measures unsafe-outcome recall.
Future observations Test subset Place Bread Pass Knife Ground-truth future Full dataset 94.77% 89.70% AC-WM-predicted future Full evaluation set 79.91% 73.12% AC-WM-predicted future Unsafe cases only 77.56% 70.13% As shown in the table,accuracy on unsafe cases is nearly equal to that on the full dataset, showing that the pipeline detects unsafe outcomes rather than achieving high overall accuracy primarily through safe cases. We also monitor safety-classification accuracy across ten alignment iterations. Moverover, the newly added Fig. 14 shows broadly stable accuracy as the policy evolves, supporting the continued reliability of the frozen AC-WM.
Failure coverage and training-data composition. Failure coverage is deliberately included during AC-WM adaptation. In simulation, we collect three sources: (i) original successful expert demonstrations, which are mostly physically safe but may contain semantic violations; (ii) base-policy rollouts with temporally consistent action perturbations, implemented as a linearly increasing bias starting at zero within each chunk; and (iii) replays of expert action trajectories in rearranged scenes, producing failure outcomes while preserving the original, unperturbed actions. We mix these sources at a ratio of 1:2:2, so 80% of the adaptation data come from deliberate failure-oriented collection. To this end, the training data contain both physical violations results from reset envrionment setup or added distrubunce, such as poking or collision, and semantic violations are naturally exists within the support of pretrained policy, such as place food in contaminated contrainer, occurring ~42% the collected trajectories Place Food.
In real-world experiments, we use original demonstrations, self-policy rollouts, and scene-rearranged replays, but without action perturbations to avoid unexpected behaviors. We adapt each AC-WM on these task-specific data and freeze it during alignment. We have documented the data recipes and action-conditioning modifications in Appx. E.5 and F.4.
Failure prediction on new tasks. A WAM’s ability to predict unsafe outcomes on a new task depends on its pretraining experience and generalization capability. Recent work such as FACT [2] incorporates failure rollouts into action-conditioned training and reduces success-biased future hallucinations, providing a promising foundation for transferring failure-prediction capabilities across tasks. Developing a broadly generalizable, failure-aware AC-WM is an important problem complementary to ForeAlign, which focuses on a general framework for anticipatory self-alignment across WAMs with different architectures and predictive capabilities. ForeAlign can directly benefit from advances in failure-aware modeling, as stronger generalization would reduce the task-specific data and adaptation budget needed for reliable safety feedback. Concretely, for the less capable WAMs evaluated in our paper, we include both safe and unsafe task-specific trajectories during AC-WM adaptation. Starting from pretrained WAMs, our task-specific AC-WM adaptation takes approximately 12 hours on 320 episodes using 8 A800 GPUs (96 GPU-hours). For context, GigaWorld-Policy reports 6,000 GPU-hours for multi-task pretraining [3]. Although the training scopes differ, this illustrates that adapting an existing WAN to AC-WM requires a substantially smaller compute budget than directly undertaking world model training.
Thank you for the valuable review (3/3)
-
Q2(b) Sensitivity to Failure Coverage.
Our original data mixture was chosen heuristically. To further assess its sensitivity, we vary the proportion of unsafe-oriented data from sources (ii) and (iii): expert-only training (0), half the original proportion (0.5p_{failure}=0.4), and the original proportion (p_{failure}=0.8). Here, p_{failure} denotes the share of data collected through failure-oriented procedures, rather than the observed unsafe-event frequency. We collect additional expert demonstrations to keep the total dataset size fixed when reducing this proportion, while retaining the same pretrained initialization, adaptation budget, and evaluation protocol.
Table. Effect of unsafe-oriented data proportion during AC-WM adaptation on safety-classification accuracy (%).
Task / Unsafe-oriented data proportion 0 0.4 0.8 (original) Place Bread 76.55% 79.10% 79.91% Pass Knife 57.64% 70.70% 73.12% Including unsafe-oriented data improves safety-classification accuracy on both tasks. Halving its proportion retains most of the benefit, whereas removing it causes a larger decline, particularly on Pass Knife. These results support deliberate failure coverage during AC-WM adaptation and demonstrate robustness to a moderate reduction in its proportion. We have added these results and their analysis to Appx. G.2.
References:
[1] Angelopoulos, Anastasios N., and Stephen Bates. "Conformal Prediction: A Gentle Introduction." Foundations and Trends in Machine Learning 16(4): 494–591, 2023.
[2] Peng, Quanquan, et al. "FACT: Failure-Aware Causal Training for World-Action Models." arXiv preprint arXiv:2608.10232, 2026.
[3] Ye, Angen, et al. "GigaWorld-Policy: An Efficient Action-Centered World–Action Model." arXiv preprint arXiv:2603.17240, 2026.
Thank you for the valuable review (1/3)
We sincerely appreciate your careful review and insightful feedback. Your comments have helped us examine the reliability of the safety evaluator more thoroughly and clarify how failure experience contributes to AC-WM adaptation.
We have added calibration and prediction-set evaluations, investigated sensitivity to failure-data coverage, and expanded the description of the adaptation procedure. Below, we address your concerns individually and explain the supporting evidence and changes to the manuscript.
W1 & Q2(a) Probability calibration
We thank the reviewer for highlighting the distinction between classification accuracy and probability calibration. To assess probability calibration, we compare safety estimates from eight VLM queries per context–trajectory pair with execution-grounded labels. Both tasks use 100 held-out evaluation candidates, with 412 calibration candidates for Place Bread and 250 for Pass Knife. Pass Knife uses a sigmoid probability mapping and a classification threshold fitted on its calibration subset. Imputed judgments are excluded, with scores computed from the remaining observed calls. Moreover, we also evaluate conformal prediction, which complements probability calibration by constructing prediction sets that retain both labels when a judgment is uncertain. Following [1], we construct sets using scores ∣p^−y∣ at a 90% target coverage.
Table. Calibration and empirical prediction-set results on held-out candidates.
As shown in the table, The low calibration errors on both tasks indicate that the evaluated safety probabilities broadly agree with observed outcomes. Prediction sets provide single-label judgments for most candidates with higher accuracy than the overall decisions. Empirical coverage exceeds the target on Place Bread but falls below it on Pass Knife. The Pass Knife analysis reuses the calibration subset for fitting the probability mapping and selecting prediction-set thresholds, so these results do not carry an independent split-conformal coverage guarantee.
Despite this analysis validates the VLM’s safety evaluation, our extended theory further clarifies the connection between evaluator reliability and the group-relative implementation. Eq. (10) uses relative training weights rather than calibrated execution-safety probabilities. The added analysis in Appx. A.3 shows that these weights preserve the ideal NFT update direction under the stated assumptions, and that ideal reweighting improves expected safety when evaluator acceptance is positively correlated with true safety. We have added the evaluation protocol and full results to Appx. G.4 (Table 14).