acceptodds
Under review as a conference paper at ICLR 2027

ForeAlign: Anticipatory Self-Alignment of World Action Models for Safer Manipulation

Abstract

World Action Models (WAMs) advance robotic manipulation by jointly predicting actions and their consequences, but greater autonomy also expands the range of potential hazards. Existing safety approaches rely largely on predefined specifications and costly environmental interaction, limiting safety coverage and making risk discovery expensive. We propose ForeAlign, a framework for Anticipatory Self-Alignment that iteratively probes imagined futures, incorporates safety feedback into policy updates for continuous improvement. It unifies heterogeneous safety requirements under a general safety distribution and extends Diffusion Negative-aware Fine-Tuning (DiffusionNFT) into a monotonic improvement operator with theoretically bounded policy shift, mitigating the safety-performance trade-off from catastrophic forgetting. To counter alignment faking, ForeAlign decouples the WAM into a cascade of an action policy and an independently parameterized action-conditioned world model, directing safety supervision to action generation. Across simulated and real-world embodied manipulation benchmarks, ForeAlign improves safe rate by 35.6% and task success rate by 24.6% on average.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Safety Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_8AwWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ForeAlign uses an action-conditioned world model to predict safety outcomes of candidate actions and update the action policy using DiffusionNFT. The predictor is frozen during policy training to prevent the policy from improving its scores by mis-predicting unsafe outcomes as safe. Experiments show improved safety and task performance across several WAMs on simulated and real robot tasks.

Critical strengths
  • Separating the action policy from the frozen outcome predictor addresses the alignment faking problem.
  • Experimental results show improved safety and task completion across various WAMs and tasks.
Critical weaknesses
  • The safety probabilities do not appear to be calibrated. Figure 4(b) reports binary classification accuracy, but does not verify whether the assigned probabilities match observed safety event frequencies. Perhaps the safety probabilities can be calibrated with some held out safe and unsafe events using conformal prediction?
  • It is unclear how reliable the world model prediction of unsafe outcomes is. Also, it is unclear how the WAMs were trained to enable them to imagine unsafe situations. Prior work, like "StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement", explicitly required training the base WAMs on unsafe data. How without being exposed to unsafe data related to a new task, the WAM can imagine unsafe rollouts reliably in this paper?
Questions for authors
  • Clarification: What unsafe events occur in the AC-WM training data, and how frequently? Is failure coverage deliberately ensured?
  • Validation: (a) Are the safety probabilities calibrated against actual execution outcomes? (b) How sensitive are results to failure coverage in world-model training?
5 comments
AuthorsOpenReview ↗

Thank you for the valuable review (1/3)

We sincerely appreciate your careful review and insightful feedback. Your comments have helped us examine the reliability of the safety evaluator more thoroughly and clarify how failure experience contributes to AC-WM adaptation.

We have added calibration and prediction-set evaluations, investigated sensitivity to failure-data coverage, and expanded the description of the adaptation procedure. Below, we address your concerns individually and explain the supporting evidence and changes to the manuscript.

  • W1 & Q2(a) Probability calibration

    We thank the reviewer for highlighting the distinction between classification accuracy and probability calibration. To assess probability calibration, we compare safety estimates from eight VLM queries per context–trajectory pair with execution-grounded labels. Both tasks use 100 held-out evaluation candidates, with 412 calibration candidates for Place Bread and 250 for Pass Knife. Pass Knife uses a sigmoid probability mapping and a classification threshold fitted on its calibration subset. Imputed judgments are excluded, with scores computed from the remaining observed calls. Moreover, we also evaluate conformal prediction, which complements probability calibration by constructing prediction sets that retain both labels when a judgment is uncertain. Following [1], we construct sets using scores ∣p^−y∣|\hat p-y| at a 90% target coverage.

    Table. Calibration and empirical prediction-set results on held-out candidates.

    EvaluationMetricPlace BreadPass Knife
    ClassificationMean single-query accuracy75.75%69.35%
    ClassificationAggregated decision accuracy80.00%74.00%
    Probability calibrationECE (10 equal-width bins)0.08250.0572
    Probability calibrationBrier score0.14410.1741
    Conformal predictionEmpirical prediction-set coverage93.00%87.00%
    Conformal predictionSingle-label prediction rate69.00%71.00%
    Conformal predictionAccuracy among single-label predictions89.86%81.69%

    As shown in the table, The low calibration errors on both tasks indicate that the evaluated safety probabilities broadly agree with observed outcomes. Prediction sets provide single-label judgments for most candidates with higher accuracy than the overall decisions. Empirical coverage exceeds the target on Place Bread but falls below it on Pass Knife. The Pass Knife analysis reuses the calibration subset for fitting the probability mapping and selecting prediction-set thresholds, so these results do not carry an independent split-conformal coverage guarantee.

    Despite this analysis validates the VLM’s safety evaluation, our extended theory further clarifies the connection between evaluator reliability and the group-relative implementation. Eq. (10) uses relative training weights rather than calibrated execution-safety probabilities. The added analysis in Appx. A.3 shows that these weights preserve the ideal NFT update direction under the stated assumptions, and that ideal reweighting improves expected safety when evaluator acceptance is positively correlated with true safety. We have added the evaluation protocol and full results to Appx. G.4 (Table 14).

AuthorsOpenReview ↗

Thank you for the valuable review (2/3)

  • W2, Q1 World-model reliability and unsafe-data coverage

    Reliability of predicted safety feedback. We would like to humbly clarify that Fig. 4(b), left, evaluates VLM safety judgments on both actual and AC-WM-predicted outcomes against ground-truth safety labels, assessing the evaluator and the combined prediction–evaluation pipeline. We further evaluate predicted futures on the unsafe subset of the same test dataset to testify the evaluation reliability on unsafe samples.

    Table. Safety-classification accuracy (%). Unsafe-case accuracy measures unsafe-outcome recall.

    Future observationsTest subsetPlace BreadPass Knife
    Ground-truth futureFull dataset94.77%89.70%
    AC-WM-predicted futureFull evaluation set79.91%73.12%
    AC-WM-predicted futureUnsafe cases only77.56%70.13%

    As shown in the table,accuracy on unsafe cases is nearly equal to that on the full dataset, showing that the pipeline detects unsafe outcomes rather than achieving high overall accuracy primarily through safe cases. We also monitor safety-classification accuracy across ten alignment iterations. Moverover, the newly added Fig. 14 shows broadly stable accuracy as the policy evolves, supporting the continued reliability of the frozen AC-WM.

    Failure coverage and training-data composition. Failure coverage is deliberately included during AC-WM adaptation. In simulation, we collect three sources: (i) original successful expert demonstrations, which are mostly physically safe but may contain semantic violations; (ii) base-policy rollouts with temporally consistent action perturbations, implemented as a linearly increasing bias starting at zero within each chunk; and (iii) replays of expert action trajectories in rearranged scenes, producing failure outcomes while preserving the original, unperturbed actions. We mix these sources at a ratio of 1:2:2, so 80% of the adaptation data come from deliberate failure-oriented collection. To this end, the training data contain both physical violations results from reset envrionment setup or added distrubunce, such as poking or collision, and semantic violations are naturally exists within the support of pretrained policy, such as place food in contaminated contrainer, occurring ~42% the collected trajectories Place Food.

    In real-world experiments, we use original demonstrations, self-policy rollouts, and scene-rearranged replays, but without action perturbations to avoid unexpected behaviors. We adapt each AC-WM on these task-specific data and freeze it during alignment. We have documented the data recipes and action-conditioning modifications in Appx. E.5 and F.4.

    Failure prediction on new tasks. A WAM’s ability to predict unsafe outcomes on a new task depends on its pretraining experience and generalization capability. Recent work such as FACT [2] incorporates failure rollouts into action-conditioned training and reduces success-biased future hallucinations, providing a promising foundation for transferring failure-prediction capabilities across tasks. Developing a broadly generalizable, failure-aware AC-WM is an important problem complementary to ForeAlign, which focuses on a general framework for anticipatory self-alignment across WAMs with different architectures and predictive capabilities. ForeAlign can directly benefit from advances in failure-aware modeling, as stronger generalization would reduce the task-specific data and adaptation budget needed for reliable safety feedback. Concretely, for the less capable WAMs evaluated in our paper, we include both safe and unsafe task-specific trajectories during AC-WM adaptation. Starting from pretrained WAMs, our task-specific AC-WM adaptation takes approximately 12 hours on 320 episodes using 8 A800 GPUs (96 GPU-hours). For context, GigaWorld-Policy reports 6,000 GPU-hours for multi-task pretraining [3]. Although the training scopes differ, this illustrates that adapting an existing WAN to AC-WM requires a substantially smaller compute budget than directly undertaking world model training.

AuthorsOpenReview ↗

Thank you for the valuable review (3/3)

  • Q2(b) Sensitivity to Failure Coverage.

    Our original data mixture was chosen heuristically. To further assess its sensitivity, we vary the proportion of unsafe-oriented data from sources (ii) and (iii): expert-only training (0), half the original proportion (0.5p_{failure}=0.4), and the original proportion (p_{failure}=0.8). Here, p_{failure} denotes the share of data collected through failure-oriented procedures, rather than the observed unsafe-event frequency. We collect additional expert demonstrations to keep the total dataset size fixed when reducing this proportion, while retaining the same pretrained initialization, adaptation budget, and evaluation protocol.

    Table. Effect of unsafe-oriented data proportion during AC-WM adaptation on safety-classification accuracy (%).

    Task / Unsafe-oriented data proportion00.40.8 (original)
    Place Bread76.55%79.10%79.91%
    Pass Knife57.64%70.70%73.12%

    Including unsafe-oriented data improves safety-classification accuracy on both tasks. Halving its proportion retains most of the benefit, whereas removing it causes a larger decline, particularly on Pass Knife. These results support deliberate failure coverage during AC-WM adaptation and demonstrate robustness to a moderate reduction in its proportion. We have added these results and their analysis to Appx. G.2.

References:

[1] Angelopoulos, Anastasios N., and Stephen Bates. "Conformal Prediction: A Gentle Introduction." Foundations and Trends in Machine Learning 16(4): 494–591, 2023.

[2] Peng, Quanquan, et al. "FACT: Failure-Aware Causal Training for World-Action Models." arXiv preprint arXiv:2608.10232, 2026.

[3] Ye, Angen, et al. "GigaWorld-Policy: An Efficient Action-Centered World–Action Model." arXiv preprint arXiv:2603.17240, 2026.

Reviewer_7A2WOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ForeAlign, a framework for safety post-training of World Action Models. The method collects contexts using the current policy, samples candidate action chunks, predicts their consequences using an independently parameterized and frozen action-conditioned world model, and updates the action policy using DiffusionNFT. Safety feedback combines geometric constraints with VLM-based semantic judgments. The results show improvements in both safety and task performance.

Critical strengths
  1. The paper identifies an important failure mode when safety rewards are computed from a model’s own predicted outcomes: joint optimization may improve the apparent safety of predictions without improving the underlying actions. Using an independently parameterized, frozen AC-WM removes the direct route through which policy optimization could modify the predictor to obtain higher safety scores. The ablation without decoupled world modeling in Table 2 provides useful empirical support for this design.

  2. DiffusionNFT accommodates both positive and negative feedback without requiring tractable diffusion-policy likelihoods or differentiable safety evaluators. This is a useful match for heterogeneous feedback, including geometric constraints and binary VLM judgments. Anchoring updates to a reference policy is also well motivated when safety post-training does not include an auxiliary task reward.

  3. The evaluation goes beyond a single base policy or purely simulated experiments. Applying the framework to Lingbot-VA, Fast-WAM, and Cosmos3-Nano supports its architectural applicability. The real-world tasks include contact-sensitive manipulation and semantic hazards, such as food contamination and inappropriate plug insertion.

Critical weaknesses
  1. The temporal scope of safety assessment is insufficiently specified and evaluated. Equation (8) defines prediction over a future chunk, but the paper does not clearly report the actual prediction horizon, number of generated frames, or correspondence between action steps and visual observations for each model. The real-world configuration specifies an action chunk size of 32, but this does not establish how far into the future safety is assessed. Moreover, Appendix C.2 describes semantic evaluation using initial and final frames. Such evaluation can miss transient violations within a chunk, while chunk-local feedback can miss delayed hazards beyond the prediction horizon. Algorithm 1 does not clearly describe recursive multi-chunk imagination or a mechanism for propagating future safety costs to earlier decisions. This matters because the reported Safe Rate requires no violation throughout the entire executed episode. A horizon ablation and tasks with delayed or transient hazards would substantially strengthen the evaluation.

  2. Important implementation details are missing. The paper specifies an AC-WM training objective but does not sufficiently explain how each base WAM is converted into an action-conditioned predictor. For Lingbot-VA or Fastwam in particular, it is unclear how candidate actions are supplied to future-observation prediction, whether causal attention dependencies are changed, and which modules are copied, retained, or trained during AC-WM initialization. The context-sharing masks do not fully resolve these architectural questions. The policy’s task-conditioning interface, effective context window, AC-WM prediction configuration, and initialization-training budget also need clearer documentation. These omissions make faithful reproduction difficult.

  3. The computational cost of the full alignment loop is not reported. Each iteration involves environment rollouts, generation of multiple candidate futures, VLM evaluation, and policy optimization. However, the paper reports neither a runtime breakdown nor total wall-clock time, GPU-hours, VLM request counts, or API cost. The results support improved use of an environment-interaction budget, but do not establish end-to-end computational efficiency. Matching interaction and optimization budgets across methods is useful, yet it does not account for differences in world-model sampling and external evaluation costs.

Questions for authors
  1. For each base model, what are the action chunk length, predicted frame count, temporal sampling rate, and effective prediction duration? Does ForeAlign ever recursively imagine beyond one chunk?

  2. Does semantic scoring inspect intermediate frames, or only the initial and final frames? How does the method handle temporary violations or hazards that emerge after the prediction horizon?

  3. Can you report predicted versus executed safety for matched candidate actions under joint and decoupled alignment, including how this gap changes across iterations?

  4. With the same frozen AC-WM, candidate samples, and safety evaluator, how does DiffusionNFT compare with rejection fine-tuning and a matched alternative policy-update objective?

  5. The paper reports evaluation after five alignment iterations and lists epoch/inner-epoch settings in the appendices, but their correspondence to Algorithm 1 is unclear. Please specify the number of data-collection rounds and gradient updates used for each main result.

  6. Additionally, what task-conditioning inputs are provided to the action policy and AC-WM? Are policies conditioned on language instructions, goal images/videos, or trained separately for each task?

3 comments
AuthorsOpenReview ↗

Thank you for the valuable review (1/3)

We deeply appreciate the time and effort you invested in reviewing our manuscript. Your detailed comments and suggestions have been valuable in improving the clarity of our experimental setting and the completeness of our implementation details.

In response, we have expanded the implementation descriptions, reported the computational cost, and added comparisons of predicted and executed safety and policy-update objectives. Below, we provide detailed responses to your questions and summarize the relevant revisions.

  • W1, Q1 & Q2 Temporal scope of safety assessment

    We thank the reviewer for pointing this out. In simulation, interaction is quasi-static, so we specify horizons in discrete steps rather than FPS or duration. For real-world Fast-WAM, the action chunk contains 32 steps at 30 Hz (approximately 1.07 s); sampling visual observations every 4 action steps yields 8 future frames at 7.5 Hz. We summarize these hyper-parameters as follows:

    Setting / modelAction stepsFuture framesVisual intervalAction / visual rateDuration
    Simulation / Lingbot-VA3284------
    Simulation / Fast-WAM3284------
    Simulation / Cosmos3-Nano64164------
    Real world / Fast-WAM328430 Hz / 7.5 Hzapproximately 1.07 s

    ForeAlign conditions on histories from actual environment rollouts and does not recursively extend predictions beyond one chunk. For each history, the policy samples candidate action chunks, and the frozen AC-WM predicts one future observation chunk conditioned on the history and each candidate action. In the revised paper, we have expanded Table 5 with the action-chunk length, predicted-frame count, and action-to-visual sampling interval for each simulation backbone, and Table 9 with the corresponding real-world settings, including action and visual rates and prediction duration. We have also clarified the one-chunk prediction protocol and quasi-static simulation setting in Appx. E.5.

    For transient hazards, we acknowledge that the current semantic evaluator may miss violations occurring between the initial and final frames. As described in Appx. D.2, we use these two frames to reduce API token consumption. However, Fig. 4b supports the accuracy of this simplified evaluation against ground-truth safety labels for the investigated cases. Moreover, this design is one instantiation of ForeAlign's general safety-evaluation interface, which can also accept the entire video chunk to provide richer temporal evidence for detecting intermediate violations.

    For delayed hazards, we also agree that consequences emerging beyond the predicted chunk may go outside the current assessment horizon. But our framework can be extended to predict farther into the future and incorporate safety feedback on delayed consequences into policy updates to mitigate this issue. We have discuss these hazards and their handling in Appx. D.2.

  • W2 AC-WM implementation details

    We thank the reviewer for pointing this out and apologize for omitting these implementation details. For all backbones, AC-WM adaptation preserves the original history-context window and future-prediction window. We change the prediction task from generating future observation–action chunks to predicting future observations conditioned on the corresponding clean candidate action chunk. The AC-WM is initialized from the pretrained WAM, adapted on offline action–outcome trajectories, and frozen during policy alignment.

    As for the specific modifications for AC-WM adaptation, we retain Lingbot-VA's causal attention structure while modifying the attention mask so that each future observation chunk can attend to its corresponding clean action chunk without modifying its model architecture. For Fast-WAM, we enable its official action_conditioned configuration: a newly initialized linear layer projects clean candidate actions into latent embeddings, which are appended after the text embeddings as conditions. For Cosmos3-Nano, which uses a flow-matching generator with bidirectional attention, we switch to the framework's forward-dynamics mode: candidate actions are marked as conditioning and encoded by the pretrained action encoder without noise or timestep embedding, and only vision tokens undergo flow-matching and appear in the generation loss. For all three architectures, all modules except the VAE are fine-tuned on AC-WM data; the action encoder for Cosmos3-Nano is kept trainable consistent with the official policy training recipe. We have added a dedicated "Implementation details of adapting WAM for AC-WM" paragraph in Appx. E.5, which provides comprehensive documentation beyond the architectural modifications, including the data collection and construction pipeline, data recipe, and corresponding training settings. The corresponding configurations are also included in the released code.

AuthorsOpenReview ↗

Thank you for the valuable review (2/3)

  • W3 Computational cost

    We thank the reviewer for pointing this out and apologize for omitting the computational cost. ForeAlign primarily aims to reduce costly physical interactions by evaluating candidate actions through imagined consequences. We report the average cost per alignment iteration for Lingbot-VA using eight RTX 5090 GPUs. Each iteration takes 87.7 minutes, corresponding to approximately 14.6 hours over ten iterations.

    Table. Average computational cost per alignment iteration. GPU-hours equal wall-clock minutes × 8 / 60 for stages using GPU inference or training. API calls are approximate.

    StageWall-clock time (min)GPU-hoursAPI calls
    Environment rollouts50.26.690
    Data preprocessing0.7—0
    Candidate construction, AC-WM prediction, and safety scoring16.62.21$\approx$3K
    Policy optimization18.12.410
    Model loading, saving, and other overhead2.1—0
    Total per iteration87.711.32$\approx$3K

    AC-WM prediction and API safety scoring run asynchronously: completed predictions are submitted for scoring while the AC-WM generates subsequent candidates. Their overlapping execution is included in the combined 16.6-minute stage. Across 10 iterations, the GPU stages require approximately 113.2 GPU-hours. Environment rollouts dominate wall-clock time, while candidate construction, prediction, and scoring together account for 18.9%. These costs are incurred only during post-training, while deployment uses the aligned policy without additional AC-WM prediction or VLM evaluation. We have added the computational breakdown and hardware configuration to Appx. E.7, "Computational Overhead Analysis."

  • Q3 Predicted vs. executed safety

    We thank the reviewer for suggesting this comparison. We test whether improvements in imagined safety translate into safer execution on Place Bread and Pass Knife. At each alignment iteration, we measure predicted and gold Safe Rates using matched candidate actions from the same simulator contexts, comparing joint alignment with ForeAlign's decoupled world modeling (DWM). And the results are presented in Fig. 15 of the revised paper.

    The iteration curves in Fig. 15(a, b, d, e) show that ForeAlign improves predicted and executed safety together. In contrast, predicted safety rises much faster than gold Safe Rate, leaving a widening gap between imagined and actual outcomes under joint alignment (b, e). Futhermore, Figure 15(c, f) directly compares predicted and gold Safe Rates. With ForeAlign, higher predicted safety generally accompanies safer execution. under joint alignment, large increases in predicted safety correspond to much smaller execution gains. This weaker correspondence reveals the hacking behavior: the model receives increasingly favorable safety judgments without making its actions comparably safer. DWM mitigates this failure by freezing the world model and updating only the policy. The predictor can no longer change to make unchanged unsafe actions appear safe, so better safety scores must come from changes to the actions evaluated by the fixed model.

    We have added this comparison, its evaluation protocol, and analysis to Appx. G.6, “Direct Verification of Alignment Faking” (Fig. 15).

  • Q4 Comparison of policy-update objectives

    We thank the reviewer for suggesting this controlled comparison. Sec. 3.2 motivates DiffusionNFT through its use of both positive and negative safety feedback, whereas rejection fine-tuning imitates only accepted candidates. To isolate the effect of the update objective, we compare DiffusionNFT with rejection fine-tuning and Flow-GRPO [1], using the same initial policy, frozen AC-WM, cached candidate samples, safety judgments, and 10 alignment iterations. We reuse the corresponding DiffusionNFT results from Table 1 of the paper.

    Table. Comparison of policy-update objectives under matched experimental settings. Each entry reports Success Rate (%) / Safe Rate (%); higher is better for both metrics.

    Update objectivePlace BreadPass Knife
    Rejection fine-tuning95 / 4993 / 50
    Flow-GRPO [1]100 / 79100 / 65
    DiffusionNFT100 / 87100 / 74

    DiffusionNFT achieves higher Safe Rates than both alternatives while matching or improving their Success Rates under the same ten-iteration budget. These results support its effectiveness in improving safety while preserving task competence with controlled updates. We have added this comparison and its matched experimental settings to Appx. G.2 (Additional Ablation Studies), in the paragraph "Effect of the policy-update objective" and Table 11.

AuthorsOpenReview ↗

Thank you for the valuable review (3/3)

  • Q5 Alignment rounds and gradient updates

    We thank the reviewer for pointing this out. We would like to humbly clarify that we report the corresponding hyperparameters for simulation in Table 5 of Appx. E.6 (Implementation Details of Baselines) and for real-world experiments in Table 10 of Appx. F.4 (Implementation Details of ForeAlign), but did not clearly explain their correspondence to the alignment procedure. For clarity, we summarize them below.

    Table. Alignment hyperparameters using unified terminology.

    Setting / modeliterationsRollouts per iterationUpdates per sample
    Simulation / Lingbot-VA106450
    Simulation / Fast-WAM106450
    Simulation / Cosmos3-Nano106450
    Real world / Fast-WAM84025

    We further apologize for our inconsistent terminology. In the appendix, an "epoch" corresponds to one alignment iteration in Algorithm 1, comprising rollout collection, candidate evaluation, and policy optimization; "inner epochs" denotes the number of updates per sample within that iteration. As specified in its caption, Fig. 4 reports ablations evaluated after 5 alignment iterations. We have consistently updated all terminology throughout the revised paper and appendix to use "alignment iterations" and "updates per sample" (see Tables 5 and 10 in Appx. E and F, respectively) to remove this ambiguity.

  • Q6 Task-conditioning inputs

    We thank the reviewer for requesting this clarification and apologize for the unclear description. In both simulation and real-world experiments, we train and align a separate policy for each task. This applies to all three simulation models (Lingbot-VA, Fast-WAM, and Cosmos3-Nano) and to Fast-WAM in the real-world experiments. Each reported result therefore uses the corresponding task-specific policy. We have clarified this setup in Appx. E.5 and Appx. F.4 accordingly.

References:

[1] Liu, Jie, et al. "Flow-GRPO: Training Flow Matching Models via Online RL." Advances in Neural Information Processing Systems 38: 40783–40818, 2026.

Reviewer_Jg9POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ForeAlign, a framework that improves the safety of World Action Models (WAMs) by evaluating candidate actions based on their world-model-predicted outcomes and updating the policy with safety feedback. ForeAlign has three components: (i) a unified safety formulation covering both physical constraints and context-dependent risks, (ii) DiffusionNFT-based policy updates that improve safety without task rewards while limiting policy drift to mitigate OOD collapse, and (iii) a separate, frozen action-conditioned world model that directs safety feedback toward changing actions rather than predicted outcomes, mitigating alignment faking. Across simulated and real-world manipulation tasks, ForeAlign improves both safety and task success over existing methods.

Critical strengths
  1. Important problem and flexible safety framework. Safety in robotic manipulation is an important problem, especially for semantic and context-dependent risks that predefined constraints struggle to capture. The proposed framework provides a general interface for plugging in different safety evaluators, from constraint-based costs to VLM judgments, without changing the policy optimization objective, making it applicable to a broad range of safety requirements.

  2. Broad experimental coverage. The method is tested across multiple WAM architectures and task settings, including real-robot experiments, and the gains over base policies hold consistently, supporting the generality and practical applicability of the approach.

Critical weaknesses
  1. Limited methodological novelty. The three main components of ForeAlign largely build on existing ideas: (i) Bayesian policy reweighting for safety alignment, (ii) DiffusionNFT for policy updates, and (iii) a separate frozen world model, as commonly used in model-based RL. While combining these components into a unified safety alignment framework for WAMs is practically meaningful, the core methods and much of the theoretical analysis closely follow prior work. The contribution therefore appears stronger in system integration and empirical validation than in fundamental methodological innovation.

  2. Limited analysis of the role of safety feedback in performance improvements. The paper reports substantial improvements in both safety and task success using only safety feedback. However, the contributions of informative safety feedback and additional self-training are not fully disentangled. Moreover, some safety criteria overlap with task-success requirements, such as object alignment in the Pour Water task. A more targeted ablation would help clarify what drives the task-success gains and how they relate to safety optimization.

Questions for authors

Q1. Could the authors provide a small ablation study to better understand how safety feedback contributes to task success? Specifically, it would be helpful to compare ForeAlign with (i) uniform positive scores (r = 1) for self-imitation, (ii) randomly shuffled safety scores within each candidate group, and (iii) safety scores excluding criteria that overlap with task-success requirements (e.g., the horizontal-offset constraint between the bottle and cup openings in Pour Water). Results on one or two tasks would suffice.

Q2. Fig. 4b (right) reports a mean Safe Rate of 62.75% at 100% classification accuracy, while Table 1 reports an average of 84% for the same four tasks using Lingbot-VA. Could the authors clarify what differs between these experimental settings?

5 comments
AuthorsOpenReview ↗

Thank you for the valuable review

We sincerely thank you for your thoughtful assessment and constructive feedback. Your questions have helped us sharpen the presentation of our contributions and better explain the relationship between safety alignment and task performance. Below, we respond to each point and wish following clarifications and results could address your concerns.

  • W1 Limited methodological novelty

    We thank the reviewer for this thoughtful assessment. ForeAlign introduces anticipatory self-alignment of pretrained WAMs: a policy repeatedly probes its predicted action consequences for risks, internalizes the resulting safety feedback, and uses its updated behavior to guide subsequent risk discovery. The central contribution is making this loop work under safety-only supervision, where the model must improve its actions without losing task competence or merely making its predictions appear safer.

    Realizing this paradigm requires adapting the underlying methods to these challenges. First, we formulate an evaluator-agnostic safety distribution that brings physical constraints and semantic judgments into the same alignment objective. Beyond the standard Bayesian reweighting identity, our added analysis addresses the practical group-relative implementation, establishing preservation of the ideal NFT update direction and conditions for expected safety improvement. Second, we connect DiffusionNFT to this safety-reweighted target and characterize its bounded per-iteration displacement, providing a theoretical basis for controlling policy drift during safety-only adaptation. Finally, we identify alignment faking arising from the WAM’s coupled action–observation generation and introduce action-conditioned decomposition to block this shortcut. Therefore, this resulting architecture aims to ensure that alignment changes behavior rather than the predictions used to judge that behavior.

    These contributions jointly turn a pretrained WAM’s predictive capability into a unique mechanism for safety improvement. We have added a discussion in Appx. B clarifying these methodological contributions and their relationship to the established techniques on which they build.

  • W2 & Q1 Role of safety feedback in the performance gains

    We thank the reviewer for this valuable suggestion. Physical safety generally has a stronger correlation with task success than semantic safety. Physical safety violations, such as collisions that knock over a bottle, typically cause task failure unless the policy can recover. In contrast, semantic safety violations, such as inappropriate object handling, are less closely tied to task completion: a task can succeed despite violating these requirements.

    To distinguish the contribution of safety feedback from additional self-training on domain-randomized data, we compare ForeAlign with (i) self-imitation using uniform positive scores, (ii) DiffusionNFT updates with safety scores randomly shuffled within each candidate group, and (iii) alignment using safety scores that exclude physical constraints overlapping with task success. We choose Place Bread because it combines physical safety against gripper poking and semantic safety against food contamination, as specified in Table 4 of Appx. E.2. This mixed safety setting allows us to remove the physical constraint while retaining semantic feedback, directly testing whether safety gains extend beyond constraints associated with task success.

    Table. Role of safety feedback on Place Bread. Each entry reports Success Rate (%) / Safe Rate (%).

    MethodPlace Bread
    Base80 / 45
    (i) Uniform scores71 / 42
    (ii) Shuffled scores80 / 41
    (iii) Excluding physical constraints78 / 60

    Neither uniform-score self-training nor shuffled safety feedback improves safety over the base policy. Self-imitation also reduces task success, consistent with reinforcing existing behavioral flaws without corrective feedback. Shuffling removes the systematic association between candidate actions and their safety scores, leaving no informative safety preference in expectation, while DiffusionNFT’s reference-anchored updates retain the base policy’s task success. Excluding the physical constraint still improves Safe Rate but not yields task competence improvement, which further validate that semantic safety contributes less to task success than physical safety.

  • Q2 Fig. 4b vs. Table 1

    We thank the reviewer for noticing this discrepancy. We would like to humbly clarify the setting difference that causes this discrepancy: Fig. 4b evaluates after 5 alignment iterations (as stated in its caption), while Table 1 reports results after 10 alignment iterations. We apologize for not stating this distinction clearly in the main text. We have modified the "Base policies and implementations" paragraph to clarify that main results use 10 iterations while ablation studies of Fig 4(b) right use 5 iterations.

Thank you for your response.

  1. Putting these components together and making the whole loop work is a good contribution. This addresses my concern.
  2. Please consider slightly qualifying the claim regarding task-success improvements. In the new ablation, removing the physical constraint yields 78/60 on Place Bread, compared with 100/87 for the full method. This suggests that physical safety constraints, which may overlap with task-completion requirements, play an important role in the observed success gains. It would be helpful to acknowledge this relationship when stating the 24.6% improvement in task completion "without auxiliary utility signals."
  3. Table 3 is still inconsistent. Its caption says 5 iterations, but the default row is identical to the 10-iteration results in Table 1.
AuthorsOpenReview ↗

Thank you for your constructive follow-up comments

We sincerely appreciate your timely follow-up comments and the time you have dedicated to reviewing our work. We are glad that our response has addressed some of your concerns. We appreciate the opportunity to address your remaining concerns and outline the manuscript revisions below in response to your suggestions.

  • Relationship between physical safety and task success.

    Following your advice, we have reconsidered the interpretation of our results and refined the relevant statements to present our findings more accurately and responsibly. We agree that physical safety constraints can overlap with task-completion requirements and contribute substantially to the observed success gains. Accordingly, we have clarified the interpretation of the reported improvement in the Introduction by acknowledging the compatibility between physical safety constraints and task-completion requirements. We have also replaced broad references to auxiliary utility signals with more precise wording about task-success rewards in both the Introduction and the "Comparison with Baselines" paragraph in Sec. 4.1 to improve the precision of our presentation. We have further added the full-method results to Table 13 and expanded the accompanying analysis in Appx. G.2 (Additional Ablation Studies) to discuss how physical and semantic safety relate differently to task completion.

  • Table 3 iteration count.

    We sincerely apologize for the ambiguity and errors in our initial submission. Specifically, Table 3 reports results after 10 iterations, and its default-setting row corresponds to the main results reported under the same settings in Table 1. We have thoroughly checked and refined the relevant descriptions throughout the paper to ensure clarity, precision, and overall consistency.

If you have any further questions regarding our paper, we would welcome the opportunity to discuss them with you and improve our work. We will make every effort to further strengthen the paper in the final version.

Reviewer_tVMfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies that ForeAlign safely aligns pre-trained world action models. It defines the safety probability for trajectory segments and formulates safe alignment as Bayesian conditioning, which is optimal under a KL-regularized objective. Its pipeline has three parts. 1), anticipatory risk probing rolls out the current policy and evaluates K=16 future candidates using a frozen action-conditioned world model, separating policy and world model to prevent prediction hacking. 2), a universal safety distribution combines physical constraint Boltzmann weights and VLM semantic safety scores normalized by groups. 3), risk internalization adapts DiffusionNFT as an update operator with displacement control to avoid catastrophic forgetting without task reward signals. Evaluated on seven RoboTwin bimanual tasks across three backbones, it lifts success by 19.9–24.6% and safety rate by 34.5–35.6%, outperforming test-time baselines and RL methods. Real-robot tests on four tasks yield a 37% safety gain and 24.8% progress improvement.

Critical strengths
  1. A WAM outputs actions and consequences, which creates both risk and an alignment interface for "imagine first, then act". Moving safety alignment from hand-specified constraints and hardware trial-and-error to iterative risk discovery in imagination, with the probe–evaluate–internalize–re-probe loop, is an unusually complete framework in robot safety.
  2. Three heterogeneous backbones, seven bimanual tasks, six baseline families, four real tasks, with safe rate and success rising together. A robustness experiment showing gains even at 60% judge accuracy.
Critical weaknesses
  1. Theory is disconnected from implementation. Both theorems are stated under the true ξ, while semantic safety uses the group-relative proxy of Eq. 10 — a within-group relative quantity, not comparable across groups and not a probability. Whether the guarantees cover the actual training signal is doubtful. Either analyze the proxy case or downgrade the theory to a heuristic explicitly.
  2. Safety judgment depends on a closed-source model. Gemini-3.7-Flash is neither reproducible nor auditable, and a version bump could change results; for a framework selling safety, a black-box judge is a real soft spot. Add at least an open-source VLM or trained classifier control.
Questions for authors
  1. The group-relative ξ of Eq. 10 is not a calibrated probability. Does the non-decreasing expected safety of Thm 3.1 still hold for it? If not, please state that the theory covers only the constraint-based case.
  2. Can you give a direct hacking experiment — after joint (weight-shared) alignment, plot the safety score of predicted consequences against the safety rate of real execution? Under hacking the former should be visibly inflated.
  3. After several iterations, does the candidate action distribution drift outside the frozen AC-WM's reliable range? Is there monitoring or retirement based on world-model prediction confidence?
  4. How much of the main result survives if the judge is an open-source VLM or a small classifier? This tests whether the method is truly evaluator-agnostic.
2 comments
AuthorsOpenReview ↗

Thank you for the valuable review (1/2)

We sincerely appreciate your detailed and constructive review. Thank you for the time and effort you devoted to evaluating our work. Your feedback has helped us strengthen the theoretical justification and provide clearer evidence for the reliability of anticipatory safety feedback.

In response to your comments, we have extended the theoretical analysis and revise the paper accordingly. Below, we address each of your concerns and describe the corresponding revisions.

  • W1 & Q1 Theory-implementation gap and probability calibration

    We thank the reviewer for identifying this gap: the group-relative weight in Eq. (10) is not a calibrated probability, so Theorem 3.1 does not apply to it directly. We therefore extend the theory to the binary group-relative case.

    Our new analysis shows that group-relative feedback preserves the ideal NFT update direction and yields non-decreasing expected safety, even though its weights are not calibrated probabilities. Specifically, if the evaluator's binary judgments are independent samples from Bernoulli⁡(q(x))\operatorname{Bernoulli}(q(x)), where q(x)q(x) denotes the evaluator's acceptance probability (equal to the true safety probability ξ(x)\xi(x) under a faithful evaluator), the group-relative weight has expectation b+d q(x)b + d\,q(x), where b≥0b \geq 0 and d>0d > 0 are constants at a fixed context and reference policy. The constant bb cancels from the ideal NFT update, which therefore keeps its original direction with its magnitude scaled by dd. We additionally bound the degradation from evaluator and policy-fitting errors. These results do not establish the guarantee under arbitrary evaluator error or approximate policy fitting. The table below summarizes the comparison.

    AspectOriginal formal caseBinary group-relative case
    Available feedbackDirect access to ξ(x)\xi(x)Binary judgments ∼Bernoulli⁡(ξ(x))\sim\operatorname{Bernoulli}(\xi(x))
    Training weightξ(x)\xi(x)Group-dependent weight with expectation b+dξ(x)b+d\xi(x)
    NFT update directionToward the flow of π∗\pi^*The same direction, scaled by dd

    The corresponding proofs are in Appx. A.3 with a concise explanation follows Eq. (10) in Sec. 3.3. Moreover, we also assess the calibration of VLM-derived safety estimates against execution-grounded labels on Place Bread and Pass Knife. The evaluated probabilities achieve ECE values of 0.0825 and 0.0572 and Brier scores of 0.1441 and 0.1741, respectively, indicating broad agreement with observed outcomes. These experiments complement the theoretical analysis by providing empirical evidence of evaluator reliability. They assess the evaluator-derived probabilities, rather than treating the group-relative training weights as calibrated probabilities. The evaluation protocol and results are provided in Appx. G.4 (Table 14).

  • W2 & Q4 Closed-source evaluator and evaluator-agnostic testing

    We thank the reviewer for this valuable suggestion on improving reproducibility. ForeAlign does not require a particular closed-source evaluator: its safety feedback can also be supplied by a trained classifier. To address Q4 directly, we replace the VLM evaluator with a ResNet-50 safety classifier. It takes the initial and final frames of a predicted chunk as input, matching the VLM's evaluation interface. We train it on 4,096 initial–final frame pairs, and divide the frame pairs into training and test subsets at a 9:1 ratio and train for 30 epochs with an initial learning rate of 10−310^{-3}, halved every 10 epochs. We select the checkpoint with the highest accuracy on the test subset. Each entry reports Success Rate (%) / Safe Rate (%) for Lingbot-VA; the Base and ForeAlign (VLM) results are taken from Table 1 of the paper.

    MethodPlace BreadPass Knife
    Base80 / 4570 / 41
    ForeAlign w/ classifier100 / 88100 / 80
    ForeAlign (VLM)100 / 87100 / 74

    We also evaluate the classifier on the same test dataset as Fig. 4(b), using either ground-truth future observations or AC-WM-predicted future observations as input.

    Classification accuracy (%) against ground-truth safety labels:

    Future observationsPlace BreadPass Knife
    Ground-truth future96.8394.54
    AC-WM-predicted future87.5987.77

    The classifier provides accurate safety feedback on both executed and predicted futures. With this feedback, ForeAlign achieves 100% task success on both tasks and higher Safe Rates than the VLM-based variant, demonstrating that its alignment gains extend to a reproducible, locally trained evaluator. We have added Appx. G.5, "Alignment with a Trained Safety Classifier," describing the classifier architecture, dataset size, data split, training schedule, checkpoint selection, and evaluation protocol. We will release the corresponding code and configurations to support reproduction without the VLM API.

AuthorsOpenReview ↗

Thank you for the valuable review (2/2)

  • Q2 Direct verification of alignment faking

    We thank the reviewer for suggesting this direct test. We compare joint, weight-shared alignment with ForeAlign’s decoupled alignment on Place Bread and Pass Knife. Gold Safe Rate measures safety during execution, while predicted Safe Rate measures safety judged from imagined consequences. Both are evaluated using matched candidate actions from the same simulator contexts. The newly added Fig. 15 tracks these rates across alignment iterations and directly compares predicted with executed safety.

    Figure 15(a, b, d, e) shows how predicted and gold Safe Rates evolve during alignment. Without Decoupled World Modeling (DWM) (b, e), predicted safety increases substantially while gold Safe Rate improves much less, creating a pronounced gap in later iterations. With ForeAlign (a, d), predicted and executed safety improve together. Figure 15(c, f) futher shows that improvements in predicted safety are more closely accompanied by improvements in executed safety for alignment with DWM. Under joint alignment, predicted safety rises substantially while actual safety changes much less. This divergence supports the alignment-faking diagnosis: optimization makes imagined outcomes appear safer without comparable improvements in executed behavior. DWM mitigates this shortcut by freezing the world model and updating only the action policy. Alignment therefore cannot improve its score merely by changing the predicted consequences of unchanged actions, directing updates toward behavioral improvement.

    We have added the plots, analysis, and matched prediction–execution evaluation protocol to Appx. G.6, "Direct Verification of Alignment Faking".

  • Q3 Candidate drift beyond the AC-WM's reliable range

    We thank the reviewer for raising this concern. To assess whether safety feedback remains reliable as the policy evolves, we evaluate safety-classification accuracy on AC-WM-predicted futures using data from each iteration's policy rollouts. The results are summarized below; Fig. 14 in Appx. G.3 shows the curve at every iteration.

    Table. Safety-classification accuracy (%) on AC-WM predicted futures. We show every other iteration and the final iteration; the mean uses all ten iterations.

    Task / Iteration1357910Mean
    Place Bread80.1881.9577.9079.4780.4678.8879.55
    Pass Knife75.0568.3171.8372.6877.8871.5473.78

    As shown in the table and figure, Safety-classification accuracy remains broadly stable across alignment iterations, with a modest overall decline. This supports reusing the frozen AC-WM across successive policy updates without repeated predictor adaptation, consistent with ForeAlign's decoupled prediction and controlled updates, helping limit abrupt changes during alignment.

    Moreover, We would also like to humbly clarify that ForeAlign is a post-training alignment approach intended to select and refine safer behaviors within the neighborhood of the pretrained policy’s behavioral distribution, using the adapted AC-WM to evaluate their consequences. If avoiding a hazard requires behavior beyond both the base distribution and the AC-WM’s reliable predictive range, ForeAlign is not intended to synthesize that behavior from scratch, since no external utility signals are available in the alignment setting. So the current implementation does not retire the AC-WM or filter candidates based on prediction confidence. We have added the evaluation protocol, full curves, and analysis to Appx. G.3 (Fig. 14).

Comments on the paper

2 comments

Hi everyone, please acknowledge the authors’ rebuttal with a brief reply, even if your assessment is unchanged (if you already did or are about to do so, please ignore this message).

AuthorsOpenReview ↗

Look forward to your further feedback

Dear Reviewers,

We sincerely appreciate your thoughtful comments and the time you have devoted to reviewing our work. We welcome any further feedback on our responses and revisions.

If our responses have satisfactorily addressed your concerns, we would be grateful if you would consider raising your scores accordingly.

If you still have any further questions regarding our paper, we would be happy to discuss them with you and further improve our paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.