acceptodds
Under review as a conference paper at ICLR 2027

SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Models

Abstract

The post-training pipeline for diffusion models currently has two stages: supervised fine-tuning (SFT) on curated data and reinforcement learning (RL) with reward models. A fundamental gap separates them. SFT optimizes the denoiser only on ground-truth states sampled from the forward noising process; once inference deviates from these ideal states, subsequent denoising relies on out-of-distribution generalization rather than learned correction, exhibiting the same exposure bias that afflicts autoregressive models, but accumulated along the denoising trajectory instead of the token sequence. RL can in principle address this mismatch, yet its terminal reward signal is sparse, suffers from credit-assignment difficulty, and risks reward hacking. We propose SOAR (Self-Correction for Optimal Alignment and Refinement), a bias-correction post-training method that fills this gap. Starting from a real sample, SOAR performs a single stop-gradient rollout with the current model, re-noises the resulting off-trajectory state, and supervises the model to steer back toward the original clean target. The method is on-policy, reward-free, and provides dense per-timestep supervision with no credit-assignment problem. On SD3.5-Medium, SOAR improves GenEval from 0.70 to 0.78 and OCR from 0.64 to 0.67 over SFT, while simultaneously raising all model-based preference scores. In controlled reward-specific experiments, SOAR surpasses Flow-GRPO in final metric value on both aesthetic and text-image alignment tasks, despite having no access to a reward model. Since SOAR's base loss subsumes the standard SFT objective, it can directly replace SFT as a stronger first post-training stage after pretraining, while remaining fully compatible with subsequent RL alignment.

Primary area:
generative models
Keywords:
Diffusion Models, Post-Training, Self-Correction, Exposure Bias, Reward-Free Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_xWkWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SOAR is a supervised post-training method that exposes diffusion/flow-matching models to their own local prediction errors. It constructs states through a detached one-step rollout and shared-noise re-noising, then supervises recovery toward the paired clean image. On SD3.5-Medium, it improves GenEval from 0.70 to 0.78 and OCR from 0.64 to 0.67 over SFT, without additional inference cost. Experiments include component ablations, supervised-state-count controls, and matched-LoRA comparisons with Flow-GRPO.

Critical strengths
  1. SOAR provides a simple way to train on model-induced errors using paired-data supervision, without reward-model queries during optimization or additional inference cost.
  2. Broad-data improvements are supported by useful controls for supervision count and LoRA parameterization, strengthening the evidence beyond a standard SFT comparison.
Critical weaknesses
  1. The contribution of the correction target is less clear than that of state augmentation. In Table 4, the standard-target substitution already improves five of six metrics over SFT, while the proposed target outperforms this substitution on only three. The evidence for the specific correction mechanism is therefore mixed.

  2. The matched-LoRA advantages over Flow-GRPO are small and lack uncertainty estimates in the main-paper tables. Computational costs are not directly matched, and reduced diversity and some non-target regressions complicate claims of an overall advantage.

Questions for authors
  1. Can repeated-run results clarify whether the correction target reliably improves over the standard-target ablation, particularly on GenEval and OCR?
  2. Can you provide uncertainty estimates and measured training costs for the matched-LoRA comparisons with Flow-GRPO?
  3. Which configuration produced the reported diversity reductions, and how do its quality and diversity compare with the standard-target ablation?
Reviewer_x69xOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SOAR, a post-training method designed to mitigate exposure bias in diffusion models. SOAR first constructs model-induced off-trajectory states through a one-step rollout, applies shared-noise re-noising, and trains the model to predict correction velocities toward paired clean targets. The method improves several generation-quality metrics over standard SFT.

Critical strengths
  1. Empirical improvements: SOAR achieves higher performance over SFT on several generation-quality metrics, particularly GenEval and OCR.

  2. Simple and practical method: SOAR extends standard flow-matching training without modifying the inference architecture or sampler.

Critical weaknesses
  1. Unclear problem formulation: The paper motivates SOAR through both exposure bias in SFT and limitations of reward-based RL. However, the proposed method mainly addresses exposure bias through supervised correction rather than optimizing a reward objective. This makes the paper's primary research objective and positioning somewhat unclear.

  2. Insufficient evidence for exposure-bias mitigation: Although mitigating exposure bias is the central motivation, the experiments mainly focus on downstream reward-related metrics rather than directly measuring exposure bias. These results do not clearly support that exposure bias is reduced compared to baselines.

  3. Limited exposure-bias baseline comparisons: Although Table 5 compares SOAR with Input Perturbation [1] and Epsilon Scaling [2], the evaluation is limited to Aesthetic scores.

  4. Unclear fairness of reward-alignment comparisons: The reward-specific experiments compare SOAR trained on reward-curated paired data with Flow-GRPO using online reward evaluations. Since these methods rely on different supervision, the comparison does not clearly establish an advantage over reward-based alignment. Moreover, recent reward-based method such as DiffusionNFT [3] is omitted.

[1] Ning, Mang, et al. "Input Perturbation Reduces Exposure Bias in Diffusion Models." ICML, 2023.

[2] Ning, Mang, et al. "Elucidating the Exposure Bias in Diffusion Models." ICLR, 2024.

[3] Zheng, Kaiwen, et al. "DiffusionNFT: Online Diffusion Reinforcement with Forward Process." ICLR, 2026.

Questions for authors
  1. Primary objective and experimental evidence: What is the primary goal of SOAR: mitigating exposure bias or improving reward alignment? If the main goal is mitigating exposure bias, why do the main experiments focus on reward-related metrics? How do these results support the claim of reduced exposure bias, and could the authors provide more direct evidence showing that SOAR effectively mitigates exposure bias?

  2. Exposure-bias baselines: Since mitigating exposure bias appears to be the main objective of the paper, comparison with exposure-bias baselines should be a more central part of the experiments. Could the authors extend Table 5 beyond Aesthetic scores to other metrics, such as GenEval and OCR?

  3. Reward-based baselines: Could the authors additionally compare SOAR with DiffusionNFT [3] in the reward-alignment experiments (Figure 3 and Table 3)?

Reviewer_WD4wOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

SOAR augments the standard flow-matching fine-tuning with an auxiliary correction loss influenced by the model's own predictions. It starts from a noisy training latent, takes a detached model step, adds back noise to the resulting state using the original noise sample, and constructs a modified velocity target that is anchored to the paired cleaned target. Experiments, using SD3.5-Medium as backbone model, compare full-parameter training and LoRA against SFT, GRPO, and exposure-bias baselines. SOAR improves performance on rule-based and model-based metrics over SFT, and performs well on reward-curated data, though results vary by metric. Finally, SOAR exhibits reduced diversity compared to SFT, indicating a quality-diversity trade-off.

Critical strengths

Exposing the model to inputs affected by its own predictions during training is an interesting approach to improving robustness during generation. SOAR provides a concrete way of incorporating such examples into the supervised training paradigm.

Additionally, the proposed methodology demonstrates meaningful empirical gains. SOAR improves performance not only compared to the SFT baseline using the same backbone model (SD3.5-Medium), but also against GRPO in the matched-LoRA setting.

Critical weaknesses
  1. Limited generalizability: The evaluation covers only SD3.5-Medium. The conducted experiments do not demonstrate whether gains transfer across different backbones, weakening support for SOAR as a broadly applicable training method.
  2. State-count controls and SFT 35k validity: In table 2, SOAR receives 7 supervised states per sample per update. The SFT 35k matches the sample count, but increases the optimizer updates (7x) and introduces a significant optimization difference. Additionally, SFT 35k performs worse than SFT 5k on 4 out of 6 reported metrics. This could be attributed to overfitting or an unstable optimization schedule. Further analysis would be needed to establish the SFT 35k as a competitive baseline.
  3. Uncertainty and evaluation reproducibility: Results lack confidence intervals and an evaluation of sensitivity to seeds, weakening the claimed performance improvements. This is particularly important for small differences reported in tables (e.g., SOAR-GRPO CLIPScore gap of 0.006 in Table 3). The lack of generation-seed handling was also identified upon inspection of the code provided as supplementary material, further weakening reproducibility.
  4. Restricted scope in the matched-LoRA comparisons: The share-adapter comparison in Table 3 is fair within the reward-curated setting that is applied. However, this curation (i.e., High-Aesthetic and High-CLIP subsets) may indirectly provide SFT and SOAR with a preference signal, reducing the benefits of explicit reward optimization by GRPO. An additional comparison on the broader data would better support the results.
  5. Missing computational analysis: The different methods used in the experiments perform different computations per update. SFT predicts from a single noisy input, SOAR additionally constructs the auxiliary states and predicts their targets, and GRPO generates samples, scores rewards, and performs updates. Equal update counts or supervised-states counts might not correspond to equal computational budgets. SOAR's computational overhead, compared to SFT, should be quantified (e.g., GPU-hours, peak memory usage).
Questions for authors
  1. How do SOAR's improvements transfer across different backbones?
  2. Why does SFT 35k underperform compared to SFT 5k on 4 out of 6 reported metrics? Were intermediate checkpoints also evaluated?
  3. Could uncertainty estimates be provided for the reported results?
  4. Could you provide a matched-LoRA comparison on the full training corpus or a randomly selected subset of it to assess whether SOAR's relative improvement over GRPO extends to non-curated subsets?
  5. Could you provide training time and peak memory usage measurements of SFT, SOAR, and GRPO under the same (or comparable) hardware settings?
Reviewer_ctZcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SOAR, a bias-correction post-training method that fills a gap between SFT and RL. SORA is a reward-free post-training method that modifies ordinary SFT by adopting a single stop-gradient ODE. Experiments also demonstrated the improvement of the proposed method.

Critical strengths

This paper proposes an interesting gap between SFT and RL. SFT provides explicit denoising targets but typically trains on ground-truth noising states, leaving recovery from self-induced errors to generalization. Online RL learns from model-generated trajectories, but terminal rewards offer limited step-level guidance. This work's findings may inform future research in this domain.

Critical weaknesses
  1. The paper is hard to follow; although some conceptual explanations are provided, the writing still needs improvement.
  2. The biggest issue is whether the proposed method degrades the diversity of the diffusion model. Since the SOAR has modified the procedure of SFT.
Questions for authors

See weakness.

Comments on the paper

1 comment

Thank you for the detailed responses and additional experiments. However, the revised version of the paper substantially changes its central positioning and experimental evidence, introducing several key results only at the end of the rebuttal period.

I believe revisions of this scope require a more thorough evaluation and more time than the remaining discussion period allows. Therefore, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.