acceptodds
Under review as a conference paper at ICLR 2027

When to Trust Inversion: Drift-Aware Preference Optimization for Diffusion Models

Abstract

Direct Preference Optimization (DPO) has emerged as an effective paradigm for aligning text-to-image diffusion models with human preferences. Recent inversion-based methods replace random forward noising with deterministic inversion to provide more precise, image-specific preference supervision. However, as inversion progresses, the resulting latents increasingly drift away from states typically encountered during Gaussian-initialized denoising. This distributional mismatch is particularly pronounced at high-noise timesteps, undermining the effectiveness of inversion-based preference learning. To address this challenge, we propose Inversion-aware Fallback for Preference Optimization (IFPO), a drift-aware preference optimization framework with a dynamic fallback mechanism. IFPO mitigates the impact of inversion drift by adaptively reverting to standard forward noising at high-noise timesteps, while preserving the benefits of inversion-based supervision where it remains useful. Experiments across diffusion and flow-matching backbones demonstrate that IFPO consistently improves text-to-image alignment and visual fidelity over existing preference optimization methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, Preference Optimization, Text-to-Image Generation, Diffusion Preference Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VEMpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IFPO combines inversion-based and forward-noised preference supervision. A latent-variance statistic, calibrated on pretrained generation trajectories, determines when to fall back to forward noising; the winner’s decision is applied to both images. Experiments on SDXL and SD3.5 report preference gains, supplemented by checkpoint-switching diagnostics, static-hybrid controls, and an SDXL human study.

Critical strengths
  1. Distinguishing reconstruction fidelity from suitability for preference training is useful. The stage-wise DPO/InPO experiments provide an informative motivation for hybrid supervision.
  2. Results span diffusion and flow-matching backbones, with additional Parti-Prompts evaluation. The blinded human study and static-training comparison provide meaningful evidence of empirical benefit.
Critical weaknesses
  1. The sample-dependent fallback is only weakly distinguished from a fixed timestep cutoff. The fallback rate almost exactly matches the static schedule and the static-train variant recovers most of the improvement, yet the paper never reports how the decision varies across samples at a fixed timestep, so the reader cannot tell whether the calibrated variance test does anything beyond “fall back above some noise level.” The inference-time evidence for prompt matching rests on a generate–invert–reselect loop with a self-consistency criterion that has no principled justification and does not reflect the training-time method.
  2. The diagnosis is thinly supported and the proxy is crude. A single scalar variance over all latent coordinates cannot characterize distributional drift, and no direct measurement of the reachability gap (a distribution distance or a probe between inverted and generated latents per timestep) is given beyond one decoded example; the appendix derivation is a telescoping sum plus an alignment assumption that is never verified. The checkpoint-switching experiment mixes two separately trained models at inference, which tests which model is better at which step, not which supervision is better at which step.
  3. Gains are modest and the baselines look fragile. On the diffusion backbone the multi-dimensional preference metric does not improve over DPO on one of the two benchmarks, and on the flow backbone every reproduced baseline, including DPO, falls below the pretrained model, which signals an under-tuned setup in which IFPO’s own margin over the untuned model is small. The reproduced InPO is weaker than DPO on the training-distribution prompts, contradicting the original paper, so the effect size relative to the method’s own starting point is unclear, and the tolerance study spans a very narrow range with a backbone-dependent default that is not explained.
Questions for authors
  1. Does the variance score predict which training pairs benefit from inversion after controlling for timestep, and how often does the winner-only gate retain a badly drifted loser?
  2. Does IFPO outperform a shuffled training gate with identical per-timestep fallback rates across seeds and in direct human comparisons?
  3. Do the gains persist with more accurate inversion and calibration matched to training resolution at comparable total compute?
2 comments

We thank the reviewer for recognizing the value of distinguishing reconstruction fidelity from suitability for preference training, and the evidence from our stage-wise experiments, backbone coverage, and human study.

1. Stage-wise supervision and sample-dependent selection (Q1, Q2)

(Q2, shuffled training gate) Following the reviewer’s suggestion, we trained a shuffled-gate control that preserves IFPO’s fallback rate at each timestep while randomly selecting which samples receive forward-noising supervision. All other training and evaluation settings are identical. Mean scores on the 500 Pick-a-Pic test prompts using SDXL are:

MethodPickScoreHPSImageRewardMPS
Diffusion-DPO22.6528.500.9512.17
InPO22.5628.270.8411.94
Static train22.7828.800.9612.07
Shuffled gate22.9328.711.0212.12
IFPO22.9328.831.0312.17

With per-timestep fallback rates matched, IFPO achieves comparable PickScore and higher mean HPS, ImageReward, and MPS. This provides additional training-time evidence for the benefit of sample-dependent supervision selection under the evaluated settings.

(Q1, variation within a timestep) We examined the first fallback occurrence for 500 samples on a 10-point inversion grid. The counts at timesteps 601, 701, 801, and 901 were 25, 332, 140, and 3. Thus, fallback has a strong timestep dependence while retaining sample variation. At timestep 701, at least 332 samples trigger fallback, while 143 have yet to trigger it. These observations demonstrate that samples receive different decisions at the same timestep. The shuffled-gate experiment above further tests the performance contribution of this sample selection.

The effectiveness of Static train also supports the stage-wise complementarity identified in our paper. Our contributions include identifying the mismatch between inverted supervision states and ordinary generation states, demonstrating stage-wise complementarity, and developing effective hybrid supervision informed by these findings. Fixed splitting is a simple implementation of this insight and forms part of our empirical contribution. IFPO further uses a calibrated variance criterion to select supervision. The Static train and shuffled-gate comparisons help distinguish the benefits of stage-wise supervision allocation and sample-dependent selection.

2. Scope of the variance criterion and pair-level selection (Q1)

(Q1, characterizing drift) A scalar variance statistic captures one aspect of the latent distribution. We use it as an inexpensive selection signal whose effectiveness is supported by training comparisons, including Static train and the new shuffled-gate control. Our claim concerns its empirical effectiveness as one implementation for mitigating inversion drift, with its optimality left open.

The diagnostic experiments provide complementary evidence. Checkpoint switching directly demonstrates the stage-wise complementarity of two trained models and motivates hybrid supervision. Static train, IFPO, and shuffled gate provide training-time comparisons of supervision rules. The appendix derivation describes drift accumulation under its stated assumptions.

(Q1, retaining a drifted loser) Among pairs where the winner remains below the threshold and inversion is retained, 3.56% of losers exceed the same threshold at the same timestep. This directly quantifies the frequency of retaining an above-threshold loser under winner-only selection.

We also trained alternative pair-level selection rules under identical SD3.5 settings. “Either” and “both” specify how many images must exceed the threshold to trigger fallback.

Method / rulePickScoreHPSImageRewardMPS
Base22.1428.491.0311.93
Diffusion-DPO22.1128.481.0511.99
InPO22.2628.631.0812.13
IFPO, winner22.4328.701.0912.26
IFPO, loser22.2628.731.1012.19
IFPO, either22.2928.721.1012.22
IFPO, both22.3728.671.0812.26

All four rules exceed the three baselines on all four unrounded means, with trade-offs across metrics. These results support winner-only as an effective choice and demonstrate gains across several pair-level rules. They appear in Appendix C.2, Table 7.

3. Baselines and resolution consistency (Q3)

(Q3, matched resolution) The additional SD3.5 experiments above use 512 × 512 throughout calibration, training, and inference, with λ=1, matching the SDXL default.

Under this setting, IFPO exceeds Base, Diffusion-DPO, and InPO on all four mean metrics. InPO also exceeds Base on all four means. These results provide a baseline comparison with consistent calibration and training resolution and show that IFPO remains effective with the shared λ=1 setting. They have been added to Section 5.3 and Appendix C.1, Table 6.

Thank you for the additional experiments and clarifications. The shuffled-gate control and matched-resolution experiments have addressed important gaps in the evaluation. However, the gains over the shuffled gate are modest, with identical reported PickScore, so the benefit of sample-dependent selection remains unclear. Its connection to harmful drift is also not directly validated, and the comparison with more accurate inversion at comparable compute remains missing. I therefore tend to maintain my original score.

Reviewer_oZwEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the suitability of inversion-based supervision for preference optimization in text-to-image diffusion models. The authors argue that inverted latents can deviate from states encountered during Gaussian-initialized generation, particularly at high noise levels. A checkpoint-switching experiment reveals stage-wise complementarity: using a DPO model during high-noise denoising and an InPO model during low-noise denoising outperforms either model alone. Motivated by this observation, the paper proposes Inversion-aware Fallback for Preference Optimization (IFPO), which uses a calibrated variance-deviation criterion to select between inversion-based supervision and standard forward noising. Experiments on SDXL and SD3.5 Medium show improvements across several preference benchmarks.

Critical strengths
  • Clear motivation. The paper distinguishes reconstruction fidelity from the suitability of inversion trajectories for generation-oriented preference learning. This is a useful perspective on a potential limitation of inversion-based alignment.

  • Interesting stage-wise diagnostic. Figure 3 is a particularly informative result. The authors compare both switching directions under shared prompts and initial noise, revealing directional complementarity between DPO and InPO rather than merely a generic benefit from model composition. Although stage-specific denoising has precedents, this particular empirical finding is valuable.

  • Simple and practical method. IFPO changes the selection of supervisory states while retaining the preference objective. The resulting model uses the ordinary inference procedure without requiring checkpoint switching.

  • Strong empirical results on both backbones. IFPO achieves strong preference-alignment results on both SDXL and SD3.5 Medium across multiple benchmarks.

Critical weaknesses
  • Limited evaluation on recent large-scale models. I do not consider SDXL and SD3.5 sufficient platforms for advancing diffusion DPO research. Evaluating IFPO on models such as FLUX and Qwen-Image would provide stronger evidence that the identified inversion drift, stage-wise complementarity, and proposed fallback mechanism generalize to current model families.

  • The SD3.5 calibration and training resolutions are inconsistent. SD3.5 is trained at 512×512, whereas its reference thresholds are calibrated using 1024×1024 generation trajectories. The paper does not establish whether these thresholds transfer reliably across resolutions. This also complicates the interpretation of the different tolerance values used for SDXL and SD3.5.

Questions for authors

None

1 comment

We thank the reviewer for recognizing the motivation, stage-wise complementarity analysis, practical design, and empirical results of our work. We address the two main concerns below.

1. Experimental platforms and data selection for diffusion DPO (Weakness 1)

We appreciate the suggestion to extend our evaluation to models such as FLUX and Qwen-Image. Our choice of experimental platforms follows established practice in diffusion DPO research: related methods such as InPO (CVPR 2025), Self-Supervised DPO (NeurIPS 2025), and the recent TCPO (ECCV 2026) conduct evaluations using Stable Diffusion models. SDXL enables controlled comparisons with directly related methods, while SD3.5 additionally allows us to evaluate IFPO on a flow-matching backbone.

For diffusion DPO, extending evaluation to stronger backbones also requires considering the suitability of preference data: the data should provide informative preference distinctions appropriate to the target model's capabilities. Such an extension therefore involves not only replacing the backbone, but also preparing and validating suitable data. This consideration also motivates our use of Pick-a-Pic v2 for SDXL and HPD v3 preference pairs involving SD3-generated images for SD3.5.

We agree that experiments on FLUX and Qwen-Image would provide further evidence of generalization. We currently lack the suitable data and training resources needed for systematic DPO comparisons on these models and regard this as a direction for future extension. Our present experiments focus on validating inversion drift and the adaptive fallback mechanism, covering both diffusion and flow-matching backbones.

2. Consistent calibration and training resolution and tolerance settings (Weakness 2)

We thank the reviewer for highlighting calibration and training resolution consistency. To directly address this concern, we conducted additional SD3.5 experiments using 512×512 resolution throughout calibration, training, and inference, with the tolerance set to λ=1, matching the default value used for SDXL. Under the same experimental settings, the mean evaluation scores on the Pick-a-Pic v2 test set are:

MethodPickScoreHPS v2ImageRewardMPS
Base22.1428.491.0311.93
Diffusion-DPO22.1128.481.0511.99
InPO22.2628.631.0812.13
IFPO (λ=1)22.4328.701.0912.26

With consistent resolution and λ=1, IFPO still outperforms Base, Diffusion-DPO, and InPO on all four mean metrics. These results demonstrate that IFPO's effectiveness on SD3.5 does not depend on the original cross-resolution configuration or the choice of λ=2.5.

These results have been added to the “Calibration at matched resolution” paragraph in Section 5.3 and to Appendix C.1, Table 6, of the revised manuscript. We have also clarified the calibration configurations of the original and additional experiments in Appendix B.2 and recommend calibrating at the target training resolution.

Reviewer_HUT7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Inversion-aware Fallback for Preference Optimization (IFPO) to address the Gaussian reachability gap in inversion-based diffusion preference optimization, where inverted latents deviate from standard generation states, particularly at high-noise timesteps. IFPO introduces a variance-based dynamic fallback mechanism that adaptively switches between inversion-based supervision and standard forward noising while retaining the original DPO objective. Experiments on SDXL and SD3.5 Medium demonstrate improved preference alignment over existing methods across multiple benchmarks.

Critical strengths
  1. The paper identifies the Gaussian reachability gap in existing inversion-based preference optimization methods, highlighting the distributional mismatch between inverted latents and states encountered during standard Gaussian-initialized generation, particularly at high-noise timesteps.
  2. The proposed IFPO introduces a variance-based dynamic fallback mechanism that adaptively selects between inversion-based supervision and standard forward noising. The method does not require additional reward models or modifications to the original DPO objective, making it relatively straightforward to integrate into existing diffusion preference optimization frameworks.
  3. The experiments cover both diffusion and flow-matching backbones, including SDXL and SD3.5 Medium, and evaluate performance across multiple benchmarks and preference metrics.
Critical weaknesses
  1. Heuristic fallback design and limited ablations. ● The paper uses spatial variance deviation as a proxy for inversion drift, but does not sufficiently justify its advantages over alternative measures, such as absolute or normalized variance deviation. ● The fallback threshold relies on the 95th percentile of 2,000 offline generation trajectories and an additional tolerance parameter \lambda. These choices are largely heuristic, while Table 3 only explores a narrow range of \lambda values on SD3.5. ● Table 2 compares different switching strategies, these experiments mainly concern inference-time model composition rather than training-time comparisons of alternative drift metrics and thresholding strategies.
  2. Potential computational redundancy. ● IFPO requires iterative DDIM inversion before evaluating the fallback condition. When forward noising is selected, the preceding inversion computations may become redundant, introducing unnecessary overhead.
  3. Insufficient validation of calibration and winner-only selection. ● Both backbones use Pick-a-Pic v2 prompts for threshold calibration, while SD3.5 is trained on HPD v3 with a different training resolution. The impact of these differences on calibration robustness remains unclear. The fallback decision is based solely on the winner's inversion drift, although winner and loser images may exhibit different inversion errors. The paper lacks ablations validating this winner-only selection strategy.
Questions for authors
  1. Why is spatial variance deviation chosen to measure inversion drift? Have the authors considered alternative metrics, such as absolute variance deviation, normalized variance deviation, or other latent distribution statistics? Could the authors compare these alternatives under the same training settings?
  2. What motivates the use of 2,000 calibration trajectories and the 95th percentile for threshold estimation?
  3. When fallback is triggered, does the implementation still perform DDIM inversion for both the winner and loser images? My understanding is that the inversion computations already performed to evaluate the fallback condition may become redundant once standard forward noising is selected. Could the authors clarify how this computational overhead is handled and provide training-time or computational cost comparisons among IFPO, InPO, and Diffusion-DPO under identical hardware settings?
  4. Since the thresholds are calibrated using prompts from Pick-a-Pic v2, is recalibration necessary when changing the preference training dataset or image resolution? In particular, SDXL uses Pick-a-Pic v2 for both calibration and preference training, whereas SD3.5 uses HPD v3 for preference training. Could the authors clarify the rationale for this setup and evaluate the sensitivity of IFPO to different calibration data distributions?
  5. Why is the fallback decision based only on the variance deviation of the winner image? Have the authors considered loser-only or joint winner–loser decision strategies, and how would these alternatives affect preference optimization performance?
  6. Could the authors clarify how the timestep-dependent threshold \tau_t in Eq. (12) is obtained and applied during training? According to Eq. (11), my understanding is that \tau_t is precomputed from offline generation trajectories and remains fixed for each timestep. Is it implemented as a lookup table shared across all preference pairs at the same timestep, or is any sample-specific calibration performed? A more explicit description of the practical implementation would be helpful.
  7. Does the reported 30.9% fallback rate mean that approximately 30.9% of the sampled training instances use standard forward noising rather than DDIM inversion for constructing supervisory latents?
2 comments

We thank the reviewer for highlighting the distinction between inverted latents and the states encountered during generation.

1. Choice of drift metric and pair-level fallback rule (Q1, Q5)

(Q1, variance deviation) Our criterion already uses the suggested absolute deviation. Let V(x)=d−1∑j=1d(xj−xˉ)2V(x)=d^{-1}\sum_{j=1}^d(x_j-\bar{x})^2 denote latent variance. We score an inverted latent by its absolute difference from the expected forward-noising variance at the same timestep:

Δt=∣V(xtinv)−[μt2V(x0)+σt2]∣.\Delta_t=\left|V(x_t^{\mathrm{inv}})-\big[\mu_t^2 V(x_0)+\sigma_t^2\big]\right|.

Variance gives us an inexpensive signal for selecting supervision. We claim empirical effectiveness for this choice and leave its optimality open. Normalizing the deviation is a distinct alternative that we have yet to compare in training.

The checkpoint-switching experiments examine stage-wise complementarity at inference. Table 2 also compares two rules during SDXL training. Static train uses a fixed timestep cutoff and falls back on 30.0% of training instances, compared with 30.9% for IFPO. On the 500 Pick-a-Pic prompts, Static train scores 22.78 / 28.80 / 0.96 / 12.07 and IFPO scores 22.93 / 28.83 / 1.03 / 12.17 (mean PickScore / HPS / ImageReward / MPS). IFPO thus improves all four means over the fixed cutoff at a similar overall fallback rate.

(Q5, winner-only selection) We use the preferred image to select a shared supervision branch for the pair. Winner-only is one effective choice. To test how much this choice matters, we trained four variants under identical SD3.5 settings. “Either” and “both” specify how many images must exceed the threshold to trigger fallback. The table reports mean scores on Pick-a-Pic v2.

Fallback rulePickScoreHPS v2ImageRewardMPS
Base22.1428.491.0311.93
Diffusion-DPO22.1128.481.0511.99
InPO22.2628.631.0812.13
IFPO, winner22.4328.701.0912.26
IFPO, loser22.2628.731.1012.19
IFPO, either22.2928.721.1012.22
IFPO, both22.3728.671.0812.26

All four rules exceed the three baselines on all four unrounded means. Winner-only gives the highest PickScore, while loser-only gives the highest HPS and ImageReward. Appendix C.2, Table 7 reports these training ablations.

2. Calibration and threshold implementation (Q2, Q4, Q6)

(Q2, percentile and sample size) We use the 95th percentile to limit the influence of extreme trajectories on the upper reference value. During calibration, we observed that the per-timestep threshold estimates had stabilized by 2,000 trajectories.

(Q4, transfer across settings) We calibrate on ordinary generation trajectories from the backbone. We reuse the thresholds when only the preference dataset changes and recalibrate when the backbone or resolution changes. For SD3.5, we used Pick-a-Pic v2 calibration prompts and HPD v3 preference training pairs. The additional experiments above use 512 × 512 throughout calibration, training, and inference, with λ=1\lambda=1 (Section 5.3 and Appendix C.1, Table 6).

(Q6, threshold lookup) Yes. We precompute one τt\tau_t per timestep offline. All training pairs at timestep tt share that fixed value. Each pair compares its own winner score Δt\Delta_t with λτt\lambda\tau_t, and both images take the selected branch.

3. Inversion cost and fallback frequency (Q3, Q7)

(Q3, computation) IFPO reuses InPO’s inversion. When fallback triggers, the inversion work already performed is unused. Evaluating the variance rule adds no model forward calls during training. A winner-first implementation can also skip loser inversion whenever the winner triggers fallback. Diffusion-DPO uses forward noising directly and avoids the inversion cost of both methods.

The one-time calibration costs 2,000 × 50 = 100,000 per-sample model calls. Inversion alone accounts for about 400,000 × 5 = 2 million calls over training. Calibration is thus about 5% of inversion calls and, assuming comparable per-call cost, less than 5% of total compute once training is included.

(Q7, meaning of 30.9%) Yes. In 30.9% of sampled training instances, both images in the pair receive forward-noising supervision.

Thanks to the authors for the responses. Some of my concerns are resolved, but the overall gains remain modest, so I will keep my score at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.