acceptodds
Under review as a conference paper at ICLR 2027

AdvFidelity: Unifying Preference Alignment and Few-Step Image Generation within Reinforcement Learning

Abstract

Practical image generation demands both alignment with human preference and few-step efficiency, yet the corresponding post-training procedures, reinforcement learning (RL) and few-step distillation, remain difficult to unify. We observe that few-step RL training carries an intrinsic distillation effect. Although few-step sampling on its own collapses into incoherent structures, generations evolve toward structural coherence as RL training proceeds. However, this effect could not remove grid-pattern artifacts, and preference rewards are inaccurate on such low-level degradations. Therefore, we introduce a fidelity reward that makes the intrinsic distillation effect explicit and reliable by supervising the removal of these artifacts. In AdvFidelity, this reward is realized as a discriminator that is updated dynamically along with the generator. The discriminator is trained on freshly constructed pairs at each training iteration. The negatives in each pair are drawn from the current generator and differ from their clean counterparts primarily in artifacts, so the discriminator keeps tracking the generator's evolving failure modes. The preference and fidelity rewards are differentiable and are optimized along the sampling trajectory used at inference via leap-based gradient propagation. In this way, a single RL stage performs preference alignment and few-step distillation simultaneously. AdvFidelity generates preference-aligned images in 8 steps without classifier-free guidance, matching or exceeding the quality of full-step CFG-guided generation.

Primary area:
generative models
Keywords:
Diffusion, Distillation, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2qCuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies unifying preference alignment and few-step image generation within a single RL stage. Post-training of image generation models typically involves two separate procedures, RL for preference alignment and few-step distillation for efficient sampling. Running them sequentially lets distillation wash out RL gains, and joint approaches such as DMDR [5] and RTDMD [3] introduce an external distillation anchor that is fixed before RL and blind to the reward. This paper observes that RL on few-step sampling trajectories already carries an implicit distillation effect, but the generations retain grid-pattern artifacts that preference rewards fail to detect. The authors therefore propose AdvFidelity, which adds a fidelity reward to the RL objective, provided by a discriminator trained online and adversarially alongside the generator. The discriminator architecture follows Adv-GRPO [1], a frozen DINOv2 backbone with a trainable MLP head. Its training data uses the proposed content-matched pair construction, where a high-quality positive is perturbed with moderate noise and completed by the current generator to form a negative, which is intended to reduce differences in high-level content and focus the discriminator on artifacts. Positives come from three anchor sources in ascending order of cost, real photos, full-step samples from the base model, and full-step samples from a model after a vanilla RL stage. The framework builds on LeapAlign [4], a ReFL-family method, and propagates reward gradients through two-step leap trajectories. Experiments cover three settings, preference alignment on HPDv2 with FLUX.1-dev and the HPSv2 reward, compositional generation on GenEval, and virtual dressing with Qwen-Image. The 8-step CFG-free model matches or slightly exceeds the 50-step full RL baseline on most automatic metrics and outperforms RTDMD [3] at equal step budgets, while the 4-step model comes close to the full-step level. Ablations attribute gains to the online update, target-styled positives, and content-matched pairs respectively.

Critical strengths

1. A clear and useful diagnosis of the phenomenon. The paper demonstrates two things cleanly. RL on few-step trajectories carries an implicit distillation effect, and preference rewards are largely blind to the grid-pattern artifacts that remain in few-step generations (Figure 1). This pinpoints a concrete, previously underexplored gap in current post-training pipelines and directly motivates why a fidelity reward is the missing piece. The diagnosis is useful to practitioners of few-step RL regardless of the proposed method.

2. A simple design with clean ablations and clear practical value. The method replaces the DMD machinery (fake score network, multi-stage schedule) with a single online reward term, which simplifies the pipeline considerably. The content-matched pair construction is simple and effective, and the cumulative ablation in Table 4 supports the role of each component, including a frozen discriminator in the initial configuration and the sequential LeapAlign-plus-DMD control. The additional control with a frozen discriminator in the full configuration is described but its results are not reported. Results are consistent across three settings, one of which is a production-level virtual dressing task, and 4 to 8 step CFG-free sampling is a real efficiency gain.

Critical weaknesses

1. The central fidelity claim lacks direct quantitative validation. The central claim is that few-step CFG-free generation matches or exceeds full-step quality. Appendix A shows on 2,000 content-matched images that HPSv3 [6] and HPSv2 assign higher scores to artifact-laden images, while PickScore, ImageReward, Aesthetic Score, and CLIPScore are insensitive to the artifacts. These results question the reliability of these metrics as evidence of artifact removal. The main experiments also report GenEval, DINO, UnifiedReward, and Gemini scores, but the appendix does not assess their sensitivity to the targeted artifacts. There is no human study, no distributional metric such as FID, and no artifact-specific measurement, so the fidelity improvement rests mainly on qualitative figures. The margin over the 50-step baseline is small (HPSv2 0.4125 versus 0.4096). The same appendix analysis finds both HPSv2 and HPSv3 [6] unreliable on these pairs, yet only HPSv3 is shaded out of the main comparison, while HPSv2 remains both the training reward and the headline metric.

2. The single-stage claim is conditional in the headline setting, and the increment over the closest prior work is limited. In the main HPDv2 setting, positives are 50-step samples from the official full-step LeapAlign [4] checkpoint, so a complete RL run is still required as data preparation, in the authors' own wording a one-off data preparation. The claim of eliminating the separate stage holds unconditionally only in the GenEval and virtual dressing settings. Meanwhile the discriminator architecture and the online adversarial reward follow Adv-GRPO [1], and using a discriminator inside distillation is established in DMD2 [2]. The increment of this paper is the content-matched pair construction and its coupling with RL, which is narrower than the framing suggests.

Generalization across preference reward models also lacks quantitative validation. Both HPDv2 and GenEval use HPSv2 for training, while virtual dressing uses a DINO feature similarity reward. The OCR example in Figure 1a provides qualitative evidence, but other preference models such as PickScore and ImageReward are used only for evaluation. Scores from multiple evaluation metrics do not replace experiments that change the training reward, so the current results do not establish consistent effectiveness across preference reward models.

Questions for authors
  1. Human evaluation and artifact-sensitive metrics. The paper motivates the method by showing that preference rewards do not reliably identify low-level artifacts, yet the main HPDv2 results still rely on these preference metrics, and the sensitivity of the other evaluation metrics to the targeted artifacts is not established. Could you add a human pairwise study focusing on low-level artifacts and overall fidelity, and report artifact-sensitive quantitative metrics such as FID against the positive set, to directly verify the central claim that 8-step CFG-free generation matches or exceeds full-step generation.
  2. Generalization across training reward models. Both HPDv2 and GenEval use HPSv2 as the training reward, virtual dressing uses DINO feature similarity, and OCR has only a qualitative example. Could you train with other preference models, such as PickScore and ImageReward, using the same base model, data, sampling steps, and training budget, and compare against few-step RL without the fidelity reward and the corresponding full-step RL baseline, reporting both preference gains and artifact removal. Please specify the positive sample source and any hyperparameter adjustments for each reward setting.
  3. Inconsistent treatment of HPSv3 and HPSv2. Appendix A shows that both HPSv3 [6] and HPSv2 prefer artifact-laden images. Why is only HPSv3 shaded out of the main comparison, while HPSv2 remains both the training reward and the headline in-domain metric. In addition, under that analysis HPSv3 should favor artifact-laden few-step outputs, yet in Table 1 the few-step models score lower on HPSv3 than the full-step baseline. How should this ordering be interpreted.
  4. Dependence on a prior RL run in the HPDv2 setting. Positives there are 50-step samples from the official full-step LeapAlign checkpoint. What happens if base-anchored positives, that is full-step samples from the base model, are used on HPDv2 instead. This would directly test whether the claim of requiring no preparatory RL stage holds under a strongly stylized reward.
  5. Training cost. Part of the motivation is that the sequential pipeline is cumbersome. Could you report GPU-hours or wall-clock for AdvFidelity single-stage training versus the sequential pipeline (full-step RL followed by DMD) and versus RTDMD [3], including the cost of preparing target-anchored positives.
  6. Definition of the local fidelity loss, and generalization to other RL frameworks. Section 3.3 uses only the [CLS] token and evaluates local regions through random crops. Does the patch-level loss weight of 0.3 in Section 4.1 refer to this crop loss. Section 3.3 also states the fidelity reward is compatible with GRPO-style methods, but all experiments instantiate a ReFL-family framework. Could you show results under an advantage-based framework, and a comparison against DMDR [5] under the same setting.
  7. Broken citations in the appendix and the Gemini evaluation protocol. In Appendix A the citations for Aesthetic Score and CLIPScore are broken and render as question marks. Please complete them. The Gemini VLM evaluation protocol appears neither in the main text nor in the appendix, for example the judging prompt, the scoring procedure, and the sample size. Please provide these details.

References:

[1] Mao et al. The image as its own reward: Reinforcement learning with adversarial reward for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.

[2] Yin et al. Improved distribution matching distillation for fast image synthesis. In Advances in Neural Information Processing Systems, 2024.

[3] Huang et al. Reinforcing few-step generators via reward-tilted distribution matching. arXiv preprint arXiv:2605.26108, 2026.

[4] Liang et al. LeapAlign: Post-training flow matching models at any generation step by building two-step trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.

[5] Jiang et al. Distribution matching distillation meets reinforcement learning. arXiv preprint arXiv:2511.13649, 2025.

[6] Ma et al. HPSv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025.

12 comments

Thank you for your helpful suggestions. We are currently conducting additional experiments following your suggestions, such as PickScore reward experiments. We will share the results and provide a more detailed response as soon as they are available. We are sorry for the delay and appreciate your patience.

AuthorsOpenReview ↗

Response part 1/8

R3.W1 / Q1 — Metric reliability and visual fidelity

Thank you for raising this important concern about the reliability of the evaluation metrics. Appendix A examines the sensitivity of preference metrics to specific low-level artifacts in content-matched reconstructions. To complement that analysis, we have added matched 4-step and 8-step LeapAlign controls using the HPSv2 reward alone, evaluated on full generations from pure noise.

These controls show an overall pattern of higher HPSv2 scores but poorer out-of-domain evaluation performance than AdvFidelity and the official LeapAlign model trained with 25-step sampling and evaluated with 50-step inference. Figure 6 in the revised paper also shows pronounced artifacts in the few-step LeapAlign outputs, supporting the diagnosis of reward hacking.

The 8-step live/EMA comparison illustrates this effect. The live parameters achieve higher HPSv2 scores than the EMA parameters from the same training run (0.4323 vs. 0.4130), while scoring lower on PickScore (23.1283 vs. 23.7752), ImageReward (1.5363 vs. 1.5658), UR-Align (3.4077 vs. 3.4491), and UR-IQ (3.5773 vs. 3.6926). Thus, these out-of-domain metrics provide complementary signals that can partially penalize reward hacking in this setting.

The two analyses probe different aspects of metric reliability: Appendix A focuses on sensitivity to isolated low-level artifacts, while the new rollout controls test how metrics respond to the quality degradation associated with reward hacking. We therefore interpret HPSv2 as the in-domain optimization score and assess visual quality through complementary out-of-domain metrics, FID, and a blinded human GSB study.

FID is computed using 50-step FLUX outputs as the positive reference set. The 8-step AdvFidelity model achieves an FID of 50.13, compared with 54.70 for 50-step LeapAlign. Deliberately degraded reconstructions yield nearly unchanged FID scores: 55.08 at noise level 0.5 with two reconstruction steps, and 54.72 at noise level 0.3 with one step. This suggests that FID alone may not adequately capture localized artifacts.

Human raters judge AdvFidelity as better in 27%, comparable in 65%, and worse in 8% of comparisons with full-step LeapAlign. Thus, it matches or exceeds the baseline's perceived fidelity in 92% of comparisons. The visual comparisons show cleaner object boundaries and less noise in fine details while preserving complete object structures. These results support comparable or better visual quality with only 8-step inference. Additional visual comparisons are shown in Figure 8 of the supplementary material.

ImagesFID ↓
LeapAlign 50-steps54.70
noise 0.555.08
noise 0.354.72
Ours 8 step50.13
ComparisonG (%) ↑S (%)B (%) ↓
AdvFidelity (8 steps) vs. LeapAlign (50 steps)27658

R3.Q2 — PickScore as the training reward

Thank you for the constructive suggestion. Both our HPDv2 and GenEval experiments in the original submission use HPSv2 as the reward, following LeapAlign's main experimental settings. We adopt these settings to enable a fair comparison with LeapAlign (using the same reward and training/evaluation data) and reduce the effort required to reproduce the baselines. Following your suggestion, we now add experiments with PickScore to evaluate generalization across training rewards.

For the additional PickScore experiment, we follow Flow-GRPO's PickScore training-data setup: we train on prompts from the Pick-a-Pic dataset at a resolution of 512 × 512 and evaluate on DrawBench. We still use FLUX.1-dev as the base model. We also implement a PickScore-only LeapAlign baseline under this setting. We train AdvFidelity for 300 iterations and PickScore-only LeapAlign for 300 iterations. Our LoRA settings, learning rates, and hyperparameters remain the same as in the HPDv2 experiments.

We compare our 8-step AdvFidelity with 8- and 50-step PickScore-only LeapAlign baselines; the published FLUX results of Flow-GRPO are included as references. The comparisons with PickScore-only LeapAlign test the contribution of fidelity supervision under a different preference reward and compare our few-step model with the corresponding full-step RL baseline.

MethodNFEPickScore ↑HPSv2 ↑ImageReward ↑UR-Align ↑UR-IQ ↑
Flow-GRPOFull-step23.97—1.32——
LeapAlignFull-step23.690.301.373.363.68
LeapAlign823.860.311.293.293.62
AdvFidelity824.240.321.373.373.69
AuthorsOpenReview ↗

Response part 2/8

R3.W2 / Q4 — Single-stage training

Thank you for highlighting this point. We understand that this concern arises from our use of images generated by a LeapAlign checkpoint in the HPDv2 experiment, which raises the question of whether our single-stage claim holds.

Our method introduces an adversarial, co-evolving fidelity reward. It requires suitable positive images from the target distribution but does not require a separate RL stage to produce them. For real-world production tasks, existing real images, Internet images, base-model samples, and ground-truth images from earlier SFT can often meet the positive-sample needs. We used RL-generated positives in HPDv2 because the reward changes the image style, the provided paired images score lower on HPSv2 than base-model generations, and we chose not to filter the HPDv2 images by HPSv2 score or collect additional positive images from the Internet.

As discussed in Section 3.4, many industrial applications, such as e-commerce and film production, seek realistic outputs. Suitable existing real images can therefore be directly reused as positives without generating additional reference images. Our virtual dressing experiment demonstrates this case. When the reward preserves the overall image style, as in instruction following, text correctness, face identity preservation, and garment consistency, base-model samples can also serve directly as positives. These are common RL objectives in production, and our GenEval experiment demonstrates the use of base-model positives.

Our choice of positives in HPDv2 reflects the requirements of this experimental setting. HPSv2 substantially changes the base model’s image style, making real images and base-model samples less suitable as positives. HPDv2 does provide paired images, but their HPSv2 scores are low—even lower than those of base-model generations, as shown in Appendix A of the revised paper—so we did not use them as positives or apply score-based filtering. We could instead filter the HPDv2 images by HPSv2 score or collect additional images from the Internet to obtain suitable positives. We therefore chose the existing LeapAlign checkpoint as a simple available way to generate positives in the target style. Specifically, we prepare 50,000 reference image–prompt pairs once using 50-step samples from the official HPSv2-aligned LeapAlign checkpoint, without reward-score filtering.

Table 4 also directly tests the effect of the reference source. The “+Adaptive” variant uses full-step base-model positives, while “+Teacher-anchored pairs” replaces them with target-style positives. Both variants use online discriminator updates and unpaired negatives, isolating the reference-source change. HPSv2 increases from 0.3558 to 0.3920 with target-style positives. Thus, the base-anchored variant operates without an RL-aligned reference model, but target-style positives provide better HPSv2 performance in this strongly stylized setting.

If the use of a LeapAlign checkpoint to generate positives still raises concerns about our single-stage claim, we see two possible ways to address them: (i) selecting suitable images from HPDv2 or collecting additional images from the Internet, and (ii) limiting our claim to single-stage generator training given suitable positive images. We would be happy to hear your suggestions.

R3.Q5 — Training cost

Thank you for pointing out the missing training-cost information. We compare the total GPU-hours of AdvFidelity, standalone LeapAlign, sequential LeapAlign + DMD, and RTDMD, with details added to Section C of the supplementary material.

All experiments use FLUX.1-dev on NVIDIA H20 GPUs at a resolution of 720 × 720, following LeapAlign’s official implementation. AdvFidelity jointly optimizes preference alignment and fidelity for 8-step generation over 400 generator updates. Standalone LeapAlign follows its official training configuration, using 25-step sampling for 300 iterations. The cost of sequential LeapAlign + DMD combines this LeapAlign training cost with the subsequent DMD distillation cost to obtain an 8-step generator. For RTDMD, we estimate the cost of obtaining an 8-step generator through 10,000 AC-DMD cold-start iterations, corresponding to 1,000 generator updates, followed by 500 joint-training iterations.

MethodTotal (GPU-hours)
AdvFidelity54.72
LeapAlign47.12
LeapAlign + DMD125.00
RTDMD157.72
Reviewer_XQuYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdvFidelity augments reward-based post-training of image generators with an online adversarial fidelity reward. The discriminator is trained on content-matched pairs consisting of a clean positive image and a negative obtained by partially noising that image and denoising it with the current generator. Preference and fidelity rewards are jointly optimized through LeapAlign. Experiments on HPDv2, GenEval, and virtual dressing demonstrate competitive image quality and alignment with 4- or 8-step generation.

Critical strengths
  1. The combination of an online discriminator and content-matched training pairs is well motivated. Online updates allow the fidelity reward to track the generator’s evolving artifacts, while paired construction reduces opportunities to discriminate using differences in content or style. These choices address a concrete limitation of preference rewards in supervising low-level image quality.

  2. The evaluation demonstrates promising results across tasks and model families. Experiments cover preference alignment, compositional generation, and virtual dressing, suggesting that the approach can improve few-step generation in different settings. The cumulative ablations further support the usefulness of online discriminator updates, appropriate positive anchors, and content-matched pairs.

Critical weaknesses
  1. The additional benefit of the fidelity reward is not clearly isolated from few-step RL itself. Table 4 compares variants that still use a fidelity discriminator, while Tables 1–2 report full-step LeapAlign. A matched 4-/8-step RL baseline without the fidelity reward is needed to show how much the proposed reward adds beyond the structural improvements already obtained through reward optimization.

  2. The fidelity reward is trained on reconstructions of partially noised clean images, but applied to images generated from pure noise. These two processes may produce different artifacts. Content matching reduces semantic and style differences within training pairs, but does not by itself establish that the discriminator detects the failure modes of full few-step generation. Appendix A examines the constructed pairs rather than directly validating this transfer.

Questions for authors
  1. Could the authors report an otherwise identical 4-/8-step RL baseline with the fidelity reward removed (λ=0)? Keeping the initialization, preference reward, optimizer, and number of generator updates fixed would help quantify how much few-step capability is already obtained by the baseline and what the proposed reward adds.

  2. How well does the fidelity discriminator distinguish clean and artifact-laden images generated from pure noise? Could the authors clarify whether its judgments remain reliable beyond the partial-noise reconstructions used for training?

2 comments

R2.1 — W1/Q1: Matched 4-/8-step RL baselines

Response

Thank you for suggesting this control experiment. Following your suggestion, we add 4-step and 8-step LeapAlign baselines trained with the HPSv2 reward alone (λ = 0), reducing the training-time sampling steps from the 25 used in the official implementation. The initialization, LoRA parameterization, training data, preference reward, optimizer, reward margin, and number of generator updates are matched to AdvFidelity at each sampling budget. The live and EMA parameters are obtained from the same training run at each budget. We have added the results of these control experiments to Table 5 of the revised paper.

Few-step LeapAlign achieves exceptionally high HPSv2 training-reward scores, far exceeding those of the official LeapAlign model trained with 25-step sampling and evaluated with 50-step inference, yet shows poorer out-of-domain performance than both AdvFidelity and the official model. This effect is especially pronounced with the live parameters, whose HPSv2 scores reach 0.4411 at 4 steps and 0.4323 at 8 steps. EMA attenuates this reward inflation, consistent with its expected role in mitigating reward hacking.

Figure 6 in the revised paper provides visual confirmation: both 4-step and 8-step LeapAlign outputs contain pronounced artifacts, showing that the higher optimized reward does not translate into better visual fidelity. This is expected for ReFL-style training, which can easily lead to reward hacking.

Compared with reward-only few-step LeapAlign, AdvFidelity improves overall out-of-domain evaluation performance and reduces visible artifacts at both 4 and 8 steps. With the sampling steps and number of generator updates matched, these improvements isolate the additional fidelity benefit of our reward beyond few-step RL alone.

MethodNFEHPSv2 ↑PickScore ↑ImageReward ↑UR-Align ↑UR-IQ ↑
LeapAlign-4 (live)40.441123.09101.44803.36653.5555
LeapAlign-4 (EMA)40.419322.64171.33973.35843.5187
AdvFidelity40.405523.62111.51643.49813.7338
LeapAlign-8 (live)80.432323.12831.53633.40773.5773
LeapAlign-8 (EMA)80.413023.77521.56583.44913.6926
LeapAlign (official)500.409623.72391.51053.48373.7284
AdvFidelity80.412523.76851.52553.50243.7509

R2.2 — W2/Q2: Discriminator accuracy on generated and constructed images

Response

Thank you for raising this important concern. To directly test whether the fidelity discriminator generalizes beyond partial-noise reconstructions, we evaluate its pairwise ranking accuracy on independent samples with the evaluated checkpoints frozen. Positive images are generated by the base model using 50-step sampling. We measure the fraction of pairs satisfying D(x⁺) > D(x⁻), using two negative-image sources: 8-step generations from pure noise and partial-noise reconstructions. We report results for both the base model and a generator checkpoint obtained during training, with whole-image, crop-level, and combined scores:

ModelImage sourcewhole imagecroptotal
original FLUX.1-dev8-step sampling from pure noise0.94600.93040.9527
1- or 2-step from partially noised positive images0.96760.99870.9820
intermediate AdvFidelity checkpoint8-step sampling from pure noise0.85650.97130.8941
1- or 2-step from partially noised positive images0.88180.99810.9271

or outputs from the original FLUX.1-dev model, the combined pairwise ranking accuracy is 95.27% on pure-noise generations and 98.20% on partial-noise reconstructions. For outputs from the training checkpoint, the corresponding accuracies are 89.41% and 92.71%. Although accuracy is slightly lower on pure-noise generations, the discriminator retains strong ranking performance in both settings. These results provide direct evidence that its discrimination ability transfers beyond the reconstruction process used for training to inference-time few-step generations.

Thank you for the detailed response and additional experiments. After considering the method’s insights and the experimental results, I have decided to keep my current rating.

Reviewer_bYgLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdvFidelity combines preference alignment and few-step image generation by augmenting preference rewards with an online adversarial fidelity reward. The discriminator learns from content-matched pairs, where clean reference images serve as positives and their partially noised, generator-reconstructed counterparts serve as negatives. Both rewards are optimized through LeapAlign. Experiments cover preference alignment, compositional generation, and virtual dressing, reporting competitive results with 4- or 8-step inference. Presentation: 2: Generally understandable, but requires substantial effort in places. Informativeness: 2: I learned something interesting or useful.

Critical strengths
  1. The method is well motivated, and content-matched pairs offer a sensible way to reduce style shortcuts in fidelity learning.
  2. Results across two model families and three settings demonstrate competitive generation quality with only 4–8 sampling steps.
Critical weaknesses
  1. The single-stage claim needs qualification: HPDv2 uses positives from an already RL-trained model, and total training costs are not reported.
  2. Table 4 omits the full-configuration frozen-discriminator baseline described in the text, weakening evidence for the necessity of online adaptation.
  3. Small gains on preference and virtual-dressing metrics need statistical validation. GenEval improves overall, but positional accuracy declines.
Questions for authors
  1. Can you provide the full-configuration frozen-discriminator results described in Section 4.4?
  2. What discriminator update ratio is used, and how does total training cost compare with RTDMD and sequential RL-then-distillation?
2 comments
AuthorsOpenReview ↗

Response part 1/2

R1.1 — W1/Q2: Single-stage training and training cost

Single-stage training

Thank you for highlighting this point. We understand that this concern arises from our use of images generated by a LeapAlign checkpoint in the HPDv2 experiment, which raises the question of whether our single-stage claim holds.

Our method fundamentally introduces an adversarial, co-evolving fidelity reward. It only requires suitable positive images from the target distribution and does not prescribe an RL-first stage for producing them. For real-world production tasks, existing real images, Internet images, base-model samples, and ground-truth images from preceding SFT can cover the vast majority of positive-sample needs. We used RL-generated positives in HPDv2 because the reward changes the image style, the provided paired images score lower on HPSv2 than base-model generations, and we chose not to filter the HPDv2 images by HPSv2 score or curate additional positive images from the Internet.

As discussed in Section 3.4, many industrial applications, such as e-commerce and film production, seek photorealistic outputs. Existing real images can therefore be directly reused as positives at no additional data-construction cost. Our virtual dressing experiment demonstrates this case. When the reward preserves the image style, as in instruction following, text correctness, face identity preservation, and garment consistency, base-model samples can also serve directly as positives. These are common RL objectives in production, and our GenEval experiment demonstrates the use of base-model positives.

Our choice of positives in HPDv2 reflects the constraints of this experimental setting. HPSv2 substantially changes the base model's image style, making real images and base-model samples less suitable as positives. HPDv2 does provide paired images, but their HPSv2 scores are very low—even lower than those of base-model generations (see Appendix A of the revised paper)—so we did not use them as positives. We could instead filter the HPDv2 images by HPSv2 score or curate additional images from the Internet to obtain suitable positives. We therefore chose the existing LeapAlign checkpoint as the simplest available way to generate positives in the target style, avoiding additional score-based filtering or data curation for this restricted academic setting.

If you feel that our use of a LeapAlign checkpoint to generate positive images undermines our single-stage claim, we have two ideas: (i) selecting images with high HPSv2 scores from HPDv2 or sourcing additional images from the Internet, and (ii) narrowing our claim to single-stage training given suitable positive images. We would be happy to hear your suggestions.

Training cost

Thank you for pointing this out. We have added a training-cost report in Section C of the supplementary material, including the total GPU-hours of AdvFidelity, standalone LeapAlign, sequential LeapAlign + DMD, and RTDMD.

All experiments use FLUX.1-dev on NVIDIA H20 GPUs at a resolution of 720 × 720; the resolution follows LeapAlign's official implementation. AdvFidelity targets 8-step generation and is trained for 400 generator updates, jointly optimizing preference alignment and fidelity. The standalone LeapAlign uses 25-step sampling during training and is optimized for 300 iterations, following its official implementation. For sequential LeapAlign + DMD, the reported GPU-hours are the sum of the LeapAlign alignment cost, using 25-step sampling for 300 training iterations, and the subsequent DMD distillation cost for producing an 8-step generator. For RTDMD, we estimate the cost of training an 8-step generator with 10,000 AC-DMD cold-start iterations (1,000 generator updates) followed by 500 joint-training iterations.

MethodTotal (GPU-hours)
AdvFidelity54.72
LeapAilgn47.12
LeapAlign + DMD125.00
RTDMD157.72
AuthorsOpenReview ↗

Response part 2/2

R1.2 — W2/Q1: Frozen-discriminator control and the full configuration

Response

We include the frozen version of the full configuration in the comparison below. Both variants use Target references and content-matched pairs; the intended difference is whether the discriminator remains frozen or is updated online. We have added this new ablation experiment to Table 4 and Figure 5 in the revision to provide a more comprehensive comparison.

ConfigurationReferencesMatched pairsDiscriminatorHPSv2 ↑PickScore ↑ImageReward ↑UR-Align ↑UR-IQ ↑
Ours w/ frozenTargetYesFrozen0.407423.18041.51613.47253.7167
Ours (Full)TargetYesOnline0.412523.76851.52553.50243.7509

R1.3 — W3: Updated try-on evaluation and category-level results

Response

Thank you for pointing out this issue. After inspecting the cases, we found that our previous evaluator, Gemini 2.5 Flash, was not sufficiently reliable at detecting garment consistency errors. We have now upgraded the evaluator to the stronger Gemini 3.1 Pro, and the updated evaluation shows more pronounced improvements. Each sample is evaluated three times, and we report the mean ± standard deviation to reduce stochastic scoring noise and provide evidence for evaluation stability.

MethodCFGNFERealism ↑Consistency ↑Alignment ↑Coherence ↑
Qwen-Image-Edit-2511Yes405.96 ± 0.868.60 ± 0.498.77 ± 0.457.91 ± 0.51
LeapAlignNo408.06 ± 0.629.35 ± 0.409.03 ± 0.448.19 ± 0.49
LeapAlignYes408.33 ± 0.429.28 ± 0.429.06 ± 0.398.30 ± 0.46
AdvFidelityNo88.45 ± 0.369.54 ± 0.279.11 ± 0.458.32 ± 0.42

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

Thank you for your careful reviews and constructive discussion, which helped us strengthen the evidence and improve the manuscript.

AdvFidelity makes few-step RL a route to joint preference alignment and efficient generation, targeting the performance of corresponding full-step RL. RL itself carries an implicit distillation effect. We make this effect more reliable through an online fidelity discriminator trained on content-matched reference–reconstruction pairs, without a separate DMD stage. The resulting 8-step models run without classifier-free guidance.

1. Progress Through Reviewer Discussions

We thank Reviewer bYgL for recognizing the motivation of content-matched fidelity learning and competitive 4-8-step results across settings. We added the frozen-discriminator control in the full configuration, reported training costs, and clarified reference-image preparation. We upgraded virtual-dressing evaluation to Gemini 3.1 Pro, with three evaluations per sample and mean ± standard deviation reporting.

We thank Reviewer XQuY for recognizing the motivation for online adaptation and content matching, the breadth of the evaluation, and the supporting ablations. We added 4-/8-step reward-only LeapAlign controls with matched initialization, data, optimizer, and generator-update budgets. We also tested discriminator ranking on independent pure-noise generations, examining transfer beyond the reconstructions used for training.

We thank Reviewer 2qCu for recognizing the useful diagnosis of RL’s implicit distillation effect and the practical value of the method. We added human GSB evaluation, FID, LeapAlign–PickScore experiments, a DiffusionNFT experiment, and detailed Gemini prompts. In the follow-up, 2qCu described the human study, reward/framework experiments, cost table, and prompts as “substantive additions”, and explicitly acknowledged that the paired analysis answered Q3 and corrected the earlier interpretation of HPSv2.

2. Stronger Experimental Evidence and Manuscript Quality

  • Retaining full-step RL performance. On HPDv2, 8-step AdvFidelity matches or exceeds 50-step LeapAlign on the five primary metrics, including HPSv2: 0.4125 vs. 0.4096 and PickScore: 23.7685 vs. 23.7239. On GenEval, its overall score is 0.7759 vs. 0.7420 for full-step LeapAlign.
  • Controlled fidelity supervision. The full-configuration frozen-discriminator and matched reward-only controls strengthen the component analysis. Combined discriminator ranking accuracy on pure-noise generations is 95.27% for base-model outputs and 89.41% for an intermediate training checkpoint.
  • Complementary visual evaluation. In the blinded human GSB comparison against full-step LeapAlign, AdvFidelity is judged better in 27%, comparable in 65%, and worse in 8% of cases. FID and qualitative comparisons complement these judgments, with FID’s limited artifact sensitivity explicitly examined.
  • New LeapAlign–PickScore experiments. On FLUX.1-dev, 8-step AdvFidelity achieves 24.24, compared with 23.86 for 8-step PickScore-only LeapAlign and 23.69 for full-step LeapAlign. It also exceeds full-step Flow-GRPO’s reported 23.97.
  • Generalization to another RL framework. The SD3.5-M experiment with DiffusionNFT reports higher scores on all five evaluated metrics at 8 steps, compared with the 40-step baseline.

We have expanded the evaluation and clarified experimental descriptions, metric interpretation, and reference-image choices, distinguishing added experiments from the original submission.

3. Broader Contribution and Research Potential

The contribution combines a useful diagnosis of RL’s implicit distillation effect, content-matched fidelity supervision, and a strong performance milestone: few-step, CFG-free models that match the corresponding full-step RL reference through a DMD-free training route. This demonstrates that few-step efficiency can be obtained while retaining the alignment performance attainable by the corresponding full-step RL method. Experiments span FLUX.1-dev and Qwen-Image-Edit, including production-level virtual dressing. GenEval uses base-model positives, while virtual dressing reuses real ground-truth images, demonstrating that fidelity references need not inherently require a preparatory RL run.

These findings motivate adaptive fidelity rewards and more general few-step RL methods that preserve alignment gains while reducing generation steps.

The revised manuscript and supplementary material are available on OpenReview. If accepted, we will integrate these updates into the camera-ready paper. Thank you again to the Area Chair and all three reviewers.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.