- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies unifying preference alignment and few-step image generation within a single RL stage. Post-training of image generation models typically involves two separate procedures, RL for preference alignment and few-step distillation for efficient sampling. Running them sequentially lets distillation wash out RL gains, and joint approaches such as DMDR [5] and RTDMD [3] introduce an external distillation anchor that is fixed before RL and blind to the reward. This paper observes that RL on few-step sampling trajectories already carries an implicit distillation effect, but the generations retain grid-pattern artifacts that preference rewards fail to detect. The authors therefore propose AdvFidelity, which adds a fidelity reward to the RL objective, provided by a discriminator trained online and adversarially alongside the generator. The discriminator architecture follows Adv-GRPO [1], a frozen DINOv2 backbone with a trainable MLP head. Its training data uses the proposed content-matched pair construction, where a high-quality positive is perturbed with moderate noise and completed by the current generator to form a negative, which is intended to reduce differences in high-level content and focus the discriminator on artifacts. Positives come from three anchor sources in ascending order of cost, real photos, full-step samples from the base model, and full-step samples from a model after a vanilla RL stage. The framework builds on LeapAlign [4], a ReFL-family method, and propagates reward gradients through two-step leap trajectories. Experiments cover three settings, preference alignment on HPDv2 with FLUX.1-dev and the HPSv2 reward, compositional generation on GenEval, and virtual dressing with Qwen-Image. The 8-step CFG-free model matches or slightly exceeds the 50-step full RL baseline on most automatic metrics and outperforms RTDMD [3] at equal step budgets, while the 4-step model comes close to the full-step level. Ablations attribute gains to the online update, target-styled positives, and content-matched pairs respectively.
Critical strengths
1. A clear and useful diagnosis of the phenomenon. The paper demonstrates two things cleanly. RL on few-step trajectories carries an implicit distillation effect, and preference rewards are largely blind to the grid-pattern artifacts that remain in few-step generations (Figure 1). This pinpoints a concrete, previously underexplored gap in current post-training pipelines and directly motivates why a fidelity reward is the missing piece. The diagnosis is useful to practitioners of few-step RL regardless of the proposed method.
2. A simple design with clean ablations and clear practical value. The method replaces the DMD machinery (fake score network, multi-stage schedule) with a single online reward term, which simplifies the pipeline considerably. The content-matched pair construction is simple and effective, and the cumulative ablation in Table 4 supports the role of each component, including a frozen discriminator in the initial configuration and the sequential LeapAlign-plus-DMD control. The additional control with a frozen discriminator in the full configuration is described but its results are not reported. Results are consistent across three settings, one of which is a production-level virtual dressing task, and 4 to 8 step CFG-free sampling is a real efficiency gain.
Critical weaknesses
1. The central fidelity claim lacks direct quantitative validation. The central claim is that few-step CFG-free generation matches or exceeds full-step quality. Appendix A shows on 2,000 content-matched images that HPSv3 [6] and HPSv2 assign higher scores to artifact-laden images, while PickScore, ImageReward, Aesthetic Score, and CLIPScore are insensitive to the artifacts. These results question the reliability of these metrics as evidence of artifact removal. The main experiments also report GenEval, DINO, UnifiedReward, and Gemini scores, but the appendix does not assess their sensitivity to the targeted artifacts. There is no human study, no distributional metric such as FID, and no artifact-specific measurement, so the fidelity improvement rests mainly on qualitative figures. The margin over the 50-step baseline is small (HPSv2 0.4125 versus 0.4096). The same appendix analysis finds both HPSv2 and HPSv3 [6] unreliable on these pairs, yet only HPSv3 is shaded out of the main comparison, while HPSv2 remains both the training reward and the headline metric.
2. The single-stage claim is conditional in the headline setting, and the increment over the closest prior work is limited. In the main HPDv2 setting, positives are 50-step samples from the official full-step LeapAlign [4] checkpoint, so a complete RL run is still required as data preparation, in the authors' own wording a one-off data preparation. The claim of eliminating the separate stage holds unconditionally only in the GenEval and virtual dressing settings. Meanwhile the discriminator architecture and the online adversarial reward follow Adv-GRPO [1], and using a discriminator inside distillation is established in DMD2 [2]. The increment of this paper is the content-matched pair construction and its coupling with RL, which is narrower than the framing suggests.
Generalization across preference reward models also lacks quantitative validation. Both HPDv2 and GenEval use HPSv2 for training, while virtual dressing uses a DINO feature similarity reward. The OCR example in Figure 1a provides qualitative evidence, but other preference models such as PickScore and ImageReward are used only for evaluation. Scores from multiple evaluation metrics do not replace experiments that change the training reward, so the current results do not establish consistent effectiveness across preference reward models.
Questions for authors
- Human evaluation and artifact-sensitive metrics. The paper motivates the method by showing that preference rewards do not reliably identify low-level artifacts, yet the main HPDv2 results still rely on these preference metrics, and the sensitivity of the other evaluation metrics to the targeted artifacts is not established. Could you add a human pairwise study focusing on low-level artifacts and overall fidelity, and report artifact-sensitive quantitative metrics such as FID against the positive set, to directly verify the central claim that 8-step CFG-free generation matches or exceeds full-step generation.
- Generalization across training reward models. Both HPDv2 and GenEval use HPSv2 as the training reward, virtual dressing uses DINO feature similarity, and OCR has only a qualitative example. Could you train with other preference models, such as PickScore and ImageReward, using the same base model, data, sampling steps, and training budget, and compare against few-step RL without the fidelity reward and the corresponding full-step RL baseline, reporting both preference gains and artifact removal. Please specify the positive sample source and any hyperparameter adjustments for each reward setting.
- Inconsistent treatment of HPSv3 and HPSv2. Appendix A shows that both HPSv3 [6] and HPSv2 prefer artifact-laden images. Why is only HPSv3 shaded out of the main comparison, while HPSv2 remains both the training reward and the headline in-domain metric. In addition, under that analysis HPSv3 should favor artifact-laden few-step outputs, yet in Table 1 the few-step models score lower on HPSv3 than the full-step baseline. How should this ordering be interpreted.
- Dependence on a prior RL run in the HPDv2 setting. Positives there are 50-step samples from the official full-step LeapAlign checkpoint. What happens if base-anchored positives, that is full-step samples from the base model, are used on HPDv2 instead. This would directly test whether the claim of requiring no preparatory RL stage holds under a strongly stylized reward.
- Training cost. Part of the motivation is that the sequential pipeline is cumbersome. Could you report GPU-hours or wall-clock for AdvFidelity single-stage training versus the sequential pipeline (full-step RL followed by DMD) and versus RTDMD [3], including the cost of preparing target-anchored positives.
- Definition of the local fidelity loss, and generalization to other RL frameworks. Section 3.3 uses only the [CLS] token and evaluates local regions through random crops. Does the patch-level loss weight of 0.3 in Section 4.1 refer to this crop loss. Section 3.3 also states the fidelity reward is compatible with GRPO-style methods, but all experiments instantiate a ReFL-family framework. Could you show results under an advantage-based framework, and a comparison against DMDR [5] under the same setting.
- Broken citations in the appendix and the Gemini evaluation protocol. In Appendix A the citations for Aesthetic Score and CLIPScore are broken and render as question marks. Please complete them. The Gemini VLM evaluation protocol appears neither in the main text nor in the appendix, for example the judging prompt, the scoring procedure, and the sample size. Please provide these details.
References:
[1] Mao et al. The image as its own reward: Reinforcement learning with adversarial reward for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.
[2] Yin et al. Improved distribution matching distillation for fast image synthesis. In Advances in Neural Information Processing Systems, 2024.
[3] Huang et al. Reinforcing few-step generators via reward-tilted distribution matching. arXiv preprint arXiv:2605.26108, 2026.
[4] Liang et al. LeapAlign: Post-training flow matching models at any generation step by building two-step trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.
[5] Jiang et al. Distribution matching distillation meets reinforcement learning. arXiv preprint arXiv:2511.13649, 2025.
[6] Ma et al. HPSv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025.
12 comments
Response part 1/8
R3.W1 / Q1 — Metric reliability and visual fidelity
Thank you for raising this important concern about the reliability of the evaluation metrics. Appendix A examines the sensitivity of preference metrics to specific low-level artifacts in content-matched reconstructions. To complement that analysis, we have added matched 4-step and 8-step LeapAlign controls using the HPSv2 reward alone, evaluated on full generations from pure noise.
These controls show an overall pattern of higher HPSv2 scores but poorer out-of-domain evaluation performance than AdvFidelity and the official LeapAlign model trained with 25-step sampling and evaluated with 50-step inference. Figure 6 in the revised paper also shows pronounced artifacts in the few-step LeapAlign outputs, supporting the diagnosis of reward hacking.
The 8-step live/EMA comparison illustrates this effect. The live parameters achieve higher HPSv2 scores than the EMA parameters from the same training run (0.4323 vs. 0.4130), while scoring lower on PickScore (23.1283 vs. 23.7752), ImageReward (1.5363 vs. 1.5658), UR-Align (3.4077 vs. 3.4491), and UR-IQ (3.5773 vs. 3.6926). Thus, these out-of-domain metrics provide complementary signals that can partially penalize reward hacking in this setting.
The two analyses probe different aspects of metric reliability: Appendix A focuses on sensitivity to isolated low-level artifacts, while the new rollout controls test how metrics respond to the quality degradation associated with reward hacking. We therefore interpret HPSv2 as the in-domain optimization score and assess visual quality through complementary out-of-domain metrics, FID, and a blinded human GSB study.
FID is computed using 50-step FLUX outputs as the positive reference set. The 8-step AdvFidelity model achieves an FID of 50.13, compared with 54.70 for 50-step LeapAlign. Deliberately degraded reconstructions yield nearly unchanged FID scores: 55.08 at noise level 0.5 with two reconstruction steps, and 54.72 at noise level 0.3 with one step. This suggests that FID alone may not adequately capture localized artifacts.
Human raters judge AdvFidelity as better in 27%, comparable in 65%, and worse in 8% of comparisons with full-step LeapAlign. Thus, it matches or exceeds the baseline's perceived fidelity in 92% of comparisons. The visual comparisons show cleaner object boundaries and less noise in fine details while preserving complete object structures. These results support comparable or better visual quality with only 8-step inference. Additional visual comparisons are shown in Figure 8 of the supplementary material.
| Images | FID ↓ |
|---|---|
| LeapAlign 50-steps | 54.70 |
| noise 0.5 | 55.08 |
| noise 0.3 | 54.72 |
| Ours 8 step | 50.13 |
| Comparison | G (%) ↑ | S (%) | B (%) ↓ |
|---|---|---|---|
| AdvFidelity (8 steps) vs. LeapAlign (50 steps) | 27 | 65 | 8 |
R3.Q2 — PickScore as the training reward
Thank you for the constructive suggestion. Both our HPDv2 and GenEval experiments in the original submission use HPSv2 as the reward, following LeapAlign's main experimental settings. We adopt these settings to enable a fair comparison with LeapAlign (using the same reward and training/evaluation data) and reduce the effort required to reproduce the baselines. Following your suggestion, we now add experiments with PickScore to evaluate generalization across training rewards.
For the additional PickScore experiment, we follow Flow-GRPO's PickScore training-data setup: we train on prompts from the Pick-a-Pic dataset at a resolution of 512 × 512 and evaluate on DrawBench. We still use FLUX.1-dev as the base model. We also implement a PickScore-only LeapAlign baseline under this setting. We train AdvFidelity for 300 iterations and PickScore-only LeapAlign for 300 iterations. Our LoRA settings, learning rates, and hyperparameters remain the same as in the HPDv2 experiments.
We compare our 8-step AdvFidelity with 8- and 50-step PickScore-only LeapAlign baselines; the published FLUX results of Flow-GRPO are included as references. The comparisons with PickScore-only LeapAlign test the contribution of fidelity supervision under a different preference reward and compare our few-step model with the corresponding full-step RL baseline.
| Method | NFE | PickScore ↑ | HPSv2 ↑ | ImageReward ↑ | UR-Align ↑ | UR-IQ ↑ |
|---|---|---|---|---|---|---|
| Flow-GRPO | Full-step | 23.97 | — | 1.32 | — | — |
| LeapAlign | Full-step | 23.69 | 0.30 | 1.37 | 3.36 | 3.68 |
| LeapAlign | 8 | 23.86 | 0.31 | 1.29 | 3.29 | 3.62 |
| AdvFidelity | 8 | 24.24 | 0.32 | 1.37 | 3.37 | 3.69 |
Response part 2/8
R3.W2 / Q4 — Single-stage training
Thank you for highlighting this point. We understand that this concern arises from our use of images generated by a LeapAlign checkpoint in the HPDv2 experiment, which raises the question of whether our single-stage claim holds.
Our method introduces an adversarial, co-evolving fidelity reward. It requires suitable positive images from the target distribution but does not require a separate RL stage to produce them. For real-world production tasks, existing real images, Internet images, base-model samples, and ground-truth images from earlier SFT can often meet the positive-sample needs. We used RL-generated positives in HPDv2 because the reward changes the image style, the provided paired images score lower on HPSv2 than base-model generations, and we chose not to filter the HPDv2 images by HPSv2 score or collect additional positive images from the Internet.
As discussed in Section 3.4, many industrial applications, such as e-commerce and film production, seek realistic outputs. Suitable existing real images can therefore be directly reused as positives without generating additional reference images. Our virtual dressing experiment demonstrates this case. When the reward preserves the overall image style, as in instruction following, text correctness, face identity preservation, and garment consistency, base-model samples can also serve directly as positives. These are common RL objectives in production, and our GenEval experiment demonstrates the use of base-model positives.
Our choice of positives in HPDv2 reflects the requirements of this experimental setting. HPSv2 substantially changes the base model’s image style, making real images and base-model samples less suitable as positives. HPDv2 does provide paired images, but their HPSv2 scores are low—even lower than those of base-model generations, as shown in Appendix A of the revised paper—so we did not use them as positives or apply score-based filtering. We could instead filter the HPDv2 images by HPSv2 score or collect additional images from the Internet to obtain suitable positives. We therefore chose the existing LeapAlign checkpoint as a simple available way to generate positives in the target style. Specifically, we prepare 50,000 reference image–prompt pairs once using 50-step samples from the official HPSv2-aligned LeapAlign checkpoint, without reward-score filtering.
Table 4 also directly tests the effect of the reference source. The “+Adaptive” variant uses full-step base-model positives, while “+Teacher-anchored pairs” replaces them with target-style positives. Both variants use online discriminator updates and unpaired negatives, isolating the reference-source change. HPSv2 increases from 0.3558 to 0.3920 with target-style positives. Thus, the base-anchored variant operates without an RL-aligned reference model, but target-style positives provide better HPSv2 performance in this strongly stylized setting.
If the use of a LeapAlign checkpoint to generate positives still raises concerns about our single-stage claim, we see two possible ways to address them: (i) selecting suitable images from HPDv2 or collecting additional images from the Internet, and (ii) limiting our claim to single-stage generator training given suitable positive images. We would be happy to hear your suggestions.
R3.Q5 — Training cost
Thank you for pointing out the missing training-cost information. We compare the total GPU-hours of AdvFidelity, standalone LeapAlign, sequential LeapAlign + DMD, and RTDMD, with details added to Section C of the supplementary material.
All experiments use FLUX.1-dev on NVIDIA H20 GPUs at a resolution of 720 × 720, following LeapAlign’s official implementation. AdvFidelity jointly optimizes preference alignment and fidelity for 8-step generation over 400 generator updates. Standalone LeapAlign follows its official training configuration, using 25-step sampling for 300 iterations. The cost of sequential LeapAlign + DMD combines this LeapAlign training cost with the subsequent DMD distillation cost to obtain an 8-step generator. For RTDMD, we estimate the cost of obtaining an 8-step generator through 10,000 AC-DMD cold-start iterations, corresponding to 1,000 generator updates, followed by 500 joint-training iterations.
| Method | Total (GPU-hours) |
|---|---|
| AdvFidelity | 54.72 |
| LeapAlign | 47.12 |
| LeapAlign + DMD | 125.00 |
| RTDMD | 157.72 |
Thank you for your helpful suggestions. We are currently conducting additional experiments following your suggestions, such as PickScore reward experiments. We will share the results and provide a more detailed response as soon as they are available. We are sorry for the delay and appreciate your patience.