acceptodds
Under review as a conference paper at ICLR 2027

Scored Consistency Distillation: Choosing Which Teacher Trajectory to Imitate

Abstract

Few-step distillation trains a student to imitate a teacher's sampling trajectory, but which trajectory the teacher produces is decided by the seed. For a single caption a teacher generates outcomes that differ widely in how faithfully they realise the prompt, and standard distillation imitates one of them at random, inheriting the teacher's compositional failures as readily as its successes. We show that choosing the trajectory to imitate, and adding a reward on the student's own clean-latent estimates, improves compositional alignment with low online overhead. For each caption the frozen teacher samples several candidates; we select the one whose mean-pooled DINOv2 patch embedding is closest to the caption's real photograph, and distil on that trajectory alone. A lightweight projector maps the student's terminal latent directly into DINO space, so the same score applies to the student's own predictions. The projector tracks the true score on the student's recent predictions, and the two are updated on separate steps, which prevents direct co-adaptation between the reward and the policy it scores. On SD3.5-Medium distilled to four steps, the method improves T2I-CompBench++ from to and GenEval2 from to , with the largest gains on the spatial categories, followed by texture and numeracy. The four-step student surpasses the ten-step teacher it was distilled from and matches the teacher at its recommended 40-step setting, at a fraction of the inference budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Stable Diffusion, Text-to-Image Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oSKzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Scored Consistency Distillation. The main motivation is that, for the same text prompt, teacher trajectories generated with different random seeds can differ substantially in compositional quality. The authors therefore sample multiple teacher trajectories and select the best one for distillation based on the DINOv2 feature similarity between the final generated image and the paired ground-truth image.

Critical strengths

1、The paper identifies a simple but important issue in trajectory distillation. Existing methods often treat different teacher trajectories as equally suitable for distillation, while this paper shows that selecting higher-quality trajectories can significantly improve the student. This observation is useful and insightful. 2、The paper separately studies the effects of trajectory selection and reward through controlled experiments, and validates the results with multiple random seeds, large-scale training, and image fidelity metrics.

Critical weaknesses

1、The experimental coverage is still limited. The main experiments are all based on SD3.5-Medium and only study distillation to 4 steps. Since the method depends on quality differences across teacher trajectories and on the latent and image representations, it remains unclear whether the findings generalize to other models or distillation settings. An additional model or a clearly different student step count would strengthen the paper. 2、In addition to teacher trajectories, the method uses paired ground-truth images and additional supervision from DINO. Therefore, the claim that the 4-step student outperforms the teacher should not be interpreted as distillation alone extracting knowledge beyond the teacher, since the student receives external supervision that is not used during teacher sampling.

Questions for authors

1、Can these improvements be reproduced on another model or with a clearly different number of student sampling steps? 2、Could the authors clarify the meaning of "on-policy reward"? Based on Eq. (8) and Algorithm 1, the student appears to be trained on states from the selected teacher trajectory rather than from its own rollout. If so, this differs from the usual notion of on-policy distillation, where training states are sampled from the student's own distribution. 3、Given the additional use of ground-truth images and DINO supervision during training, how should the claim that the "student outperforms the teacher" be interpreted?

3 comments
AuthorsOpenReview ↗

Response to Reviewer oSKz

We thank the reviewer for recognising the central observation that teacher trajectories are not equally suitable for distillation, and for the positive view of the controlled ablations. We ran the requested second-model experiment and adopt both wording corrections.

W1 / Q1 — another model, or a clearly different step count. Done: the improvements reproduce on FLUX.2-klein-base-4B.

We trained a random-trajectory student and the full method on FLUX.2-klein-base-4B with the same 3k COCO-caption pool, 6,000 updates, four-step inference and three paired training seeds. Candidates and projector were built for FLUX.

FLUX.2-klein-base-4B, 4 stepsRandom-trajectory studentFull SCDSeed-paired changep
T2I-CompBench++ ↑0.4710 ± 0.00610.4833 ± 0.0022+0.01230.034
GenEval2 ↑18.52 ± 0.9319.71 ± 0.61+1.190.023
CMMD ↓0.763 ± 0.0640.620 ± 0.024−0.1430.025
Precision ↑0.311 ± 0.0270.378 ± 0.005+0.0670.040
Recall ↑0.122 ± 0.0110.143 ± 0.017+0.0210.033
FID ↓31.61 ± 0.5430.89 ± 1.51−0.720.391

SCD improves the results for the eight CompBench categories, with the largest increases in numeracy and texture (Table R5). Both students use four network evaluations and run at 153–155 ms per image on an H200, so the gain adds no inference module or computation. A different latent space, a different transformer, and a different teacher sampler give the same qualitative result. We kept four steps so that the backbone is the only change.

Q2 — the meaning of 'on-policy reward'. The reviewer's reading is correct, and we will change the term.

The reward scores the student's own current clean-latent estimates, but the inputs to those estimates are states of the selected teacher trajectory, not states of a student rollout. We will replace 'on-policy reward' with 'reward on current student predictions at teacher-trajectory states'. The idea remains the same: the projector is refreshed on recent student predictions, and student and projector are never updated in the same step.

W2 / Q3 — how to read 'the student outperforms the teacher'. As a statement about the resulting system. We agree it is not distillation alone, and we will say so.

The student is trained with paired photographs and a DINO score but the teacher does not use them when it samples. The claim is that this training-time supervision, which any image-caption corpus provides, is converted into better low-budget generation. Neither the photograph nor the scorer is needed at deployment. On SD3.5 the four-step student reaches 0.5036 CompBench++, against 0.4755 for the ten-step source teacher and 0.5027 for the 40-step teacher (Tables 1 and 5).

We report the second backbone the same way. On FLUX.2 the full student improves over its matched baseline but stays below its eight-step source teacher and the 50-step teacher (0.4833 against 0.5063 and 0.5207, Table R1).

We hope the second-backbone result and the two clarifications address the remaining concerns, and we would be grateful if the reviewer would consider raising the score.

Reviewer_oSKzOpenReview ↗

Official Comment by Reviewer oSKz

Thank you for the FLUX.2 experiment and the clarifications. The second-backbone results, revised reward terminology, and clearer scope of the teacher comparison address my main concerns.

We sincerely thank the reviewer for taking the time to follow up with us and for their thoughtful engagement throughout the review process. We are very glad to hear that the FLUX.2 experiment, the revised reward terminology, and the clearer scope of the teacher comparison have addressed the main concerns. Your feedback has genuinely strengthened the paper, and we will ensure all three updates are carefully incorporated into the final revised version.

Thank you once again for your time, guidance, and constructive insights. If you feel that all concerns have now been resolved, we would be deeply grateful if you would consider reflecting this in your final rating.

Sincerely,

Authors

Reviewer_drbCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Scored Consistency Distillation (SCD) for few-step text-to-image generation. For each training caption, the method generates multiple trajectories from a frozen flow teacher and selects the trajectory with the highest DINOv2 similarity to a paired reference image. The student is trained to match the teacher’s denoising estimates across different noise levels, with an additional learned projector reward encouraging semantic alignment. Experiments using SD3.5-Medium report improvements in compositional accuracy and image quality over a random-trajectory distillation baseline, enabling four-step generation. The central contribution is using reference-based trajectory selection to improve the supervision provided during distillation.

Critical strengths
  • The method uses DINOv2 similarity to select teacher trajectories, providing a straightforward way to improve distillation supervision without modifying the teacher.
  • The reported results show gains in compositional accuracy and image quality over the authors’ random-trajectory baseline with four-step inference.
  • The ablation results provide evidence that trajectory selection and the projector reward contribute to downstream performance.
  • The paper investigates which teacher trajectories are most useful for distillation, highlighting supervision quality as a complementary direction to objective and architecture design.
Critical weaknesses
  • The baseline is labeled “vanilla CD” despite using a different training objective from standard Consistency Distillation. The proposed objective matches the student’s prediction at the current state to a frozen flow teacher’s denoising estimate at a lower-noise state. This differs from standard CD, which trains a consistency function to predict the same endpoint along a sampling trajectory. The paper neither clearly identifies this modification as a separate design choice nor adequately validates it. Consequently, the comparison with “vanilla CD” demonstrates the benefits of trajectory selection and reward within the authors’ regression framework. Establishing improvements over standard CD or CTM requires comparisons under matched conditions and ablations that replace the distillation target.
  • The main results and category-level results contain unexplained numerical inconsistencies. The manuscript presents Table 2 as the category-level breakdown of Table 1, but the arithmetic means of its eight categories do not match the aggregate scores in Table 1. For example, the proposed method scores 0.5036 in Table 1, whereas its category scores in Table 2 average to 0.6071. The corresponding values for vanilla CD are 0.4463 and approximately 0.4937. The manuscript does not explain whether different aggregation weights or experimental configurations were used. These discrepancies directly affect the interpretation of the improvement magnitude and the student–teacher comparison. The experimental configurations, checkpoints, seed aggregation procedures, and aggregate-score definitions need to be specified consistently to substantiate the main performance claims.
  • The validity of DINO similarity as a criterion for selecting compositionally correct trajectories is insufficiently analyzed. DINO similarity to a reference photograph can reflect background, viewpoint, composition, and texture, alongside object identity, attributes, and spatial relationships. A higher score therefore does not necessarily distinguish faithful realization of the prompt’s compositional requirements from visual resemblance to a particular reference photograph. The ablations demonstrate downstream benefits from trajectory selection, but provide limited evidence about which errors the selection criterion actually corrects. Relevant analyses include correlations between candidate scores and compositional accuracy, controlled comparisons involving spatial relations, counts, and attribute binding, sensitivity to the choice of reference photograph, and comparisons with alternative scoring functions.
  • Missing training and evaluation details limit reproducibility and statistical interpretation. The paper does not clearly specify the student initialization, optimizer, learning rate, trainable parameters, or exact timesteps used for four-step inference. It is also unclear whether the random baseline fixes a trajectory for each caption or resamples it on each visit. Furthermore, the main results do not clearly establish whether “five seeds” refers to independent training runs or generation seeds, or how results across seeds enter the prompt-paired significance tests. These details are necessary to assess the baseline implementation, the stability of the reported improvements, and the reproducibility of the results.
Questions for authors
  • How does the proposed distillation objective relate to standard CD and CTM, given that it uses a frozen flow teacher’s denoising estimate at a lower-noise state as the target? What justifies labeling this baseline “vanilla CD”? Can the authors provide matched comparisons with standard CD or CTM, including an ablation that changes the distillation target while keeping trajectory selection and the projector reward fixed?
  • How are the aggregate scores in Table 1 computed from the category-level results in Table 2?
  • What evidence shows that higher DINO similarity to a reference photograph identifies trajectories with greater compositional accuracy? Can the authors measure the relationship between candidate scores and correctness on spatial relations, counting, and attribute binding, assess sensitivity to the reference photograph, and compare DINO similarity with alternative scoring functions?
  • Can the authors specify the student initialization, optimizer, learning rate, trainable parameters, and four-step inference timesteps?
5 comments
AuthorsOpenReview ↗

Response to Reviewer drbC

We thank the reviewer for the detailed review and for recognising supervision quality as a direction complementary to objective and architecture design. The four weaknesses concern the baseline's name, the bookkeeping of two tables, the analysis of the criterion, and missing details. We answer each below; none of them changes a reported number.

W1 / Q1 — the 'vanilla CD' label, and the relation to standard CD and CTM. We agree on the label and will rename it. The comparison is controlled, and the baseline is as strong as an established consistency recipe on the same backbone.

Relation to CD and CTM. In Eqs. (7)–(9) the student's clean estimate at a noisier state is regressed onto the frozen teacher's clean estimate at a cleaner state of the same teacher trajectory, under a stop-gradient and the Pseudo-Huber metric of Song & Dhariwal (2024). Standard CD and CTM use the target from a stop-gradient or EMA copy of the student after a teacher solver step, SCD takes it from the frozen teacher. Section 2.2 presents this as a deliberate choice (the target never depends on the student, so the distillation term is stable by construction), but the baseline's name does not reflect it. We will rename the baseline 'random-trajectory regression' throughout and state the difference from CD and CTM in Section 3.2.

What the controlled comparison isolates. The two arms share the objective, seeds, data order, schedule, supervision window and gap distribution, and differ only in which candidate is imitated and in the reward term (Section 4). The 2 × 2 factorial separates the two: 0.4751 ± 0.0032 with neither, 0.4819 with the reward, 0.4868 with selection, 0.4923 ± 0.0021 with both, over five training seeds per arm (Table 7).

Matched comparison with established consistency training. We ported the four-phase adversarial PCM recipe, whose target is the online network under stop-gradient, to SD3.5-Medium and trained it on the same 3k caption pool with three seeds (Table R2). PCM reaches 0.4650 ± 0.0103 CompBench++ and 16.97 on GenEval2 at four evaluations. Additionally, our regression baseline reaches 0.4738 ± 0.0017 and 22.5, and the full method 0.4887 ± 0.0025 and 23.5 (paper Table 6).

Selection inside the PCM objective.

PCM port, SD3.5-MediumEvaluationsRandom endpoints: CompBench++ / GenEval2Selected endpoints: CompBench++ / GenEval2
Four steps, guidance 1.040.4650 / 16.970.4646 / 16.77
Four steps, guidance 1.2 (official)80.4724 / 17.670.4730 / 17.05

Selection alone has no detectable aggregate effect in PCM. The likely reason is structural: in PCM the selected endpoint only supplies the clean latent that is noised into training states, while the target comes from the online network. In our objective the selected trajectory is the target itself, and the selection alone is worth +0.0117. Our claim is correspondingly scoped: selection and the reward improve trajectory-imitation distillation, on SD3.5-Medium and on FLUX.2 (Table R1).

W2 / Q2 — Table 1 aggregates do not equal the mean of Table 2's categories. The two tables report different configurations and different non-spatial evaluators. Nothing in Table 1 is computed from Table 2.

Table 1 uses the 5k COCO-caption training configuration; its aggregate is the unweighted mean of the eight category scores, with the non-spatial category scored by CLIPScore. Table 2 uses a 3k configuration and scores the non-spatial category with Share-CoT, as its column header states, following the published CTCal setting it is compared with. The sentence calling Table 2 'the per-category breakdown' of Table 1 is our error, and we will correct it.

The aggregate definition can be checked inside the paper. In Table 11, the Mean row equals the unweighted mean of the eight categories for both models (0.4578 and 0.4930), with CLIPScore non-spatial scores near 0.31. Share-CoT scores are on a different scale (0.74–0.80 in Table 2), which is why averaging Table 2 gives the larger values the reviewer computed. The new tables follow the same rule: the eight FLUX.2 categories of Table R5 average exactly to the 0.4710 and 0.4833 of Table R1.

The matched comparison is 0.4463 vs 0.5036 under Table 1's protocol. The student-teacher comparison (0.5036 against 0.4755 and 0.5027) is made entirely within that protocol as well (Tables 1 and 5). In the revision, every caption will name its configuration and evaluator.

(response continued in the next comment)

W3 / Q3 — is DINO similarity to a photograph a valid criterion for compositional correctness? The criterion only has to rank the candidates of one caption, and the evidence that it ranks them usefully comes from evaluators that never see the photograph.

(1) Only the ranking is used. Best-of-J selection is invariant to any increasing transformation of the score (Eq. 5). Resemblance in background or viewpoint adds noise to the ranking, but it cannot produce gains on benchmark prompts that have no reference photograph and are graded by B-VQA, UniDet and the GenEval2 evaluator, none of which involves DINO.

(2) Selection alone, with the reward off, improves held-out compositional accuracy from 0.4751 → 0.4868 with p=5×10−6p = 5 \times 10^{-6} (Table 7). Weighting all four candidates uniformly gives +0.0004 (Table 12). The gain therefore comes from which candidate the score picks.

(3) Which errors it corrects. The pattern is the one Section 3.1 predicts for an appearance-based score. At 118k captions (Table 11): 2D-spatial +0.0495, 3D-spatial +0.0480, numeracy +0.0528, texture +0.0505 and colour +0.0468, each with p ≤ 0.05, while shape and non-spatial relations do not change significantly. On GenEval2 (Table 4): object 78.5 → 86.6 and count 34.4 → 41.6, both p < 0.001, while position and verbs do not change significantly. Spatial layout on CompBench, counts and attribute binding, the three cases the reviewer names, are where selection helps; relations that are not visible in appearance are where it does not, as Section 3.1 and the Limitations say.

(4) The signal transfers. On FLUX.2 all eight category means increase (Table R5).

A candidate-level correlation between DINO rank and evaluator correctness, a reference-substitution test, and text-aware scorers. Because selection consumes only a ranking, each is a drop-in analysis; we will add the candidate-level analysis and CLIPScore / VQAScore selectors to Table 12.

W4 / Q4 — student initialisation, optimiser, learning rate, trainable parameters, sampling timesteps, baseline policy and seeds. All specified below and, for every configuration, in Table R7.

SettingSD3.5-Medium student (3k configuration)
Student initialisationTeacher weights
Trainable parametersFull transformer
OptimiserAdamW, β = (0.9, 0.999), weight decay 0
Learning rate1e-5, 300 warm-up updates, then constant
Batch / gradient clippingOne caption / 1.0
Updates / Steps6,000 updates
Four-step sampling sigmas[1, 0.9, 0.75, 0.5, 0], guidance 1
Random baselineOne uniform candidate draw per caption, cached; fixed across visits and training seeds

Seeds. The five seeds of the submitted main and factorial studies are independent training runs, not generation seeds of one model. The random baseline uses one cached candidate draw per caption, fixed across seeds, so paired comparisons do not confound selection with a changing random-target policy. In the new PCM and FLUX.2 studies, generation noise is seeded by prompt index and shared across models, and p-values are two-sided paired t-tests on the three per-seed scores. Weight averaging happens within a run and adds no statistical replicates.

Regarding the statistical tests, we have five models for five different random seeds and each model gets one score per metric. Runs of one arm are then compared with runs of the other arm. That gives five seed-level differences and a two-sided paired t-test with n = 5. All models use the same generation noise for each prompt.

Reward weight. It follows the calibration rule of the paper, not benchmark tuning: on FLUX.2 the reward-to-consistency gradient ratio at coefficient 1 was 0.00154, which gives 130 for the 0.20 target.

We hope the renamed baseline, the PCM comparison, the table clarification and the full configurations resolve the soundness concerns, and we would be grateful if the reviewer would reconsider the rating.

Thank you for the detailed responses and additional experiments. Most of my concerns have been addressed, and I am considering raising my score. I would appreciate clarification on two remaining points.

  • Could the authors provide the category-level CompBench++ results corresponding to Table 1’s main 5k-caption setting? Since Table 2 reports a different configuration and evaluator, it would be helpful to see the breakdown underlying the main aggregate scores.

  • I would like to better understand the implications of the distillation objective. CD, CTM, and flow-map approaches aim to learn consistency along teacher trajectories or approximate the teacher’s ODE solution map, with theoretical guarantees under appropriate assumptions. In contrast, SCD regresses toward the frozen teacher’s clean-data estimate at a lower-noise intermediate state, which need not coincide with the trajectory endpoint. The connection between this objective and preservation of the teacher’s distribution therefore seems less clear. Given the empirical improvements, I do not consider the absence of such a guarantee, by itself, a decisive weakness. However, I am interested in why even the random-trajectory regression baseline outperforms the PCM port, and whether this involves any trade-offs in sampling flexibility or diversity. Could the authors report results at sampling step counts other than four, together with a comparison of sample diversity against PCM under comparable evaluation conditions?

Reviewer_eTyXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the few-step distillation of the T2I models by proposing the trajectory/roll-out sampling-based approach. Specifically, the teacher generates N rollouts. DINO patch-based similarity finds the closest rollout. And the consistency distillation uses this selected trajectory as target. Additionally, a decode-free reward is introduced that maps the latents to DINO space and uses this as reward for the optimization. Experiments on T2I compbench++ and GenEval shows the improvements compared to the baseline.

Critical strengths
  • The idea is straightforward and well motivated. Specifically, selecting the right target plays a crucial role for distillation to avoid distilling bad trajectories.
  • Experimental results show improvements over vanilla consistency distillation. Additionally, multiple seed experiments are conducted and consistently support the improvements over the baseline.
  • The paper is well written and easy to follow.
Critical weaknesses
  • Experiments lack other state-of-the-art baselines. Such as DMD2, Hyper-SD, TCD, or PCM-style recipes. It is important to benchmark against these methods.
  • Additionally, all experiments are on SD3.5, which is very old model and hard to gauge the proposed method's usefulness on SoTA models such as Qwen-Image-2/Flux.2/etc.
  • The proposed method is essentially rejection sampling. The importance of DINO-based rejection sampling requires further study on other potential reward model designs.
    • A text-aware scorer (VQAScore, ImageReward, HPS, PickScore, CLIPScore) can also judge semantics better, which is exactly where DINO is weak.
Questions for authors
  • Trajectory sampling is orthogonal to the other baselines such as DMD2. What if we use a selection process to guide the other distillation methods?
  • Can we ablate the different rejection sampling methods to nail down whether a pair-based DINO setup is really needed?
  • How does the 4-step student compare with established few-step SD3/SD3.5 distillations?
  • How do quality and offline cost change for J ∈ {2, 4, 8, 16}? Any rollout scaling study?
3 comments
AuthorsOpenReview ↗

Response to Reviewer eTyX

We thank the reviewer for the positive assessment of the motivation and of the multi-seed evidence. The review asks for stronger baselines, a newer backbone and a closer look at the scoring rule. We ran the first two during the rebuttal, and we answer the third with ablations that are already in the appendix, stating plainly what is not yet covered.

W1 / Q3 — state-of-the-art baselines, and established few-step SD3 / SD3.5 distillations. Done for two of the four named families: PCM on the same backbone, and the released DMD2.

PCM, same backbone. We could not find a PCM checkpoint for SD3.5-Medium, so we used the four-phase adversarial PCM recipe to it with the official hyperparameters (deviations are listed in Table R7) and trained three seeds on the same 3k COCO-caption pool as our ablations.

SD3.5-Medium, 3k captions, 4 stepsEvaluationsCompBench++ ↑GenEval2 ↑CMMD ↓
PCM port, guidance 1.040.4650 ± 0.010316.97 ± 0.850.709 ± 0.021
PCM port, guidance 1.2 (official)80.4724 ± 0.012217.67 ± 0.750.705 ± 0.004
Our baseline (random trajectory, no reward)40.4738 ± 0.001722.5 ± 0.90.837
Full SCD40.4887 ± 0.002523.5 ± 1.60.817

PCM rows: Table R2. Our rows: paper Table 6 (3k configuration). Three training seeds each, one image per prompt.

At equal cost, full SCD is better +0.024 CompBench++ and +6.5 GenEval2 compared to PCM, and it stays better than PCM at PCM's official guidance setting, which costs eight evaluations (0.4887 vs 0.4724). Our random-trajectory baseline alone already matches PCM on CompBench++ and exceeds it by 5.5 GenEval2 points, so the gains in the paper are measured against a competitive baseline. PCM's adversarial heads give it the lower CMMD on this pool (0.709 vs 0.817).

DMD2, released four-step model.

Four-step systemCompBench++ ↑GenEval2 ↑CMMD ↓FID ↓
DMD2-SDXL (released)0.440810.070.74425.17
Ours (paper Table 1)0.503623.700.70126.80

Our student is better on both alignment benchmarks, by 0.0628 CompBench++ and 13.63 GenEval2 points, with lower CMMD. DMD2 has the better FID, precision and recall. Because the backbone differs, we read this as a system-level reference and PCM as the controlled one. Hyper-SD and TCD are not included in this round.

W2 — SD3.5 is an old model; usefulness on newer models such as FLUX.2. Done: FLUX.2-klein-base-4B, three paired training seeds.

We trained a matched random-trajectory student and the full method on FLUX.2-klein-base-4B, with candidates and projector built for FLUX and the reward weight set by the same gradient-ratio rule as in the paper.

FLUX.2-klein-base-4B, 4 stepsRandom-trajectory studentFull SCDSeed-paired changep
T2I-CompBench++ ↑0.4710 ± 0.00610.4833 ± 0.0022+0.01230.034
GenEval2 ↑18.52 ± 0.9319.71 ± 0.61+1.190.023
CMMD ↓0.763 ± 0.0640.620 ± 0.024−0.1430.025
Precision ↑0.311 ± 0.0270.378 ± 0.005+0.0670.040
Recall ↑0.122 ± 0.0110.143 ± 0.017+0.0210.033
FID ↓31.61 ± 0.5430.89 ± 1.51−0.720.391

Both alignment benchmarks and three of the four fidelity metrics improve at the significant level, and the result for the average of the eight CompBench category means improves (Table R5). Both students share the inference path: four network evaluations and 153–155 ms per image on an H200. The benefit therefore extends to a current model family at an unchanged inference budget.

W3 / Q2 — 'essentially rejection sampling'; other scorer designs; is a pair-based DINO score needed? Selection happens once, at training time, and the appendix already ablates the alternatives to it.

Not test-time rejection sampling. The student generates one sample with four evaluations, with no scorer, no reference photograph and no rejected images. Test-time selection is a separate axis that compounds with ours: with an oracle choosing among each model's own samples, our student reaches 0.6052 at best-of-4 and 0.6482 at best-of-10, against 0.5592 and 0.6113 for the baseline student (Table 5).

Not only selection. The reward on the student's own clean estimates adds +0.0068 on its own and brings the total to +0.0172; a two-way analysis over 20 runs gives significant main effects for selection (p = 5×10−65 \times 10^{-6}) and for the reward (p = 0.002) (Table 7).

(response continued in the next comment)

Alternative selection rules.

Selection rule (paper Table 12, 3k)CompBench++ ↑Change vs random (p)
Random draw (baseline)0.4737 ± 0.0017–
Argmax (ours)0.4873 ± 0.0032+0.0136 (0.03)
Boltzmann, exact weighting, T = 0.04 (3× compute)0.4840 ± 0.0030+0.0102 (0.03)
Boltzmann, sampled per visit, T = 0.040.4831 ± 0.0043+0.0094 (0.07)
Uniform weighting of all J candidates0.4741 ± 0.0054+0.0004 (0.93)
Latent projector score, argmax (decode-free)0.4840 ± 0.0058+0.0103 (0.14)

No rule improves on argmax, and exact Boltzmann weighting costs three times the training compute. Showing the student all four candidates without a score recovers no useful gain (+0.0004), so the gain comes from the ordering the score provides, not from exposure to more teacher samples. A decode-free latent selector recovers about three quarters of the gain.

Why a text-free, pair-based score. There are three reasons.

(i) The selection score is independent of the models that grade the benchmarks. Both benchmarks are scored automatically by other models: T2I-CompBench++ uses a VQA model, an object detector and CLIPScore, and GenEval2 uses a VQA-based evaluator. The text-aware scorers the reviewer lists (VQAScore, CLIPScore, ImageReward, HPS, PickScore) are built on the same kinds of models. If we selected training trajectories with one of them, the student would be trained to satisfy the same kind of model that later grades it, and a higher benchmark score could reflect that overlap instead of better images. DINOv2 is a vision-only model trained without text, and neither benchmark uses it. The gains we report therefore cannot come from such overlap.

(ii) The score needs nothing beyond the training set. It compares each candidate image with the photograph already paired with the caption. Every image-caption dataset provides that photograph, so no extra annotation and no trained reward model is required, and scoring a candidate is a single DINO forward pass.

(iii) The same score also gives us the reward. The reward has to be computed on the student's latent prediction at every training step, without decoding it to an image. A small projector network can be trained to predict the DINO score directly from the latent, and that projector is the reward used in the paper. One criterion therefore drives both components of the method.

Section 3.1 states where the score is weak (relations that are not visible in appearance), and the non-spatial category is where the gain is smallest. We do not claim that DINO is the only possible scorer. Selection consumes only a ranking (Eq. 5), so a text-aware scorer is a drop-in replacement in Eq. 6. We will add CLIPScore and VQAScore selectors to Table 12.

Q1 — can selection guide other distillation methods, such as DMD2? We tested this inside PCM. Selection alone does not change the result there, which is expected from how PCM builds its training target.

We trained the PCM port twice, with three seeds each: once with a randomly chosen teacher endpoint per caption and once with the DINO-selected endpoint. Everything else is identical.

PCM port, SD3.5-MediumEvaluationsRandom endpoints: CompBench++ / GenEval2Selected endpoints: CompBench++ / GenEval2
Four steps, guidance 1.040.4650 / 16.970.4646 / 16.77
Four steps, guidance 1.2 (official)80.4724 / 17.670.4730 / 17.05

From Table R2. Three training seeds per arm; no projector reward in either arm.

PCM training takes a clean image latent, adds noise to it, and trains the network so that its prediction from this noisy input agrees with its own prediction from a slightly less noisy point, reached by one teacher solver step (Table R7). In our port, the clean latent is the endpoint of a teacher trajectory, either random or selected. Selection therefore changes only which clean latent is noised, that is, the input data. The target is still produced by the network itself and is not affected by the selection. In trajectory imitation (Eq. 4), the student is trained to match the teacher's estimates along the selected trajectory, so the selected trajectory is the target at every supervised state, and selection alone gains +0.0117 (Table 7). This is exactly the paper's claim: what matters is which teacher trajectory the student imitates.

(response continued in the next comment)

Q4 — quality and offline cost for J ∈ {2, 4, 8, 16}. Offline cost is linear in J; training and inference cost do not depend on J. A student-quality sweep over J is not in this round.

Cost. Building the pool takes 2JK teacher evaluations, J decodes and J + 1 DINO calls per caption, once (Table 8). Measured on FLUX.2 (Table R6): 16.1 GPU-hours for the whole 27k-caption pool at J = 4, 0.12 GPU-hours for the projector, and 2.13 (baseline) against 2.17 (full method) GPU-hours of student training. By linearity, J = 2, 8 and 16 scale the first figure by 0.5, 2 and 4 and leave the others unchanged.

Quality. Two results in the paper bound what a larger J can buy. The selected distribution departs from the teacher's by log J − (J − 1)/J, that is 0.64 nats at J = 4, 1.20 at J = 8 and 1.83 at J = 16, so the supervision shifts only logarithmically (Appendix H). All reported students use J = 4; we will add the J sweep to Appendix F.

We hope the same-backbone PCM comparison, the FLUX.2 results and the selection-rule evidence address the main concerns, and we would be grateful if the reviewer would reconsider the rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers for their careful reading. The reviews ask for a newer backbone, established few-step baselines, a test of selection inside another distillation objective, and complete training and evaluation details. We ran three new experiments and documented every configuration; full results are in Tables R1–R7 in the comment below. New experiments use three independent training seeds, and ± is the standard deviation over those seeds.

New backbone (Tables R1, R5). On FLUX.2-klein-base-4B the full method improves T2I-CompBench++ from 0.4710 to 0.4833 (p = 0.034) and GenEval2 from 18.52 to 19.71 (p = 0.023), with better CMMD, precision and recall, on an identical four-evaluation inference path.

Established baselines (Tables R2, R3). A four-phase adversarial PCM port on the same SD3.5-Medium backbone reaches 0.4650 CompBench++ and 16.97 GenEval2 at four evaluations, against 0.4887 and 23.5 for our full method on the same caption pool. The released four-step DMD2-SDXL reaches 0.4408 and 10.07, against 0.5036 and 23.70 for our four-step student.

Selection inside another objective (Table R2). Endpoint selection alone does not change the PCM port (0.4646 vs 0.4650). We report this result as it is and explain why it is expected for an objective whose target is not the teacher trajectory.

Reproducibility (Tables R6, R7). Initialisation, optimiser, learning rate, trainable parameters, sampling grids, seed protocol and measured costs are now specified for every configuration.

New results: Tables R1–R7

Table R1. New-backbone experiment: FLUX.2-klein-base-4B

Students: 3k COCO captions; four FLUX candidates per caption generated with eight Euler steps and guidance 4.0; 6,000 updates; three training seeds. Student inference uses four steps and guidance 1. Teacher rows use the same 512-pixel evaluation. Network evaluations count conditional and unconditional branches.

ModelNetwork evaluationsCompBench++ ↑GenEval2 ↑CMMD ↓Precision ↑Recall ↑FID ↓
Random-trajectory student40.4710 ± 0.006118.52 ± 0.930.763 ± 0.0640.311 ± 0.0270.122 ± 0.01131.61 ± 0.54
Full SCD40.4833 ± 0.002219.71 ± 0.610.620 ± 0.0240.378 ± 0.0050.143 ± 0.01730.89 ± 1.51
Teacher: 50 steps, guidance 41000.520722.300.5800.4010.26828.26
Teacher: 8 steps, guidance 4160.506321.670.5180.4320.23529.79
Teacher: 4 steps, guidance 140.21945.721.7980.2810.00767.09
Seed-paired change, SCD − random—+0.0123 ± 0.0040+1.19 ± 0.32−0.143 ± 0.040+0.067 ± 0.024+0.021 ± 0.007−0.72 ± 1.15
Paired p-value, n = 3—0.0340.0230.0250.0400.0330.391

Table R2. PCM consistency-objective experiment on SD3.5-Medium

Our port of the four-phase adversarial PCM recipe to SD3.5-Medium: LoRA training; 3k COCO captions; 20,000 iterations containing 10,000 LoRA updates; final LoRA; one image per CompBench prompt; training seeds 0–2. No projector reward in either arm. Random vs. selected differ only in whether a random or a DINO-selected teacher endpoint supplies the clean latent.

Guidance / evaluationsConditionCompBench++ ↑GenEval2 ↑CMMD ↓Precision ↑Recall ↑FID ↓
1.0 / 4random0.4650 ± 0.010316.97 ± 0.850.709 ± 0.0210.506 ± 0.0220.107 ± 0.00127.50 ± 1.27
1.0 / 4selected0.4646 ± 0.006116.77 ± 1.150.705 ± 0.0400.515 ± 0.0360.109 ± 0.00927.90 ± 0.63
1.2 / 8random0.4724 ± 0.012217.67 ± 0.750.705 ± 0.0040.528 ± 0.0120.102 ± 0.00329.17 ± 1.05
1.2 / 8selected0.4730 ± 0.005517.05 ± 0.930.707 ± 0.0350.536 ± 0.0300.104 ± 0.00829.70 ± 0.63

Table R3. DMD2 and the SD3.5 system of the paper

System-level comparison. CompBench++ uses ten images per prompt. All rows use four network evaluations.

SystemCompBench++ ↑GenEval2 ↑CMMD ↓Precision ↑Recall ↑FID ↓
DMD2-SDXL0.440810.070.7440.5000.16425.17
SD3.5 random-trajectory regression (paper Table 1, 'vanilla CD')0.446321.270.9550.4360.04633.16
Full SCD, ours (paper Table 1)0.503623.700.7010.4810.12226.80

Table R4. Component isolation (paper Table 7)

COCO 3k configuration, four-step students, 6,000 updates, five independent training seeds per arm.

DINO trajectory selectionProjector rewardCompBench++ ↑
OffOff0.4751 ± 0.0032
OffOn0.4819 ± 0.0060
OnOff0.4868 ± 0.0031
OnOn0.4923 ± 0.0021

Table R5. FLUX.2 results per category

Means over three independently trained students. Two-sided paired t-tests. All eight mean changes are positive.

CategoryRandom studentFull SCD
Colour0.75750.7654
Shape0.49240.5021
Texture0.65780.6765
Spatial0.31320.3242
3D spatial0.33670.3433
Numeracy0.53900.5742
Non-spatial0.30750.3096
Complex0.36380.3714

Table R6. Cost and inference measurements

ItemReported measurementScope
FLUX random-student training2.15 / 2.13 / 2.12 GPU-hoursSeeds 0 / 1 / 2; H200; student training
FLUX full-SCD training2.18 / 2.17 / 2.16 GPU-hoursSeeds 0 / 1 / 2; H200; student training
FLUX candidate cache16.1 GPU-hoursEntire 27k-caption pool × four candidates
FLUX projector preparation0.12 GPU-hoursSeparate from student training
FLUX reward-calibration probe0.03 GPU-hoursSeparate 40-update probe
FLUX random/full student latency153–155 ms/imageBatch 1, H200; denoising + VAE; text encoding excluded
Released DMD2 latency342 ms/imageBatch 1, H100; includes text encoding + VAE
FLUX teacher, 50 steps / guidance 43479 ms/imageBatch 1, H100; denoising + VAE
FLUX teacher, 8 steps / guidance 4572 ms/imageBatch 1, H100; denoising + VAE

Every training job used one GPU, so training GPU-hours equal training-loop wall-clock hours (checkpoint writing included, model loading and evaluation excluded). The 16.1 GPU-hour cache covers 3k student-training, 22k projector-training and 2k projector-selection captions. Only the student-to-student latency is hardware-matched; teacher and DMD2 timings use different hardware and scopes and are not a measured speed-up.

Table R7. Training and sampling configurations

The first column is the SD3.5-Medium 3k configuration; the other columns are the logged arguments of the new runs.

SettingSD3.5-MediumPCM port on SD3.5-MediumFLUX.2-klein-base-4B (random / full SCD)
Student initializationTeacher weightsTeacher weights plus seeded LoRA; same initialization in both armsTeacher weights
Trainable parametersFull transformerRank-32 LoRA on 22 module patterns; discriminator headsFull transformer
OptimizerAdamW; β = (0.9, 0.999); weight decay 0AdamW; weight decay 1e-3; discriminator-head β = (0, 0.999)AdamW; weight decay 0
Learning rate1e-5; 300 warm-up updates, then constantLoRA 5e-6 constant; heads 1e-51e-5; 300 warm-up updates, then constant
Training duration6,000 updates20,000 iterations, containing 10,000 LoRA updates6,000 updates
Batch / gradient clippingOne caption / 1.0Two / 1.0One caption / 1.0
Target networkFrozen teacher's clean estimate; no EMAOnline LoRA network under stop-gradient; no EMAFrozen teacher's clean estimate; no EMA
Teacher guidance73 in the solver step; endpoints from the guidance-7 cache4.0
Random-trajectory policyOne uniform draw per caption, cached; fixed across visits and seedsSame cached draw; use its candidate endpointSame rule on the separate FLUX cache
Four-step sampling sigmas[1, .9, .75, .5, 0][1, .9048, .75, .5132, 0][1, .9581, .884, .7175, 0]
Student sampling guidance11.0 and 1.21
AuthorsOpenReview ↗

Gentle reminder to Reviewers eTyX and drbC

Dear Reviewers eTyX and drbC,

Thank you again for your detailed reviews. As the discussion period is closing soon, we would like to kindly draw your attention to our responses and the new results posted with them. We would greatly appreciate it if you could let us know whether these responses address your concerns, and we are glad to provide any further clarification in the time remaining.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.