- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Scored Consistency Distillation. The main motivation is that, for the same text prompt, teacher trajectories generated with different random seeds can differ substantially in compositional quality. The authors therefore sample multiple teacher trajectories and select the best one for distillation based on the DINOv2 feature similarity between the final generated image and the paired ground-truth image.
Critical strengths
1、The paper identifies a simple but important issue in trajectory distillation. Existing methods often treat different teacher trajectories as equally suitable for distillation, while this paper shows that selecting higher-quality trajectories can significantly improve the student. This observation is useful and insightful. 2、The paper separately studies the effects of trajectory selection and reward through controlled experiments, and validates the results with multiple random seeds, large-scale training, and image fidelity metrics.
Critical weaknesses
1、The experimental coverage is still limited. The main experiments are all based on SD3.5-Medium and only study distillation to 4 steps. Since the method depends on quality differences across teacher trajectories and on the latent and image representations, it remains unclear whether the findings generalize to other models or distillation settings. An additional model or a clearly different student step count would strengthen the paper. 2、In addition to teacher trajectories, the method uses paired ground-truth images and additional supervision from DINO. Therefore, the claim that the 4-step student outperforms the teacher should not be interpreted as distillation alone extracting knowledge beyond the teacher, since the student receives external supervision that is not used during teacher sampling.
Questions for authors
1、Can these improvements be reproduced on another model or with a clearly different number of student sampling steps? 2、Could the authors clarify the meaning of "on-policy reward"? Based on Eq. (8) and Algorithm 1, the student appears to be trained on states from the selected teacher trajectory rather than from its own rollout. If so, this differs from the usual notion of on-policy distillation, where training states are sampled from the student's own distribution. 3、Given the additional use of ground-truth images and DINO supervision during training, how should the claim that the "student outperforms the teacher" be interpreted?
3 comments
Official Comment by Reviewer oSKz
Thank you for the FLUX.2 experiment and the clarifications. The second-backbone results, revised reward terminology, and clearer scope of the teacher comparison address my main concerns.
Thank you
We sincerely thank the reviewer for taking the time to follow up with us and for their thoughtful engagement throughout the review process. We are very glad to hear that the FLUX.2 experiment, the revised reward terminology, and the clearer scope of the teacher comparison have addressed the main concerns. Your feedback has genuinely strengthened the paper, and we will ensure all three updates are carefully incorporated into the final revised version.
Thank you once again for your time, guidance, and constructive insights. If you feel that all concerns have now been resolved, we would be deeply grateful if you would consider reflecting this in your final rating.
Sincerely,
Authors
Response to Reviewer oSKz
We thank the reviewer for recognising the central observation that teacher trajectories are not equally suitable for distillation, and for the positive view of the controlled ablations. We ran the requested second-model experiment and adopt both wording corrections.
W1 / Q1 — another model, or a clearly different step count. Done: the improvements reproduce on FLUX.2-klein-base-4B.
We trained a random-trajectory student and the full method on FLUX.2-klein-base-4B with the same 3k COCO-caption pool, 6,000 updates, four-step inference and three paired training seeds. Candidates and projector were built for FLUX.
SCD improves the results for the eight CompBench categories, with the largest increases in numeracy and texture (Table R5). Both students use four network evaluations and run at 153–155 ms per image on an H200, so the gain adds no inference module or computation. A different latent space, a different transformer, and a different teacher sampler give the same qualitative result. We kept four steps so that the backbone is the only change.
Q2 — the meaning of 'on-policy reward'. The reviewer's reading is correct, and we will change the term.
The reward scores the student's own current clean-latent estimates, but the inputs to those estimates are states of the selected teacher trajectory, not states of a student rollout. We will replace 'on-policy reward' with 'reward on current student predictions at teacher-trajectory states'. The idea remains the same: the projector is refreshed on recent student predictions, and student and projector are never updated in the same step.
W2 / Q3 — how to read 'the student outperforms the teacher'. As a statement about the resulting system. We agree it is not distillation alone, and we will say so.
The student is trained with paired photographs and a DINO score but the teacher does not use them when it samples. The claim is that this training-time supervision, which any image-caption corpus provides, is converted into better low-budget generation. Neither the photograph nor the scorer is needed at deployment. On SD3.5 the four-step student reaches 0.5036 CompBench++, against 0.4755 for the ten-step source teacher and 0.5027 for the 40-step teacher (Tables 1 and 5).
We report the second backbone the same way. On FLUX.2 the full student improves over its matched baseline but stays below its eight-step source teacher and the 50-step teacher (0.4833 against 0.5063 and 0.5207, Table R1).
We hope the second-backbone result and the two clarifications address the remaining concerns, and we would be grateful if the reviewer would consider raising the score.