- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates teacher–student feature discrepancies across Transformer layers and denoising timesteps in multi-teacher on-policy distillation for diffusion models. Based on the observation that larger discrepancies tend to appear in deeper layers and early denoising steps, the authors propose FocusOPD, which introduces directional feature alignment in these selected regions. Experiments on SD3.5-M and Z-Image show improvements over several existing OPD methods.
Critical strengths
The paper studies an interesting question about where and when feature supervision should be applied during on-policy distillation. The analysis of teacher–student discrepancies provides a reasonable motivation for the proposed method, and the selective supervision strategy is straightforward to implement. Experiments across two diffusion backbones and several OPD baselines, together with ablations on layer and timestep selection, provide useful empirical observations.
Critical weaknesses
-
The connection between feature discrepancy and distillation effectiveness is not fully established. The main motivation is that larger teacher–student discrepancies indicate where additional supervision would be more useful. However, a larger feature mismatch does not necessarily mean that aligning those features will improve generation quality. The theoretical analysis also relies on assumptions about gradient compatibility that are not directly verified in the experiments. Furthermore, the mismatch analysis uses normalized MSE, while the proposed method uses cosine similarity. It would be useful to examine whether the same depth–temporal patterns hold for the actual cosine-based supervision objective.
-
The improvements over existing OPD methods are relatively small in several settings. In Table 2, FocusOPD improves DiffusionOPD on SD3.5-M, but the additional gains over STEP-OPD are quite small. For example, GenEval increases from 0.963 to 0.967, while PickScore slightly decreases from 24.048 to 24.025. Similar observations can be made on Z-Image, where STEP-OPD improves from 0.946 to 0.948 on GenEval. Moreover, some metrics decrease after adding FocusOPD, particularly OCR accuracy and certain preference scores. I would like to see more discussion of these trade-offs and whether the improvements are consistent across different training seeds.
-
The benefit of the proposed selection strategy could be further clarified. The ablation studies show that supervising deep layers during early denoising is generally more effective than supervising shallow layers or the full trajectory. However, Table 4 also shows that supervising all blocks achieves a higher GenEval score than the selected Last-12 configuration. Since the selection strategy is central to the paper, comparisons under matched supervision budgets would be useful. For example, randomly selected layers or timesteps with the same number of supervised positions could help determine whether the improvement comes from the proposed depth–temporal structure or simply from using fewer auxiliary constraints.
-
The additional training cost. Although FocusOPD introduces no additional inference cost, it requires teacher feature extraction, storage, and additional training computation. Table 3 shows that training time per update increases from 16.54s to 18.99s, together with an additional 432 MiB feature cache. Given the relatively small improvements on some benchmarks, a more detailed discussion of the training cost and performance trade-off would help assess the practical benefit of the method.
Questions for authors
see weakness
4 comments
Cosine vs. MSE, random controls, and what the extra training time buys
W2. Improvements over existing OPD methods are small in several settings, especially over STEP-OPD, and some OCR and preference metrics decrease. Please discuss the capability trade-offs and consistency across training seeds.
For the gains over STEP-OPD. GenEval and OCR are bounded by 1, and the SD3.5-M STEP-OPD baseline already scores 0.963 and 0.961, so absolute gains are compressed in this regime. Within it, FocusOPD raises GenEval to 0.967, driven by the compositional-attribute category (0.900→0.930); on Z-Image, GenEval rises 0.946→0.948. STEP-OPD also supervises representation dynamics, so its incremental gains are smaller than over output-only baselines: adding FocusOPD to DiffusionOPD gives +0.014/+0.002/+0.147 on SD3.5-M (Tab. 2).
For consistency across seeds, we retrain each baseline three times (42/43/44). For DiffusionOPD the three-seed averages are +0.015±0.005 (GenEval), +0.003±0.003 (OCR), and +0.143±0.014 (PickScore), consistent with the single-run +0.014/+0.002/+0.147 in Tab. 2 within seed SD. For STEP-OPD:
| Metric | STEP-OPD | STEP-OPD+FocusOPD | Difference of means |
|---|---|---|---|
| GenEval | 0.960 ± 0.004 | 0.9667 ± 0.0001 | +0.0067 |
| OCR | 0.959 ± 0.002 | 0.961 ± 0.001 | +0.002 |
| PickScore | 24.047 ± 0.009 | 24.024 ± 0.005 | −0.023 |
| DB Aesthetic | 6.244 ± 0.020 | 6.260 ± 0.004 | +0.017 |
GenEval and Aesthetic improve consistently with small run-to-run variation; OCR rises slightly; PickScore shows the reported trade-off (−0.023, within seed SD).
W3. Table 4 shows All-24 can score higher on GenEval than Last-12. Matched-budget random-layer or random-timestep controls would clarify whether the benefit comes from the proposed depth–temporal placement or simply fewer auxiliary constraints.
We add random-depth and random-time controls. Random-depth selects 12 blocks, matching Last-12; random-time selects a matched adjacent window of four timestep indices. Both use the same λ, seed, replay protocol, and budget:
| Method | Layers | Timesteps | GenEval | OCR | Aesthetic |
|---|---|---|---|---|---|
| FocusOPD | Last-12 | Front-40% | 0.960 | 0.959 | 6.214 |
| Random-depth | Random 12 blocks | Front-40% | 0.958 | 0.959 | 6.205 |
| Random-time | Last-12 | Matched random-time | 0.959 | 0.961 | 6.180 |
The controls show that allocation changes trade-offs: FocusOPD has a slightly higher GenEval score than random-time (0.960 vs. 0.959) and the strongest Aesthetic score, while random-time is slightly higher on OCR (0.961 vs. 0.959). These single-run controls support a placement-dependent trade-off rather than a uniform gain on every metric. All-24/Front-40% raises GenEval to 0.965 versus 0.960, but doubles cache; Last-12 keeps the profile-guided choice with half the cache.
W4. FocusOPD adds teacher feature extraction, storage, and training computation (16.54→18.99 s/update and 432 MiB cache) despite no inference overhead. Please discuss this training-cost/performance trade-off.
The overhead depends on the base OPD protocol, because feature queries reuse output queries. DiffusionOPD and STEP-OPD already evaluate all 10 output-query states, so steps 0–3 add no distinct timestep states; only DanceOPD, with a single output-query state, goes from 1 to 4–5. Per-update time changes, measured as in Table 3: +2.45 s (SD3.5-M) / +3.94 s (Z-Image) for DiffusionOPD, −0.09 / −0.44 s for STEP-OPD (within noise), +3.92 / +3.50 s for DanceOPD.
In practice, the extra cost is fixed by the protocol, but whether it pays off varies with the backbone and the protocol together. On Z-Image, DanceOPD gains substantially (GenEval 0.726→0.897); on SD3.5-M, the same schedule moves it only slightly (0.927→0.931), because Last-12 takes a protocol with one state and supervises four to five, and the SD3.5-M runs do not convert that into a larger gain. DiffusionOPD and STEP-OPD already cover deep early states, so they pay little and still improve (SD3.5-M DiffusionOPD GenEval 0.953→0.967). FocusOPD is therefore most worth its cost on protocols that leave those states unsupervised, and the state budget is best set per backbone–protocol pair.
Thank you again for your detailed and constructive comments. We have made further revisions to the update paper based on your feedback. We appreciate your suggestions and look forward to any additional comments!
A brief follow-up on our response
Dear Reviewer 27g3,
Thank you again for your thoughtful comments. We have added experiments and clarifications on the connection between feature mismatch and supervision, the consistency of the gains, and the placement and training-cost trade-offs. We hope these additions help address your concerns. If any point in our response needs further explanation, we would be happy to clarify.
Best regards, The Authors
Thank you for the responses and additional results. My questions have been resolved, and I maintain my positive score.
Cosine vs. MSE, random controls, and what the extra training time buys
We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.
For whether larger discrepancy implies useful alignment, we connect the discrepancy to Proposition 3.3 rather than relying on discrepancy magnitude alone. Proposition 3.3 gives a conditional local criterion: a feature correction at location i helps the output objective when its local correction efficiency αi=Di⟨gout+,gi⟩ is positive and comparable across the selected locations, so that αiDi is the first-order contribution of that correction to the output objective, where Di=∥ei∥22 is the token-averaged squared feature discrepancy and αiDi=⟨gout+,gi⟩ is the first-order output-loss reduction. We therefore check this assumption on the region we actually supervise. Against a matched random placement under the same budget, the selected region has higher mean αi and higher mean αiDi on all three tasks (GenEval/OCR/PickScore): mean αi is 2.15e-4/4.62e-4/5.58e-4 for FocusOPD versus 1.37e-4/3.42e-4/4.95e-4 for random placement, and mean αiDi is 7.004e-6/2.769e-6/7.815e-6 versus 1.152e-6/1.472e-6/4.917e-6. The selected region thus has positive local compatibility on average, and more of it than random placement.
For whether the depth–temporal structure holds under cosine, we evaluate the same MSE-selected cells with dcos:
On SD3.5-M, NMSE and cosine give nearly the same ordering and the same top cells. On Z-Image, their overall agreement is weaker, but the selected region still has 11.9× larger mean dcos. The region picked by the MSE profile is also the region of largest angular mismatch, so the profile is a usable region-selection signal even when the supervision objective is cosine.
For the choice of cosine as the feature objective, raw MSE mixes token scale with direction. Normalized MSE divides the discrepancy by the token norm (teacher energy), which removes this scale dependence, and it already improves GenEval/OCR over output-only supervision. Cosine goes one step further: it removes the norm entirely and measures directional agreement on the unit sphere, 1−⟨uS,uT⟩=21∥uS−uT∥22. This is the same quantity as the angular mismatch dcos used in our diagnosis. Cosine therefore measures exactly the angular quantity used in our diagnosis, , so supervision and diagnosis are defined on the same scale-free quantity. We compare the two objectives at the same placement:
Cosine exceeds normalized MSE on GenEval and the preference metrics shown here, and OCR is slightly lower (0.963 to 0.959). Because it is computed on the same angular quantity used in the diagnosis, the supervision and the diagnosed mismatch are measured consistently.