acceptodds
Under review as a conference paper at ICLR 2027

FocusOPD: Structure-Guided On-Policy Distillation for Diffusion Models

Abstract

Multi-teacher on-policy distillation enables rectified-flow models with diffusion-transformer architectures to combine task-specific capabilities within a single student. However, matching model outputs does not guarantee that their internal representations are aligned, and it remains unclear where and when additional feature supervision is most useful. We analyze the teacher–student feature mismatch across Transformer blocks and denoising timesteps, and find that the mismatch is concentrated in deeper Transformer blocks during early denoising and persists after OPD distillation, indicating ineffective distillation. Motivated by this observation, we introduce FocusOPD, which augments the base OPD objective with token-wise directional feature alignment restricted to deep layers and early denoising steps. Across two rectified-flow backbones and multiple OPD objectives, improves capability transfer in compositional generation, text rendering, and preference-based evaluation. On the primary benchmark, it improves over DiffusionOPD from 0.953 to 0.967 on GenEval, from 23.797 to 23.944 on PickScore, and from 6.052 to 6.199 on HPSv2.1. Ablations show that selective deep and early supervision is more effective and efficient than alignment over the full network and trajectory.

Primary area:
generative models
Keywords:
diffusion, image generation, On-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_27g3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates teacher–student feature discrepancies across Transformer layers and denoising timesteps in multi-teacher on-policy distillation for diffusion models. Based on the observation that larger discrepancies tend to appear in deeper layers and early denoising steps, the authors propose FocusOPD, which introduces directional feature alignment in these selected regions. Experiments on SD3.5-M and Z-Image show improvements over several existing OPD methods.

Critical strengths

The paper studies an interesting question about where and when feature supervision should be applied during on-policy distillation. The analysis of teacher–student discrepancies provides a reasonable motivation for the proposed method, and the selective supervision strategy is straightforward to implement. Experiments across two diffusion backbones and several OPD baselines, together with ablations on layer and timestep selection, provide useful empirical observations.

Critical weaknesses
  1. The connection between feature discrepancy and distillation effectiveness is not fully established. The main motivation is that larger teacher–student discrepancies indicate where additional supervision would be more useful. However, a larger feature mismatch does not necessarily mean that aligning those features will improve generation quality. The theoretical analysis also relies on assumptions about gradient compatibility that are not directly verified in the experiments. Furthermore, the mismatch analysis uses normalized MSE, while the proposed method uses cosine similarity. It would be useful to examine whether the same depth–temporal patterns hold for the actual cosine-based supervision objective.

  2. The improvements over existing OPD methods are relatively small in several settings. In Table 2, FocusOPD improves DiffusionOPD on SD3.5-M, but the additional gains over STEP-OPD are quite small. For example, GenEval increases from 0.963 to 0.967, while PickScore slightly decreases from 24.048 to 24.025. Similar observations can be made on Z-Image, where STEP-OPD improves from 0.946 to 0.948 on GenEval. Moreover, some metrics decrease after adding FocusOPD, particularly OCR accuracy and certain preference scores. I would like to see more discussion of these trade-offs and whether the improvements are consistent across different training seeds.

  3. The benefit of the proposed selection strategy could be further clarified. The ablation studies show that supervising deep layers during early denoising is generally more effective than supervising shallow layers or the full trajectory. However, Table 4 also shows that supervising all blocks achieves a higher GenEval score than the selected Last-12 configuration. Since the selection strategy is central to the paper, comparisons under matched supervision budgets would be useful. For example, randomly selected layers or timesteps with the same number of supervised positions could help determine whether the improvement comes from the proposed depth–temporal structure or simply from using fewer auxiliary constraints.

  4. The additional training cost. Although FocusOPD introduces no additional inference cost, it requires teacher feature extraction, storage, and additional training computation. Table 3 shows that training time per update increases from 16.54s to 18.99s, together with an additional 432 MiB feature cache. Given the relatively small improvements on some benchmarks, a more detailed discussion of the training cost and performance trade-off would help assess the practical benefit of the method.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Cosine vs. MSE, random controls, and what the extra training time buys

We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.

W1. Larger feature discrepancy need not imply that alignment improves generation quality; the gradient-compatibility assumptions are not directly verified. The profile uses normalized MSE while the method uses cosine, so please check whether the same depth–temporal pattern holds for cosine discrepancy.

For whether larger discrepancy implies useful alignment, we connect the discrepancy to Proposition 3.3 rather than relying on discrepancy magnitude alone. Proposition 3.3 gives a conditional local criterion: a feature correction at location ii helps the output objective when its local correction efficiency αi=⟨gout+,gi⟩Di\alpha_i=\frac{\langle g_{\mathrm{out}}^{+},g_i\rangle}{D_i} is positive and comparable across the selected locations, so that αiDi\alpha_i D_i is the first-order contribution of that correction to the output objective, where Di=∥ei∥22D_i=\lVert e_i\rVert_2^2 is the token-averaged squared feature discrepancy and αiDi=⟨gout+,gi⟩\alpha_i D_i=\langle g_{\mathrm{out}}^{+},g_i\rangle is the first-order output-loss reduction. We therefore check this assumption on the region we actually supervise. Against a matched random placement under the same budget, the selected region has higher mean αi\alpha_i and higher mean αiDi\alpha_i D_i on all three tasks (GenEval/OCR/PickScore): mean αi\alpha_i is 2.15e-4/4.62e-4/5.58e-4 for FocusOPD versus 1.37e-4/3.42e-4/4.95e-4 for random placement, and mean αiDi\alpha_i D_i is 7.004e-6/2.769e-6/7.815e-6 versus 1.152e-6/1.472e-6/4.917e-6. The selected region thus has positive local compatibility on average, and more of it than random placement.

For whether the depth–temporal structure holds under cosine, we evaluate the same MSE-selected cells with dcos⁡d_{\cos}:

CheckSD3.5-MZ-Image
NMSE/cosine Spearman correlation0.9980.826
Top-10% cell overlap0.9830.839
Candidate/rest mean dcos⁡d_{\cos}0.0303/0.00595.36e-4/4.49e-5

On SD3.5-M, NMSE and cosine give nearly the same ordering and the same top cells. On Z-Image, their overall agreement is weaker, but the selected region still has 11.9× larger mean dcos⁡d_{\cos}. The region picked by the MSE profile is also the region of largest angular mismatch, so the profile is a usable region-selection signal even when the supervision objective is cosine.

For the choice of cosine as the feature objective, raw MSE mixes token scale with direction. Normalized MSE divides the discrepancy by the token norm (teacher energy), which removes this scale dependence, and it already improves GenEval/OCR over output-only supervision. Cosine goes one step further: it removes the norm entirely and measures directional agreement on the unit sphere, 1−⟨uS,uT⟩=12∥uS−uT∥221-\langle u^S,u^T\rangle=\frac{1}{2}\Vert u^S-u^T\Vert_2^2. This is the same quantity as the angular mismatch dcos⁡d_{\cos} used in our diagnosis. Cosine therefore measures exactly the angular quantity used in our diagnosis, , so supervision and diagnosis are defined on the same scale-free quantity. We compare the two objectives at the same placement:

ObjectiveGenEvalOCRPickScoreAestheticImgRwd
Output-only0.9510.96123.6546.1461.502
Normalized MSE0.9550.96323.5096.2101.491
FocusOPD (cosine)0.9600.95923.5456.2141.497

Cosine exceeds normalized MSE on GenEval and the preference metrics shown here, and OCR is slightly lower (0.963 to 0.959). Because it is computed on the same angular quantity used in the diagnosis, the supervision and the diagnosed mismatch are measured consistently.

AuthorsOpenReview ↗

Cosine vs. MSE, random controls, and what the extra training time buys

W2. Improvements over existing OPD methods are small in several settings, especially over STEP-OPD, and some OCR and preference metrics decrease. Please discuss the capability trade-offs and consistency across training seeds.

For the gains over STEP-OPD. GenEval and OCR are bounded by 1, and the SD3.5-M STEP-OPD baseline already scores 0.963 and 0.961, so absolute gains are compressed in this regime. Within it, FocusOPD raises GenEval to 0.967, driven by the compositional-attribute category (0.900→0.930); on Z-Image, GenEval rises 0.946→0.948. STEP-OPD also supervises representation dynamics, so its incremental gains are smaller than over output-only baselines: adding FocusOPD to DiffusionOPD gives +0.014/+0.002/+0.147 on SD3.5-M (Tab. 2).

For consistency across seeds, we retrain each baseline three times (42/43/44). For DiffusionOPD the three-seed averages are +0.015±0.005 (GenEval), +0.003±0.003 (OCR), and +0.143±0.014 (PickScore), consistent with the single-run +0.014/+0.002/+0.147 in Tab. 2 within seed SD. For STEP-OPD:

MetricSTEP-OPDSTEP-OPD+FocusOPDDifference of means
GenEval0.960 ± 0.0040.9667 ± 0.0001+0.0067
OCR0.959 ± 0.0020.961 ± 0.001+0.002
PickScore24.047 ± 0.00924.024 ± 0.005−0.023
DB Aesthetic6.244 ± 0.0206.260 ± 0.004+0.017

GenEval and Aesthetic improve consistently with small run-to-run variation; OCR rises slightly; PickScore shows the reported trade-off (−0.023, within seed SD).

W3. Table 4 shows All-24 can score higher on GenEval than Last-12. Matched-budget random-layer or random-timestep controls would clarify whether the benefit comes from the proposed depth–temporal placement or simply fewer auxiliary constraints.

We add random-depth and random-time controls. Random-depth selects 12 blocks, matching Last-12; random-time selects a matched adjacent window of four timestep indices. Both use the same λ, seed, replay protocol, and budget:

MethodLayersTimestepsGenEvalOCRAesthetic
FocusOPDLast-12Front-40%0.9600.9596.214
Random-depthRandom 12 blocksFront-40%0.9580.9596.205
Random-timeLast-12Matched random-time0.9590.9616.180

The controls show that allocation changes trade-offs: FocusOPD has a slightly higher GenEval score than random-time (0.960 vs. 0.959) and the strongest Aesthetic score, while random-time is slightly higher on OCR (0.961 vs. 0.959). These single-run controls support a placement-dependent trade-off rather than a uniform gain on every metric. All-24/Front-40% raises GenEval to 0.965 versus 0.960, but doubles cache; Last-12 keeps the profile-guided choice with half the cache.

W4. FocusOPD adds teacher feature extraction, storage, and training computation (16.54→18.99 s/update and 432 MiB cache) despite no inference overhead. Please discuss this training-cost/performance trade-off.

The overhead depends on the base OPD protocol, because feature queries reuse output queries. DiffusionOPD and STEP-OPD already evaluate all 10 output-query states, so steps 0–3 add no distinct timestep states; only DanceOPD, with a single output-query state, goes from 1 to 4–5. Per-update time changes, measured as in Table 3: +2.45 s (SD3.5-M) / +3.94 s (Z-Image) for DiffusionOPD, −0.09 / −0.44 s for STEP-OPD (within noise), +3.92 / +3.50 s for DanceOPD.

In practice, the extra cost is fixed by the protocol, but whether it pays off varies with the backbone and the protocol together. On Z-Image, DanceOPD gains substantially (GenEval 0.726→0.897); on SD3.5-M, the same schedule moves it only slightly (0.927→0.931), because Last-12 takes a protocol with one state and supervises four to five, and the SD3.5-M runs do not convert that into a larger gain. DiffusionOPD and STEP-OPD already cover deep early states, so they pay little and still improve (SD3.5-M DiffusionOPD GenEval 0.953→0.967). FocusOPD is therefore most worth its cost on protocols that leave those states unsupervised, and the state budget is best set per backbone–protocol pair.


Thank you again for your detailed and constructive comments. We have made further revisions to the update paper based on your feedback. We appreciate your suggestions and look forward to any additional comments!

AuthorsOpenReview ↗

A brief follow-up on our response

Dear Reviewer 27g3,

Thank you again for your thoughtful comments. We have added experiments and clarifications on the connection between feature mismatch and supervision, the consistency of the gains, and the placement and training-cost trade-offs. We hope these additions help address your concerns. If any point in our response needs further explanation, we would be happy to clarify.

Best regards, The Authors

Thank you for the responses and additional results. My questions have been resolved, and I maintain my positive score.

Reviewer_DDahOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FocusOPD, which augments multi-teacher on-policy distillation for diffusion models with cosine alignment of teacher and student image-token features in deep Transformer blocks and early denoising steps. The authors motivate this placement by observing that teacher–student feature mismatch is concentrated in deeper layers and at high-noise steps, and that this pattern persists after output-level distillation. They give theoretical results on discrepancy propagation, noise-dependent prior sensitivity, and a conditional local criterion for selective supervision. Experiments on SD3.5-M and Z-Image, using three task-specific teachers and three OPD baselines, report improvements on compositional generation, text rendering, and preference metrics; for example, on SD3.5-M with DiffusionOPD, GenEval rises from 0.953 to 0.967 and HPSv2.1 from 6.052 to 6.199. Ablations suggest that restricting feature supervision to deep layers and early denoising steps can be more effective and cheaper than supervising the full network and trajectory.

Critical strengths
  1. The paper examines mismatch across both network depth and denoising time, then varies the depth and temporal supervision windows. The reported results support the practical value of selective placement: for example, full-trajectory supervision does not improve GenEval over output-only distillation in the reported ablation, while early-window settings do. This gives the method a more specific empirical basis than simply adding feature loss everywhere.
  2. The paper applies it to DiffusionOPD, DanceOPD, and STEP-OPD, and evaluates both SD3.5-M and Z-Image across several types of capability. Although the gains are not uniform across every metric, the breadth of these comparisons makes the proposed feature objective potentially useful as a complement to different OPD protocols.
Critical weaknesses
  1. Table 2 reports checkpoint-1000 scores, but the paper does not provide confidence intervals, variation across training runs, or significance tests for these comparisons. Some reported changes are modest—for example, OCR accuracy on SD3.5-M with DiffusionOPD changes from 0.958 to 0.960. Without an estimate of variability, it is difficult to tell how robust the smaller improvements are.
  2. The central mismatch profiles are primarily based on hidden-feature MSE, whereas FocusOPD uses cosine alignment. The appendix notes that raw-MSE rankings do not by themselves establish the same ordering for angular discrepancy. In addition, the local utility proposition requires positive, comparable correction efficiencies, but the paper does not appear to establish that this condition holds for the selected regions. The ablations show that some placements work empirically, but the proposed mechanism and selection principle remain less directly validated than the paper’s framing suggests.
Questions for authors
  1. Were the main comparisons in Table 2 repeated across independent training runs or evaluated with prompt-level uncertainty estimates? If so, please report the variability or significance of the gains, particularly for the smaller changes.
  2. Did you measure teacher–student angular or cosine discrepancy across depth and denoising time, in addition to hidden-feature MSE? Since FocusOPD uses cosine loss, evidence that the selected regions also concentrate angular mismatch would materially strengthen the connection between the diagnosis and the method.
  3. What exact depth and timestep windows, feature-loss weight, and selection procedure were used for Z-Image? Were these choices made using development data separate from the reported evaluation prompts and metrics?
6 comments
AuthorsOpenReview ↗

How stable are the Table 2 numbers, and does the cosine check hold?

We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.

W1. Table 2 gives checkpoint-1000 scores without confidence intervals, variation across training runs, or significance tests, making the robustness of small changes such as OCR 0.958→0.960 unclear.

Q1. Were the main Table 2 comparisons repeated across independent training runs or evaluated with prompt-level uncertainty estimates? Please report variability or significance, especially for smaller gains.

Following your suggestion, we now report two complementary audits of variability. Table 2 reports a single training run per backbone, evaluated on identical prompts, samplers, guidance, evaluators, and evaluation seeds. First, for seed 42 we compute a paired prompt bootstrap (same prompts, paired Focus−baseline) per baseline, which captures prompt-level uncertainty within that run:

BaselineMetricFocusOPD−baseline95% paired CI
DiffusionOPDGenEval+0.014[+0.00062, +0.03344]
DiffusionOPDOCR+0.002[−0.00370, +0.00399]
DiffusionOPDAesthetic+0.147[+0.00827, +0.15435]
STEP-OPDGenEval+0.004[+0.00099, +0.00802]
STEP-OPDOCR+0.000[−0.00370, +0.00399]
STEP-OPDAesthetic+0.005[−0.00972, +0.01708]
DanceOPDGenEval+0.004[+0.00010, +0.02475]
DanceOPDOCR−0.007[−0.00736, +0.00084]
DanceOPDAesthetic+0.114[+0.07305, +0.19013]

DiffusionOPD's GenEval and Aesthetic intervals stay above zero, while its OCR interval includes zero; STEP-OPD shows the same pattern. For DanceOPD, GenEval and Aesthetic are positive, and OCR is negative with an interval that crosses zero.

Second, we trained the SD3.5-M DiffusionOPD and STEP-OPD comparisons across three independent seeds (42/43/44):

MethodGenEvalOCRPickScore
DiffusionOPD0.952 ± 0.0020.959 ± 0.00123.795 ± 0.014
+FocusOPD0.967 ± 0.0050.962 ± 0.00323.938 ± 0.009
STEP-OPD0.960 ± 0.0040.959 ± 0.00224.047 ± 0.009
+FocusOPD0.9667 ± 0.00010.961 ± 0.00124.024 ± 0.005

For DiffusionOPD, the three-seed means improve by +0.015 GenEval, +0.003 OCR, and +0.143 PickScore. For STEP-OPD, the three-seed means show a GenEval and OCR gain alongside a PickScore decrease. Taken together, the two audits show that the larger gains are consistent across seeds while the smaller changes need cautious reading. We now include these estimates in the paper.

AuthorsOpenReview ↗

How stable are the Table 2 numbers, and does the cosine check hold?

W2. The diagnosis mainly uses hidden-feature MSE, while FocusOPD uses cosine alignment; raw-MSE rankings need not preserve angular ordering. The local utility proposition also assumes positive, comparable correction efficiencies that have not been established for selected regions.

Q2. Did you measure angular or cosine discrepancy over depth and denoising time? Evidence that selected regions also concentrate angular mismatch would strengthen the diagnosis-to-method link.

We connect the MSE-based diagnosis to the cosine objective in four steps.

(1) Objective choice. Raw MSE mixes token scale with direction; cosine removes norm scale to measure directional agreement (1−⟨uS,uT⟩=12∥uS−uT∥221-\langle u^S,u^T\rangle=\frac{1}{2}\Vert{}u^S-u^T\Vert{}_2^2). Normalized MSE already improves GenEval/OCR over output-only, and at the same Last-12/Front-40% placement cosine matches or exceeds normalized MSE on preference metrics, so a directional objective is justified.

(2) Angular structure. For each output-only replay example, we compute dcos⁡(ℓ,s)=1P∑p=1P[1−⟨h^ℓ,s,pS,h^ℓ,s,pT⟩]d_{\cos}(\ell,s)=\frac{1}{P}\sum_{p=1}^{P}\left[1-\left\langle\hat h^S_{\ell,s,p},\hat h^T_{\ell,s,p}\right\rangle\right] at each block–step cell (ℓ,s)(\ell,s), where pp indexes conditional image tokens and PP is their number. We then average each cell's value over replay examples:

Cosine checkSD3.5-MZ-Image
Spearman(NMSE, cosine)0.9980.826
Top-10% overlap0.9830.839
Candidate/rest mean dcosd_{cos}0.0303/0.00595.36e-4/4.49e-5

SD3.5-M shows near-identical ordering. Z-Image agrees less globally (Spearman 0.826), but what matters for placement is whether the selected region is still the most angularly mismatched, and it is, with an 11.9× larger mean dcosd_{cos} than the rest. The selected region therefore concentrates angular mismatch on both backbones, which links the diagnosis to the cosine objective at the region level.

(3) Local compatibility. For the selected region we measure αi=⟨gout+,gi⟩/Di\alpha_i=\langle g_{\mathrm{out}}^{+},g_i\rangle/D_i with Di=∥ei∥22D_i=\lVert e_i\rVert_2^2, so αiDi\alpha_i D_i is the first-order loss reduction. FocusOPD versus matched random yields mean αi\alpha_i of 2.15e-4/4.62e-4/5.58e-4 vs. 1.37e-4/3.42e-4/4.95e-4, and mean αiDi\alpha_i D_i of 7.004e-6/2.769e-6/7.815e-6 vs. 1.152e-6/1.472e-6/4.917e-6 (GenEval/OCR/PickScore). This supports positive and higher-than-random average local compatibility, the empirical counterpart of Proposition 3.3's assumption, though not its pointwise conditions.

(4) Budgeted benefit. Under the fixed 12-block/40%-step budget, supervising exactly this region yields the gains in Table 3 and the placement ablation in Table 4. The chain from diagnosis to supervision is therefore: the profile selects a deep-and-early region, that region concentrates angular mismatch, its local compatibility is higher than random, and supervising it within budget helps. Cosine is the natural objective there because it is scale-invariant and matches or exceeds normalized MSE on preference metrics at the same placement.

Q3. What exact depth and timestep windows, feature-loss weight, and selection procedure were used for Z-Image? Were the choices made on development data separate from the reported evaluation prompts and metrics?

The final Z-Image setting is blocks 18–29 (Last-12), Front-40% (steps 0–3 of the 10-step trajectory). The feature-loss weight is λ=0.3\lambda=0.3, shared with the SD3.5-M experiments. The same Z-Image setting (blocks 18–29, steps 0–3, λ=0.3) is used for all three baselines, DiffusionOPD, DanceOPD, and STEP-OPD. As in Figs. 1–2, we diagnose the output-only model, compare q∈6,12,18,24q\in\\{6,12,18,24\\} and r\in\\{20\\%,30\\%,40\\%,50\\%\\} on development prompts, lock q=12,r=40q=12,r=40\\%, and then evaluate on separate held-out prompts and seeds.


Thank you again for your detailed and constructive comments. We have made further revisions to the update paper based on your feedback. We appreciate your suggestions and look forward to any additional comments!

AuthorsOpenReview ↗

A brief follow-up on our response

Dear Reviewer DDah,

Thank you again for your helpful suggestions. We have added confidence intervals, results across three training seeds, and cosine-discrepancy measurements, and clarified the Z-Image settings and selection procedure. When you have a chance, could you let us know whether these additions address your concerns? We would be happy to provide any further explanation.

Best regards, The Authors

Reviewer_CF5gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FocusOPD, a structure-guided feature supervision method for multi-teacher on-policy distillation in diffusion models. The authors observe that teacher–student representation discrepancies tend to concentrate in deeper Transformer blocks and early, high-noise denoising stages, with this structure persisting after output-level distillation. They investigate discrepancy propagation and noise-dependent prior sensitivity, and derive a conditional local criterion connecting feature mismatch to supervision utility. Motivated by these findings, FocusOPD augments existing OPD objectives with token-wise cosine alignment of conditional image features within a fixed deep-layer and early-denoising window. Experiments on SD3.5-M and Z-Image evaluate its integration with DiffusionOPD, DanceOPD, and STEP-OPD across compositional generation, text rendering, and preference-based metrics. The reported results suggest that selective feature supervision can improve capability transfer while reducing training overhead and feature-cache requirements relative to full-network, full-trajectory feature alignment.

Critical strengths
  • A relevant problem with an empirically motivated design. The paper studies where and when intermediate supervision is useful in multi-teacher diffusion distillation. Its depth–time analysis identifies recurring teacher–student discrepancies across two backbones, providing an empirical motivation for concentrating supervision on deep layers and early denoising stages.
  • Simple integration with existing distillation frameworks. FocusOPD uses a straightforward token-wise cosine alignment objective while preserving the underlying OPD output supervision. Its compatibility with several existing methods makes it practically useful, and deployment requires only the trained student, with no additional inference modules or computation.
Critical weaknesses
  • The connection between diagnosed mismatch and supervision utility remains insufficiently established. Although Proposition 3.3 provides a conditional local justification and Appendix connects it to cosine supervision, the positive and comparable correction efficiencies assumed in the comparison are not empirically characterized for the selected region. Moreover, the manuscript does not establish whether the MSE/NMSE profiles used for diagnosis exhibit the same depth–temporal structure under angular discrepancy. This leaves uncertainty about how reliably the diagnosed mismatch guides supervision placement.
  • The additional gains over the strongest baseline are modest, with capability trade-offs. Adding FocusOPD to STEP-OPD improves GenEval by 0.004 on SD3.5-M and 0.002 on Z-Image. Meanwhile, the default configuration in Table 3 improves GenEval but reduces OCR, PickScore, and ImageReward relative to output-only distillation. These results indicate that the benefits are not uniform across capabilities.
Questions for authors
  1. In Table 4, First-6+Last-6 has slightly higher reported scores on five metrics and matches OCR compared with Last-12, under the same layer count and temporal coverage. However, the paper states that “Non-contiguous selection (First-6+Last-6) does not outperform Last-12, confirming that a contiguous deep window suffices.” Could the authors clarify this discrepancy and explain why Last-12 is preferred? Are these differences within experimental variability?
  2. Equation (8) normalizes the feature loss by the full trajectory length NN, whereas Section 6.3 describes averaging over active steps. Could the authors clarify which normalization is used in the implementation, particularly when comparing different temporal windows?
  3. Feature queries are specified independently of output queries. For each baseline, does adding FocusOPD increase the number of teacher-evaluated states? If so, could the authors clarify the additional query cost and whether the performance comparisons account for this increase?
  4. The method is described as structure-guided, with qq and rr calibrated for each backbone. Could the authors clarify how the mismatch profiles are used to determine these window boundaries, and to what extent the final choices depend on downstream validation performance?
5 comments
AuthorsOpenReview ↗

Does the diagnosis actually support the supervision? Plus cost and normalization details

We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.


W1. The connection between diagnosed mismatch and supervision utility remains insufficiently established: the positive, comparable correction efficiencies assumed in Proposition 3.3 are not empirically characterized, and it is unclear whether the MSE/NMSE profiles retain the same depth–temporal structure under angular discrepancy.

For the connection between diagnosed mismatch and supervision utility, we first recall what Proposition 3.3 assumes. The proposition gives a conditional local criterion: a feature correction at location ii is useful to the output objective when its local correction efficiency αi=⟨gout+,gi⟩Di\alpha_i=\frac{\langle g_{\mathrm{out}}^{+},g_i\rangle}{D_i} is positive and comparable across the selected locations, so that αiDi\alpha_i D_i is the first-order contribution of that correction to the output objective, where Di=∥ei∥22D_i=\lVert e_i\rVert_2^2 is the token-averaged squared feature discrepancy and αiDi=⟨gout+,gi⟩\alpha_i D_i=\langle g_{\mathrm{out}}^{+},g_i\rangle is the first-order output-loss reduction. We now check this condition empirically in the selected region against a matched random placement with the same supervision budget. For GenEval/OCR/PickScore, mean αi\alpha_i is 2.15e-4/4.62e-4/5.58e-4 for the selected region versus 1.37e-4/3.42e-4/4.95e-4 for random placement; mean αiDi\alpha_i D_i is 7.004e-6/2.769e-6/7.815e-6 versus 1.152e-6/1.472e-6/4.917e-6. In all three tasks the selected region therefore has both a higher mean local correction efficiency and a larger first-order output-loss reduction than a random placement with the same supervision budget. This supports the positive, higher-than-random average local compatibility assumed in Proposition 3.3.

For the MSE/NMSE profiles, we further examine whether the profile retains the same depth–temporal structure under angular discrepancy, since the MSE-based diagnosis and the cosine objective measure different quantities. For each output-only replay example, we compute the cosine mismatch dcos⁡(ℓ,s)=1P∑p=1P[1−⟨h^ℓ,s,pS,h^ℓ,s,pT⟩]d_{\cos}(\ell,s)=\frac{1}{P}\sum_{p=1}^{P}\left[1-\left\langle\hat h^S_{\ell,s,p},\hat h^T_{\ell,s,p}\right\rangle\right] at each block–step cell (ℓ,s)(\ell,s), where pp indexes conditional image tokens and PP is their number. We average each cell's value over replay examples before comparing it with the MSE profile:

Cosine checkSD3.5-MZ-Image
Spearman(NMSE, cosine)0.9980.826
Top-10% overlap0.9830.839
Candidate/rest mean dcos⁡d_{\cos}0.0303/0.00595.36e-4/4.49e-5

On SD3.5-M the NMSE and angular profiles are nearly identical in ordering and in the top-mismatch cells. On Z-Image the global agreement is lower, but the selected region still exhibits an 11.9× larger mean angular mismatch than the rest. The depth–temporal structure used for diagnosis is therefore preserved under angular discrepancy.

For the choice of the feature objective, we start from raw MSE, which mixes token scale with direction. Normalized MSE suppresses this scale dependence by dividing the discrepancy by the token norm (teacher energy), and compared with output-only supervision it already improves GenEval/OCR. Cosine goes one step further: it removes the norm entirely and measures directional agreement on the unit sphere, 1−⟨uS,uT⟩=12∥uS−uT∥221-\langle u^S,u^T\rangle=\frac{1}{2}\Vert u^S-u^T\Vert_2^2, which is the same quantity as the angular mismatch dcos⁡d_{\cos} used in our diagnosis. Cosine therefore measures exactly the angular quantity used in our diagnosis, so the supervision objective and the diagnosed mismatch are defined on the same scale-free quantity, rather than only being scale-suppressed as in normalized MSE. We compare the two at the same placement:

ObjectiveGenEvalOCRPickScoreAestheticImgRwd
Output-only0.9510.96123.6546.1461.502
Normalized MSE0.9550.96323.5096.2101.491
FocusOPD (cosine)0.9600.95923.5456.2141.497

Cosine exceeds normalized MSE on GenEval and the preference metrics shown here, while OCR decreases slightly (0.963 to 0.959). Because it is computed on the same angular quantity used in the diagnosis, the supervision and the diagnosed mismatch are measured consistently.

AuthorsOpenReview ↗

Does the diagnosis actually support the supervision? Plus cost and normalization details

W2. Gains over the strongest baseline are modest and involve capability trade-offs. Against STEP-OPD, GenEval improves by 0.004 on SD3.5-M and 0.002 on Z-Image; the default Table 3 setting improves GenEval but reduces OCR, PickScore, and ImageReward relative to output-only distillation.

For the gains over STEP-OPD. GenEval and OCR are bounded above by 1, and the SD3.5-M STEP-OPD baseline already scores 0.963 and 0.961, leaving little headroom. FocusOPD still raises overall GenEval to 0.967, with the compositional-attribute category rising from 0.900 to 0.930; on Z-Image, STEP-OPD's overall GenEval rises from 0.946 to 0.948. STEP-OPD also supervises block-level representation dynamics, so adding FocusOPD to it gains less than adding it to output-only baselines: on SD3.5-M, adding FocusOPD to DiffusionOPD raises GenEval/OCR/PickScore by +0.014/+0.002/+0.147 (Tab. 2).

For the capability trade-offs. Across the six paired comparisons in the held-out Table 2 (three baselines × two backbones), most summary metrics rise or tie: 30 of 36 rise and 3 tie, and the only three declines are small (largest: SD3.5-M STEP-OPD PickScore 24.048→24.025; the other two are OCR changes of 0.007 and 0.003). The pattern holds across baselines: compositional and aesthetic metrics improve, OCR changes stay at or below the seed-level scale, and preference scores can shift slightly either way. The development-set ablation in Table 3 gives the same picture, with GenEval and Aesthetic improving while OCR/PickScore/ImageReward move by at most 0.11. We will report gains and trade-offs by baseline and setting.

Q1. In Table 4, First-6+Last-6 scores slightly higher on five metrics and ties OCR versus Last-12 at the same budget, yet the paper says it “does not outperform” Last-12. Why is Last-12 preferred?

Yes. First-6+Last-6 has slightly higher point estimates on five metrics and ties OCR, but the gaps are small: GenEval 0.962 vs. 0.960, PickScore 23.554 vs. 23.545, Aesthetic 6.225 vs. 6.214. We prefer Last-12 because it is the contiguous window the mismatch profile directly suggests, and it avoids a search over non-contiguous layer subsets that would add a hyperparameter without a diagnostic motivation.

Q2. Equation (8) normalizes feature loss by the full trajectory length N, while Section 6.3 says it is averaged over active steps. Which normalization is implemented when temporal windows differ?

The implementation uses the full trajectory length NN: Lfeat=1N∑s1[s∈Ifeat] Lfeats,L_{feat}=\frac{1}{N}\sum_s \mathbb{1}[s\in I_{feat}]\,L_{feat}^s, where each active-step loss is averaged over selected layers and image tokens and inactive steps contribute zero. We will correct the Section 6.3 wording accordingly: the per-step loss is averaged over selected tokens and blocks, summed over active steps, and normalized by NN. Since λ\lambda is fixed, the effective aggregate weight is λK/N\lambda K/N with K=∣Ifeat∣K=|I_{feat}|, so changing the temporal-window width changes this weight; this affects the temporal ablations in Tables 3 and 5, while Table 4 fixes K=4K=4 and matches normalization across rows.

Q3. For each baseline, does adding FocusOPD increase the number of teacher-evaluated states? Please clarify the additional query cost and whether performance comparisons account for it.

DiffusionOPD and STEP-OPD already evaluate all 10 output-query states, so FocusOPD adds no distinct timestep states; only DanceOPD changes its count, from 1 to 4–5 states (steps 0–3, overlapping queries share the teacher forward). Per-update training-time changes vs. the output-only baseline: +2.45 s (SD3.5-M)/+3.94 s (Z-Image) for DiffusionOPD, −0.09/−0.44 s for STEP-OPD, and +3.92/+3.50 s for DanceOPD.

Q4. How do the mismatch profiles determine the depth and timestep window boundaries q and r for each backbone, and how much do the final choices depend on downstream validation performance?

The output-only profiles show that mismatch concentrates in deep layers and early denoising steps, which narrows the candidate windows to this family (Fig. 2). We then choose the final q and r from a small grid on development prompts, using the observed multi-metric and cost trade-off, and lock the choice before evaluating any held-out prompt or metric. For SD3.5-M, the profile motivates deep windows and the development grid selects q=12, blocks 12–23. For Z-Image, it selects q=12, corresponding to blocks 18–29. Both backbones select r=40% from the early-window candidates.


Thank you again for your detailed and constructive comments. We have made further revisions to the update paper based on your feedback. We appreciate your suggestions and look forward to any additional comments!

AuthorsOpenReview ↗

A brief follow-up on our response

Dear Reviewer CF5g,

Thank you again for your detailed comments. We have provided additional evidence on the connection between the mismatch diagnosis and feature supervision, and clarified the normalization, window selection, and teacher-query cost. When you have a chance, could you let us know whether our responses address your concerns? We would be happy to clarify any remaining points.

Best regards, The Authors

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their responses. Could you please read the rebuttal and let us know whether it resolves the questions and concerns raised in your review? If any point remains unclear, feel free to follow up with the authors during the discussion period.

Please update your review and final rating at your earliest convenience.

Thank you very much for your time and contribution.

Best regards, AC

AuthorsOpenReview ↗

Summary of additional results and clarifications

Dear Reviewers and Area Chair,

Thank you for your time, helpful comments, and discussion. We also appreciate the Area Chair’s continued help. During the rebuttal, we have:

  • Added cosine-discrepancy measurements and matched-budget local correction analyses, supporting the connection between mismatch diagnosis and useful supervision.
  • Added confidence intervals and three-seed training results for the SD3.5-M DiffusionOPD and STEP-OPD comparisons, and discussed the gains and differences across metrics.
  • Added matched-budget random-depth and random-time controls, clarifying the benefits and trade-offs of profile-guided supervision.
  • Reported teacher-query counts and training overhead, and clarified loss normalization and the selection of supervision windows before held-out evaluation.

We have done our best to address all the concerns raised and incorporate the clarifications and evidence into the revised paper. We appreciate Reviewer 27g3’s follow-up confirming that their questions have been resolved, and hope the additional results and clarifications are also helpful to the other reviewers.

We would appreciate hearing your thoughts and welcome further discussion. If any concerns remain, we would be happy to address them.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.