- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Relevance-Guided Token Supervision Allocation (RGSA) to improve representation alignment in diffusion models. Built upon iREPA, RGSA replaces uniform token-wise loss aggregation with image-conditioned weights derived from frozen teacher representations. Specifically, it combines patch-feature norms and patch–CLS cosine similarity to estimate token relevance and adjusts the contribution of each token to the alignment loss. Experiments on ImageNet demonstrate improvements over iREPA across different model scales and visual teachers, with additional ablations supporting the effectiveness of the proposed weighting strategy.
Critical strengths
-
The proposed method is simple and computationally efficient, requiring no additional learnable parameters or inference-time overhead. The reported experiments demonstrate consistent improvements over iREPA across different model scales and visual teachers.
-
The paper provides relatively comprehensive experimental evaluations, including comparisons with alternative weighting strategies, ablation studies, and token-level alignment analyses. These experiments provide reasonable empirical support for the effectiveness of relevance-guided supervision.
Critical weaknesses
-
Limited methodological novelty. My primary concern is that the technical contribution is relatively incremental. The proposed method essentially modifies the uniform token-wise loss aggregation in iREPA by introducing manually designed relevance weights. Both relevance signals, namely patch-feature norms and patch–CLS cosine similarity, are straightforward feature statistics, and their fusion involves only a simple weighted combination and normalization. While the resulting improvements are encouraging, the method does not introduce a substantially new representation alignment mechanism or optimization framework. Therefore, I find the methodological contribution insufficiently novel for ICLR.
-
Limited significance of the empirical improvements. Although RGSA consistently improves over iREPA, the gains become quite marginal under longer training schedules. For example, at 400K training steps on SiT-XL/2, the FID improvement is only from 7.52 to 7.47 without CFG and from 1.93 to 1.91 with CFG. Moreover, the evaluation is primarily limited to ImageNet class-conditional generation, and the paper does not report variability across independently trained checkpoints. Given the simplicity of the proposed modification, stronger empirical evidence would be needed to establish its broader significance.
Questions for authors
-
Beyond introducing heuristic token-wise weights into the existing iREPA objective, what is the main methodological novelty of RGSA compared with conventional spatially weighted feature distillation approaches?
-
Could the authors provide results from multiple independent training runs, particularly for the 400K-step setting, to demonstrate that the relatively small improvements are statistically reliable?
4 comments
Response to Reviewer EdUT (Part 2 of 2)
Q3. Generalization across datasets and generator architectures
Tables 3 and 4 of the original submission compare different visual teachers and SiT model sizes, respectively. RGSA achieves lower FID than iREPA in each of the tested settings.
To further assess RGSA across datasets and generator architectures, we added two comparisons with iREPA at a resolution of . Within each experiment, both methods use the same number of training steps and the same sampling settings.
Table R2 shows that RGSA reduces FID from 16.02 to 14.50 with SiT-B/2 on AFHQv2 and from 60.74 to 58.95 with JiT-B/16 on ImageNet-1K. Both results have been added to Section 4.1 of the revised paper.
Table R2. Experimental results across datasets and architectures. These results have been added to Section 4.1 of the paper.
| Model and dataset | Method | FID ↓ | sFID ↓ | IS ↑ | Precision ↑ | Recall ↑ |
|---|---|---|---|---|---|---|
| SiT-B/2 · AFHQv2 256×256 | iREPA | 16.02 | 105.12 | 11.00 | 0.78 | 0.28 |
| SiT-B/2 · AFHQv2 256×256 | RGSA | 14.50 | 104.63 | 11.10 | 0.79 | 0.30 |
| JiT-B/16 · ImageNet-1K 256×256 | iREPA | 60.74 | 19.23 | 22.24 | 0.35 | 0.56 |
| JiT-B/16 · ImageNet-1K 256×256 | RGSA | 58.95 | 18.89 | 24.10 | 0.36 | 0.56 |
References
- [1] Yang et al. Focal and Global Knowledge Distillation for Detectors. CVPR 2022.
- [2] Karras et al. Alias-Free Generative Adversarial Networks. NeurIPS 2021.
- [3] Li and He. Back to Basics: Let Denoising Generative Models Denoise. CVPR 2026.
Thank you for the detailed rebuttal and additional experiments.
The new results across multiple training seeds, datasets, and architectures provide stronger evidence for the effectiveness and robustness of RGSA. I also appreciate the clarification regarding its differences from existing spatially weighted distillation methods.
However, my main concern remains the limited methodological novelty. RGSA essentially introduces heuristic relevance weights into the existing iREPA objective, and I am not fully convinced that this represents a sufficiently substantial technical contribution, especially given the relatively modest improvements under longer training schedules.
Overall, the rebuttal has addressed most of my empirical concerns. I am open to reconsidering my initial rating after further discussion with the other reviewers.
Response to Reviewer EdUT
Thank you for considering the additional evidence and for your willingness to reconsider your assessment during the reviewer discussion.
Response to Reviewer EdUT (Part 1 of 2)
We thank the reviewer for recognizing the efficiency of our method and the experimental analysis, and for pointing out the need to clarify our methodological contribution and long-training results. We have submitted the revised paper, with changes marked in blue.
Q1. What does RGSA contribute beyond conventional spatially weighted distillation?
FGD [1] uses foreground and background information from the detection task for spatial weighting. We focus on a question that has received relatively little attention in isolation in token-wise representation alignment for diffusion models. Once the teacher–student token correspondence is fixed, how much alignment supervision should each spatial token receive? To address this question, we propose RGSA, which uses patch feature norms and patch–CLS cosine similarities from a frozen teacher to compute per-image normalized weights without additional learnable parameters. The Related Work section of the revised paper now discusses how RGSA differs from conventional spatially weighted distillation.
We hold iREPA’s teacher, projector, token correspondence, per-token alignment loss, and global alignment coefficient fixed and replace only uniform aggregation with RGSA weighting. FID decreases from 110.87 to 108.92, and Recall increases from 0.4327 to 0.4526 (Table 5 of the original submission). The Shuffled control preserves the full set of weight values for each image and only permutes their spatial assignments. FID then rises to 112.58, and Recall drops to 0.4358. This shows that, in the tested setting, spatial allocation still affects generation performance even when the weight distribution is identical.
RGSA provides a simple baseline for studying spatial supervision allocation as an independent design dimension and for comparing and optimizing different allocation strategies.
Q2. Do the 400K-step improvements hold across independent training runs?
We added two independent training seeds to the comparison of iREPA and RGSA with SiT-XL/2 on ImageNet-1K. Table R1 shows that RGSA achieves lower FID and higher IS across all three seeds, both with and without CFG. With CFG, mean IS increases by 6.18 points (269.15 → 275.33). At 400K steps, the direction of the FID improvement is consistent across the tested seeds.
Table R1. Independent training results for SiT-XL/2 trained for 400K steps on ImageNet-1K. These results have been added to “Robustness to Training Randomness” (Appendix G, Table 13).