- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This study proposes a novel method for distilling a super-resolution teacher model into a student model without accessing the teacher's hidden states. The proposed method, AMID, consists of two novel components, GAMD and ISCD. GAMD conveys to the student how the teacher model utilizes the input image, while ISCD uses an invertible model that is isomorphic to the student in the reverse direction to mitigate representation-level mismatch. The effectiveness of the proposed method is demonstrated through comparisons with multiple prior methods across several models and datasets. Sufficient ablation studies also demonstrate the importance of each component.
Critical strengths
-
The method consistently outperforms multiple prior methods across several models and datasets. Its computational cost is also lower than that of competing methods.
-
The individual components of the method are novel, and the methodology is convincing. The analysis showing that GAMD acts as a fidelity regularizer while ISCD contributes to perceptual improvement is also interesting.
-
ISCD, in particular, is a generalizable approach and may have some impact on other domains as well.
Critical weaknesses
- The method involves a somewhat large number of hyperparameters. However, the chosen hyperparameter values appear sufficiently robust across different teacher models, so this is not a major weakness.
Questions for authors
- Tables 16 and 17 suggest that the inverse reconstruction loss plays a meaningful role. How do the authors justify not using the inverse reconstruction loss in the VGG control?
3 comments
Thank you for your clarifications. After considering your responses and the discussions with other reviewers, I will maintain my original rating.
We sincerely thank the reviewer for the careful reading and for keeping the original rating.
The question raised in the review is now stated more clearly in Appendix A.18, with the supporting loss ablation in Appendix A.15.
We thank the reviewer for the time and patience devoted to helping us improve this work.
W1. Number of hyperparameters
We agree that AMID uses a somewhat large number of hyperparameters, and we thank the reviewer for finding the chosen values robust across teachers.
Each teacher produces ISCD terms on its own numerical scale, so Appendix A.4 chooses λout, λirc, and λisc once per teacher and then keeps them at 2×, 3×, and 4×. The change stays inside a narrow set. λout remains 0.1 and λirc remains 1.0 for all five teachers. λisc takes 0.01 or 0.05, and NGAMD takes 50 or 100. We thank the reviewer for treating this limited range as sufficient and not as a major weakness.
Q1. Inverse reconstruction in the VGG control
We thank the reviewer for leading us to clarify this point further. We include the VGG control because this is the feature space used by CSD-style distillation. Section 3.2 states that these methods compare reconstructed images through a VGG extractor. Appendix A.16 follows that setting. The network is a frozen ImageNet-pretrained VGG, and its intermediate features form a fixed perceptual space for aligning the teacher and student reconstructions.
This VGG is trained for image classification and does not implement an HR-to-LR mapping. During distillation, the SR output is passed through it and the alignment uses its multi-layer features. Section 3.2 defines Lirc=∥I(S(x))−x∥1, which requires the proxy to turn the student prediction into an image that can be compared with the LR input. VGG does not produce that degraded image, so the loss has no target. Appendix A.16 therefore disables Lirc for the VGG proxy.
The rest of the objective is shared. Both runs use the reconstruction loss, the output distillation loss, and the same batch-wise contrastive form. The inverse network provides a degradation-aware feature space that matches the student architecture and can also carry Lirc. VGG provides a generic classification feature space, and the inverse reconstruction term is not available there. Appendix A.16 compares these as the complete design each proxy can support. Tables 16 and 17 in Appendix A.13 show why the inverse term is kept where it is defined. With HYPIR at 2×, adding Lirc to Lout reduces LPIPS from 0.1280 to 0.1011.