- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a new multi-teacher knowledge distillation framework called GRAFT, which is capable of collapsing capabilities of many different teacher networks into a single agglomerative backbone. The main novelty of the proposed GRAFT method, lies in its way of formulating multi-teacher knowledge distillation in a continual learning setting. The authors suggest to treat the previously distilled model as a teacher for the current student, keeping its learned capabilities while letting the student jointly learn from both the previous model and the new teacher. Experimental results reveal that this framework leads an efficient way of transferring the capabilities of a new teacher model to the agglomerative model in a progressive manner, without forgetting its earlier learned capabilities.
Critical strengths
S1. Formulating multi-teacher knowledge distillation framework in a continual learning is an interesting idea. It allows for transferring the capabilities of different foundation models to a much simpler agglomerative model in a progressive manner. Experiments demonstrate that with the suggested strategy, a single agglomerative model become capable of acquiring capabilities of a number of foundational models to a certain extent.
S2. In the continual learning setting of the proposed framework, the previously distilled model is treated as a teacher for the current student, which seems an efficient way of transferring new capabilities to the agglomerative model, while preserving its learned capabilities. It eliminates the need for repeating expensive joint distillation over the entire set of teacher foundation models.
S3. To harmonize the heterogeneous representation geometries as suggested by different teacher foundation models, the authors propose to utilize learnable teacher-specific readout tokens and a geometry-agnostic relational loss. Ablation studies show the effectiveness of these contributions, specifically on transferring capabilities of a vision–language teacher.
Critical weaknesses
W1. Since the competing multi-teacher knowledge distillation consider backbones and different set of teachers other than the ones GRAFT considers, the experimental results do not reflect a controlled comparison of the methods.
W2. Experiments demonstrate that transferring new capabilities from a new teacher does not fully preserve the previously learned capabilities that there could be some performance drops for some tasks.
W3. The paper lacks a discussion and comparison with the recently introduced C-RADIOv4 (Ranzinger et al., 2026) multi-teacher distillation method.
Questions for authors
Q1. The proposed GRAFT method enables a progressive learning mechanism of different capabilities. How is the performance of the agglomerative model affected when the order of the teacher models are changed in the continual learning setting?
Q2. For a more controlled comparison, would it be possible to train the competing multi-teacher distillation models by considering all the teacher foundations model that GRAFT is trained on?
3 comments
Thank you for your response.
The additional controlled experiments successfully address my concerns regarding a fair comparison and the ordering of the teachers. I also appreciate the inclusion of comparisons against C-RADIOv3 models, which makes the evaluation much more comprehensive.
However, I noticed that a revised version of the paper addressing the reviewers' concerns has not yet been uploaded. I am inclined to raise my rating, but I would like to see the updated manuscript before doing so.
Thanks for your constructive and timely feedback. We have revised the manuscript to address the comments and incorporated the additional experiments, analyses, and discussions as follows:
- Practical contribution: We strengthened the discussion of GRAFT's practical impact and computational efficiency in the Appendix A and highlighted this contribution in the Introduction.
- Joint distillation: We added a discussion comparing GRAFT with joint distillation in Section 4.2.
- Additional baselines: We included C-RADIOv4 and joint distillation results in Table 1, along with further discussion of C-RADIOv4 in the analysis accompanying Table 3.
- Generality of GRAFT: We added experiments extending C-RADIOv3 with new capabilities to demonstrate GRAFT's applicability to independently pre-trained agglomerative models (Appendix C).
- Teacher order sensitivity: We added experiments investigating the effect of teacher order on performance and knowledge retention (Appendix C).
- Data composition: We provided additional details on the training data composition for each distillation stage (Appendix D).
Please feel free to raise any further concerns. We would be happy to address them.
We thank the reviewer for the constructive suggestions. We have conducted additional experiments to examine controlled joint distillation, teacher-order sensitivity, and the applicability of GRAFT to an independently pre-trained agglomerative encoder.
W1 / Q2. Controlled comparison. We additionally trained a joint distillation baseline using the same four teachers (DINOv2, Multi-HMR, MASt3R, and SigLIP2), ViT-B student architecture, Teacher Specific Readout Tokens, and Geometry Agnostic Relational Loss as GRAFT.
GRAFT achieves competitive performance against joint distillation across the evaluated capabilities. The practical advantage is incremental efficiency: incorporating a new teacher into an existing DUNE encoder with GRAFT requires approximately 370 GPU hours, compared with 1,690 GPU hours for the reported four-teacher joint distillation.
W2. Retention of previous capabilities. We thank the reviewer for raising the important question of capability retention. A central objective of GRAFT is to mitigate the interference between acquiring a new heterogeneous capability and preserving previously consolidated representations.
Table 2 compares GRAFT with frozen-encoder adaptation, direct distillation, EWC, and component ablations. GRAFT achieves the strongest overall balance between acquiring new capabilities and retaining previous ones, with 81.2% ImageNet-1K accuracy and 46.0 ADE20K mIoU, compared with 77.1% / 40.2 for EWC and 82.6% / 35.4 for direct distillation. Moreover, its performance remains close to four-teacher joint distillation.
W3. C-RADIOv4 and cross-backbone applicability. We thank the reviewer for highlighting C-RADIOv4. We will include both C-RADIOv3 and C-RADIOv4 in our empirical comparison and clarify differences in teacher composition, model capacity, training protocols, and evaluation settings. Importantly, the smallest C-RADIOv4 variant, SO400M, contains 412M parameters, substantially more than the ViT-Base backbones (approximately 86M parameters) used by the agglomerative baselines in Table 1.
Furthermore, we demonstrate that even C-RADIO models could be extended by GRAFT method. To test this, we apply GRAFT to a pre-trained C-RADIOv3-B backbone (90M parameters) to incorporate 3D human mesh recovery (HMR) capabilities from the Multi-HMR teacher.
We compare the original C-RADIOv3-B and our GRAFT extension, evaluating both the preservation of previously acquired capabilities and the acquisition of human reconstruction capabilities. The additional experiment improves human mesh recovery while largely retaining the encoder's original classification and segmentation performance. It further demonstrates that GRAFT can extend an independently pre-trained agglomerative model, rather than relying on a particular initialization or incoming teacher.
Q1. Teacher order. We additionally evaluated both arrival orders of DINOv2 and SigLIP2.
The above table presents two separate experiments to assess sensitivity to teacher order. The first pair of rows (rows 1–2) shows starting with DINOv2 and adding SigLIP2, which gains classification ability while preserving dense segmentation and depth performance. The second pair (rows 3–4) reverses this sequence, starting with SigLIP2 and adding DINOv2. This results in substantial gains in segmentation and depth with minimal classification drop. Despite minor metric fluctuations between the two trajectories, both sequences achieve comparable overall performance. This suggests that GRAFT is not strongly sensitive to teacher order in the evaluated setting.