- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MARD proposes heterogeneous video knowledge distillation using a frozen foundation-model teacher and a lightweight adapter. It introduces three objectives: Global Similarity Distillation (GSD) for relational distributions, Task Conditioned Distillation (TCD) for verb–noun semantic relations, and Temporal Rhythm Distillation (TRD) for temporal alignment. The method reports substantial improvements on EK100 and SSv2 with VideoSwin-S and X3D-S students.
Critical strengths
- A clean, dimension-agnostic design for heterogeneous distillation. The three objectives operate on structures computed within each model's native feature space, avoiding feature projection, token correspondence, and temporal resampling. The use of a frozen, precomputed teacher is also practically attractive, allowing reuse across downstream tasks.
- Potentially large and consistent gains. MARD reports 5.3–9.5 percentage-point top-1 improvements over non-distilled students across two datasets and two different student architectures.
Critical weaknesses
- The necessity of relaxing pairwise relation matching is insufficiently demonstrated. GSD matches only the overall distribution of sample similarities, unlike established methods such as Similarity-Preserving Knowledge Distillation (SP; Tung and Mori, ICCV 2019, https://arxiv.org/abs/1907.09682), which preserve pairwise correspondence without direct feature alignment. GSD therefore discards potentially meaningful instance-level relations, which TCD only partially recovers through semantic grouping. Comparisons of GSD, GSD+TCD, SP, and SP+TCD under identical settings would help establish the benefit of this design choice.
- The ablations in Table 3 demonstrate the effectiveness and complementarity of the proposed objectives but provide limited insight into their underlying mechanisms. The generality of TCD is further unclear, as the paper does not explain its adaptation, or that of the verb–noun classification loss, to SSv2's action-template labels. Missing key hyperparameters and initialization details also limit reproducibility.
Questions for authors
- Could the authors compare GSD, GSD+TCD, SP, and SP+TCD under identical settings to justify relaxing pairwise correspondence?
- Could the authors provide mechanism-level controls, such as a label-only TCD target and a temporal-order control for TRD?
- How are TCD and the classification loss implemented on SSv2, and could the authors provide the missing hyperparameters, initialization checkpoints, and baseline configurations?
- Table 4 reports identical results for KL and FBT (63.94 Top-1 / 88.89 Top-5); could the authors confirm whether these values are correct?
Response to Reviewer 4YMT
Thank you for suggesting the comparison with SP. We previously evaluated SP in preliminary video-distillation experiments, where it underperformed the student trained with classification supervision alone. This led us to omit SP from the original comparison. We have now included SP in the updated VideoSwin-S/EK100 comparison in Table 1 using seeds 42, 43, and 44. SP achieves 30.42±0.41% action Top-1, compared with 34.33±1.36% for the baseline, a decrease of 3.91 percentage points in mean accuracy.
One possible explanation is that differences in architecture, feature aggregation, and pretraining objectives cause the teacher's and student's global representations to encode different contextual and semantic relations. Pairwise matching may encourage the student to preserve teacher relations that provide limited benefit for fine-grained action recognition. Scene context and object appearance can also affect similarity in egocentric videos, while SP's relation-matching objective does not explicitly use action labels to distinguish the task relevance of these relations. These explanations remain hypotheses for the observed result under our evaluated configuration.
SP preserves similarities at corresponding sample-pair positions. GSD instead matches ECF responses of the overall similarity distribution, allowing individual pairwise values to differ. This relaxation gives the heterogeneous student greater freedom to organize its representations. TCD complements GSD by aligning the allocation of similarity across task-defined semantic groups. We have added this distinction to Section 3.3.1, highlighted in blue.
Thank you for suggesting a label-only TCD target and a temporal-order control for TRD. The component ablations assess the effect of including each objective. A label-only target would help isolate the contribution of teacher relation strengths beyond label grouping. A temporal-order control would help assess the contribution of ordered feature changes.
These controls have not been completed within the available revision time. We have added a paragraph on the scope of the ablations in Section 4.3, highlighted in blue, to make the current evidence explicit. We will supplement these analyses in a more complete subsequent version.
Thank you for requesting clarification of the SSv2 setup and training configurations. The SSv2 student uses a single 174-class action-template classification head with one cross-entropy loss. TCD provided no training signal in this implementation. We retained the constrained adapter and trained with GSD, TRD, the classification objective, and the anchor penalty. Appendix A and both SSv2 table captions now explicitly identify the reported MARD results as the variant without TCD.
For initialization, MARD and AFT use the official Ego4D-pretrained AVION ViT-L checkpoint. The other EK100 distillation baselines use an AVION teacher fine-tuned on EK100. Both students use official Kinetics-400-pretrained weights. The VideoSwin-S checkpoint is
swin3d_s-da41c237.pth. These initialization details have been added to Section 4.1 in blue.Section 4.1 also reports the shared 50-epoch schedule, optimizer, learning rates, input sizes, and GSD frequency grid. Section 3.2 now specifies the adapter dimensions, biases, and 3,073 trainable parameters. Some method-specific coefficients and initialization details still require documentation. Complete training configurations will accompany the public code and evaluation scripts after the review process concludes.
Thank you for catching this transcription error. We inadvertently copied the VideoSwin-S + FBT results into the VideoSwin-S + KL row. The FBT results are 63.94% Top-1 and 88.89% Top-5. The correct KL results are 63.39% Top-1 and 89.09% Top-5. We have corrected Table 4 and highlighted the updated KL values in blue.