acceptodds
Under review as a conference paper at ICLR 2027

MARD: Multi-level Adaptive Representation Distillation across Heterogeneous Architectures for Egocentric Action Recognition

Abstract

Deploying powerful pretrained video models for egocentric action recognition is computationally expensive, motivating the transfer of their knowledge to compact students. However, heterogeneous teacher-student architectures organize video representations differently, making direct feature imitation unnecessarily restrictive. Beyond this architectural mismatch, our analysis further shows that higher teacher-student representation similarity does not necessarily lead to better recognition accuracy. Motivated by these observations, we propose Multi-level Adaptive Representation Distillation (MARD), which transfers architecture-agnostic structures from fixed features extracted by a general-purpose teacher while preserving the student's native representation. A lightweight adapter makes these fixed teacher features responsive to the target task. MARD then performs distillation at three complementary levels: Global Similarity Distillation matches the distributions of cross-video relations, Task Conditioned Distillation preserves how relational strength is allocated across verb-noun semantic groups, and Temporal Rhythm Distillation aligns temporal changes across sequences of different lengths. Since these objectives are constructed independently within the native teacher and student spaces, MARD requires neither matched feature dimensions nor aligned temporal tokens. On EPIC-KITCHENS-100, MARD improves action top-1 accuracy over the non-distilled VideoSwin-S and X3D-S baselines by 9.19 and 6.03 percentage points, respectively; on Something-Something V2, the corresponding top-1 gains are 9.54 and 5.30 percentage points. Across both datasets, MARD outperforms representative logit- and feature-based distillation methods with both students and requires no task-specific fine-tuning of the general-purpose teacher.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Action Recognition, Knowledge Distillation, Heterogeneous Teacher–Student Architectures, Video Understanding, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4YMTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MARD proposes heterogeneous video knowledge distillation using a frozen foundation-model teacher and a lightweight adapter. It introduces three objectives: Global Similarity Distillation (GSD) for relational distributions, Task Conditioned Distillation (TCD) for verb–noun semantic relations, and Temporal Rhythm Distillation (TRD) for temporal alignment. The method reports substantial improvements on EK100 and SSv2 with VideoSwin-S and X3D-S students.

Critical strengths
  1. A clean, dimension-agnostic design for heterogeneous distillation. The three objectives operate on structures computed within each model's native feature space, avoiding feature projection, token correspondence, and temporal resampling. The use of a frozen, precomputed teacher is also practically attractive, allowing reuse across downstream tasks.
  2. Potentially large and consistent gains. MARD reports 5.3–9.5 percentage-point top-1 improvements over non-distilled students across two datasets and two different student architectures.
Critical weaknesses
  1. The necessity of relaxing pairwise relation matching is insufficiently demonstrated. GSD matches only the overall distribution of sample similarities, unlike established methods such as Similarity-Preserving Knowledge Distillation (SP; Tung and Mori, ICCV 2019, https://arxiv.org/abs/1907.09682), which preserve pairwise correspondence without direct feature alignment. GSD therefore discards potentially meaningful instance-level relations, which TCD only partially recovers through semantic grouping. Comparisons of GSD, GSD+TCD, SP, and SP+TCD under identical settings would help establish the benefit of this design choice.
  2. The ablations in Table 3 demonstrate the effectiveness and complementarity of the proposed objectives but provide limited insight into their underlying mechanisms. The generality of TCD is further unclear, as the paper does not explain its adaptation, or that of the verb–noun classification loss, to SSv2's action-template labels. Missing key hyperparameters and initialization details also limit reproducibility.
Questions for authors
  1. Could the authors compare GSD, GSD+TCD, SP, and SP+TCD under identical settings to justify relaxing pairwise correspondence?
  2. Could the authors provide mechanism-level controls, such as a label-only TCD target and a temporal-order control for TRD?
  3. How are TCD and the classification loss implemented on SSv2, and could the authors provide the missing hyperparameters, initialization checkpoints, and baseline configurations?
  4. Table 4 reports identical results for KL and FBT (63.94 Top-1 / 88.89 Top-5); could the authors confirm whether these values are correct?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 4YMT

  1. Comparison with SP and the distinction from GSD

Thank you for suggesting the comparison with SP. We previously evaluated SP in preliminary video-distillation experiments, where it underperformed the student trained with classification supervision alone. This led us to omit SP from the original comparison. We have now included SP in the updated VideoSwin-S/EK100 comparison in Table 1 using seeds 42, 43, and 44. SP achieves 30.42±0.41% action Top-1, compared with 34.33±1.36% for the baseline, a decrease of 3.91 percentage points in mean accuracy.

One possible explanation is that differences in architecture, feature aggregation, and pretraining objectives cause the teacher's and student's global representations to encode different contextual and semantic relations. Pairwise matching may encourage the student to preserve teacher relations that provide limited benefit for fine-grained action recognition. Scene context and object appearance can also affect similarity in egocentric videos, while SP's relation-matching objective does not explicitly use action labels to distinguish the task relevance of these relations. These explanations remain hypotheses for the observed result under our evaluated configuration.

SP preserves similarities at corresponding sample-pair positions. GSD instead matches ECF responses of the overall similarity distribution, allowing individual pairwise values to differ. This relaxation gives the heterogeneous student greater freedom to organize its representations. TCD complements GSD by aligning the allocation of similarity across task-defined semantic groups. We have added this distinction to Section 3.3.1, highlighted in blue.

  1. Mechanism-level controls

Thank you for suggesting a label-only TCD target and a temporal-order control for TRD. The component ablations assess the effect of including each objective. A label-only target would help isolate the contribution of teacher relation strengths beyond label grouping. A temporal-order control would help assess the contribution of ordered feature changes.

These controls have not been completed within the available revision time. We have added a paragraph on the scope of the ablations in Section 4.3, highlighted in blue, to make the current evidence explicit. We will supplement these analyses in a more complete subsequent version.

  1. SSv2 implementation and reproducibility details

Thank you for requesting clarification of the SSv2 setup and training configurations. The SSv2 student uses a single 174-class action-template classification head with one cross-entropy loss. TCD provided no training signal in this implementation. We retained the constrained adapter and trained with GSD, TRD, the classification objective, and the anchor penalty. Appendix A and both SSv2 table captions now explicitly identify the reported MARD results as the variant without TCD.

For initialization, MARD and AFT use the official Ego4D-pretrained AVION ViT-L checkpoint. The other EK100 distillation baselines use an AVION teacher fine-tuned on EK100. Both students use official Kinetics-400-pretrained weights. The VideoSwin-S checkpoint is swin3d_s-da41c237.pth. These initialization details have been added to Section 4.1 in blue.

Section 4.1 also reports the shared 50-epoch schedule, optimizer, learning rates, input sizes, and GSD frequency grid. Section 3.2 now specifies the adapter dimensions, biases, and 3,073 trainable parameters. Some method-specific coefficients and initialization details still require documentation. Complete training configurations will accompany the public code and evaluation scripts after the review process concludes.

  1. Identical KL and FBT results in Table 4

Thank you for catching this transcription error. We inadvertently copied the VideoSwin-S + FBT results into the VideoSwin-S + KL row. The FBT results are 63.94% Top-1 and 88.89% Top-5. The correct KL results are 63.39% Top-1 and 89.09% Top-5. We have corrected Table 4 and highlighted the updated KL values in blue.

Reviewer_HypfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a multi-level distillation framework MARD for heterogeneous video action recognition. A frozen general-purpose video teacher (AVION ViT-L) is paired with a lightweight student (VideoSwin-S or X3D-S). MARD adds a constrained residual adapter to teacher features and transfers three architecture-agnostic structures: global cross-video similarity distributions via empirical characteristic functions (GSD), verb–noun-conditioned relation distributions via Jensen–Shannon divergence (TCD), and temporal rhythm via Soft-DTW on adjacent mean-pooled responses (TRD). Experiments on EPIC-KITCHENS-100 and Something-Something V2 show consistent gains over non-distilled and representative distillation baselines, with ablations on X3D-S/EK100 supporting component complementarity.

Critical strengths
  1. This paper introduces a clear and practical formulation for heterogeneous video distillation without requiring feature-dimension or temporal-resolution correspondence.
  2. The three proposed complementary objectives target global, task-semantic, and temporal structures; ablations support their complementarity.
Critical weaknesses
  1. The statistical and ablation evidence is insufficient: no mean/std over seeds, and ablations only on X3D-S/EK100, leaving the robustness of GSD/TCD/TRD and the adapter unverified across settings.
  2. The 'frozen teacher' claim is a bit ambiguous, and the comparison is somehow unfair: while the teacher is frozen, the adapter is trained on labels and distillation. Baselines differ in teacher fine-tuning and parameter budgets; controlled, parameter-matched comparisons are necessary. In addition, sensitivity to GSD frequencies, temperatures, loss weights, adapter size, and training/inference cost is not reported.
Questions for authors
  1. How many parameters and how much training compute does the constrained adapter add? Can you compare against baselines with matched teacher-side adaptation budgets?
  2. Can you provide full ablations on VideoSwin-S and SSv2, and sensitivity to hyper parameters of Tau_g and loss weights?
  3. Please clarify Eq. 3 and the exact residual constraint, and whether the adapter should be considered a form of target-specific teacher adaptation.
  4. Why there is no 'Conclusion' section?
  5. Will code and evaluation scripts be released?
1 comment
AuthorsOpenReview ↗

Response to Reviewer hstn

  1. AVION teacher accuracy on EK100

Thank you for pointing out this discrepancy. The original 46.36% result came from our reproduction. We used the annotated start and end frame numbers directly as indices into the decoded videos. Our inspection found that these indices use a 60 fps time base, whereas most videos we checked were at 59.94 fps and some were at 29.97 fps. This mismatch shifted some sampled clips away from their annotated action intervals.

Almost all of our EK100 experiments used teacher features cached through this pipeline. Following the sampling correction, the re-evaluated teacher achieves 52.08% action Top-1, reducing the gap to the reported 54.4% from 8.04 to 2.32 percentage points. We have also updated the VideoSwin-S/EK100 comparisons using three random seeds. Corrected X3D-S comparisons and component ablations remain to be completed. The remaining teacher discrepancy has not yet been fully explained.

  1. TCD implementation on SSv2

Thank you for requesting clarification of the SSv2 setup. Our SSv2 experiments used action-template labels, with a single 174-class classification head and one cross-entropy loss. TCD provided no training signal in this implementation. We retained the constrained adapter and trained with GSD, TRD, the classification objective, and the anchor penalty.

We have clarified these objectives in Appendix A and explicitly stated in both SSv2 table captions that the reported MARD results use GSD and TRD without TCD. The contribution of verb–noun-conditioned TCD is assessed separately on EK100.

  1. Connection to egocentric recognition

Thank you for asking us to clarify the task motivation. MARD focuses on transferring knowledge from an Ego4D-pretrained teacher to lightweight heterogeneous students for egocentric action recognition. Its task-specific design primarily lies in TCD, which uses EK100's verb–noun composition to distinguish videos sharing an entire action from those sharing only a verb or noun. GSD and TRD provide global relational and temporal supervision for this task. We did not introduce dedicated mechanisms for camera geometry, camera motion, or hand localization.

EK100 is our main evaluation setting. SSv2 provides an additional video-interaction recognition evaluation using the variant without TCD. More detailed evidence isolating the mechanisms behind each objective remains to be established.

  1. Experimental coverage and video-distillation baselines

Thank you for recommending broader coverage and comparisons with ASCD and SAKD. After identifying the EK100 sampling error raised in your first question, we prioritized correcting the pipeline, regenerating teacher features, and repeating the VideoSwin-S/EK100 experiments with three seeds.

Within the available revision time, we have not completed the ASCD and SAKD reproductions. We recognize the value of these comparisons and will include them in a more complete subsequent version, together with further evaluation across teacher and student configurations.

  1. Repeated-run statistics

Thank you for recommending repeated-run statistics. Table 1 now reports means and standard deviations for the VideoSwin-S/EK100 baseline, comparison methods, and MARD using seeds 42, 43, and 44. MARD achieves 39.79±0.57% action Top-1, compared with 34.33±1.36% for the baseline and 38.95±0.37% for PAT.

These additional statistics cover VideoSwin-S/EK100. Repeated runs for SSv2/X3D-S remain incomplete, so the 0.25-point margin over RLD should be interpreted cautiously. We will supplement the remaining settings in a subsequent version.

Reviewer_hstnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MARD, a framework for transferring knowledge from a frozen pretrained video teacher to heterogeneous student architectures for action recognition. A constrained residual adapter adjusts cached teacher features, while three objectives transfer complementary structures: Global Similarity Distillation matches distributions of cross-video similarities through empirical characteristic functions; Task Conditioned Distillation aligns similarities aggregated into verb–noun relation groups; and Temporal Rhythm Distillation aligns temporal differences of scalar feature responses using Soft-DTW divergence. Experiments on EPIC-KITCHENS-100 and Something-Something V2 demonstrate improvements for VideoSwin-S and X3D-S without target-dataset fine-tuning of the teacher backbone or additional student inference modules.

Critical strengths
  1. The paper addresses a practically relevant distillation setting. Reusing a frozen pretrained video teacher across different student architectures can reduce teacher preparation costs. Constructing relational objectives within each model’s own representation space accommodates differences in feature dimensions and layouts, while cached teacher features and an unchanged student inference architecture make the approach potentially useful for deploying smaller video models.

  2. The reported gains are substantial in several configurations, with useful component ablations. On EK100, MARD improves action top-1 accuracy over the strongest compared method by 4.95 percentage points for VideoSwin-S and 2.00 points for X3D-S. The SSv2 results provide additional evidence beyond the main dataset. Table 3 evaluates individual objectives, objective pairs, and the full method with and without the adapter, providing initial support for the complementarity of the proposed supervision signals.

Critical weaknesses
  1. Limited experimental coverage and missing video-distillation baselines. The evaluation uses a single teacher, AVION ViT-L, and two fixed student architectures, with component ablations restricted to X3D-S on EK100. This provides initial evidence but limited support for generality across teacher representations and student capacities. Moreover, directly relevant video-distillation methods such as ASCD, already cited in Section 2.3, and SAKD are missing from the comparisons. Including representative video-specific baselines would better establish MARD’s advantages. Repeated-run statistics would also help assess the smaller gains, particularly the 0.25-point improvement over RLD on SSv2 with X3D-S. ASCD; SAKD.

  2. The task-specific benefits and some central design choices remain insufficiently validated. GSD captures general relational information, TCD uses compositional labels, and TRD models temporal variation, but the experiments do not establish how these choices specifically address egocentric recognition challenges. Meanwhile, the manuscript does not explain how the verb–noun-based TCD groups are instantiated on SSv2, leaving the additional evaluation underspecified.

Questions for authors

Why does the fine-tuned AVION ViT-L teacher in Tables 1 and 2 achieve only 46.36% action top-1 accuracy on EK100? The original AVION paper and its official model zoo report 54.4% for the fine-tuned ViT-L model, a difference of 8.04 percentage points.

1 comment
AuthorsOpenReview ↗

Response to Reviewer hstn

  1. AVION teacher accuracy on EK100

Thank you for pointing out this discrepancy. The original 46.36% result came from our reproduction. We used the annotated start and end frame numbers directly as indices into the decoded videos. Our inspection found that these indices use a 60 fps time base, whereas most videos we checked were at 59.94 fps and some were at 29.97 fps. This mismatch shifted some sampled clips away from their annotated action intervals.

Almost all of our EK100 experiments used teacher features cached through this pipeline. Following the sampling correction, the re-evaluated teacher achieves 52.08% action Top-1, reducing the gap to the reported 54.4% from 8.04 to 2.32 percentage points. We have also updated the VideoSwin-S/EK100 comparisons using three random seeds. Corrected X3D-S comparisons and component ablations remain to be completed. The remaining teacher discrepancy has not yet been fully explained.

  1. TCD implementation on SSv2

Thank you for requesting clarification of the SSv2 setup. Our SSv2 experiments used action-template labels, with a single 174-class classification head and one cross-entropy loss. TCD provided no training signal in this implementation. We retained the constrained adapter and trained with GSD, TRD, the classification objective, and the anchor penalty.

We have clarified these objectives in Appendix A and explicitly stated in both SSv2 table captions that the reported MARD results use GSD and TRD without TCD. The contribution of verb–noun-conditioned TCD is assessed separately on EK100.

  1. Connection to egocentric recognition

Thank you for asking us to clarify the task motivation. MARD focuses on transferring knowledge from an Ego4D-pretrained teacher to lightweight heterogeneous students for egocentric action recognition. Its task-specific design primarily lies in TCD, which uses EK100's verb–noun composition to distinguish videos sharing an entire action from those sharing only a verb or noun. GSD and TRD provide global relational and temporal supervision for this task. We did not introduce dedicated mechanisms for camera geometry, camera motion, or hand localization.

EK100 is our main evaluation setting. SSv2 provides an additional video-interaction recognition evaluation using the variant without TCD. More detailed evidence isolating the mechanisms behind each objective remains to be established.

  1. Experimental coverage and video-distillation baselines

Thank you for recommending broader coverage and comparisons with ASCD and SAKD. After identifying the EK100 sampling error raised in your first question, we prioritized correcting the pipeline, regenerating teacher features, and repeating the VideoSwin-S/EK100 experiments with three seeds.

Within the available revision time, we have not completed the ASCD and SAKD reproductions. We recognize the value of these comparisons and will include them in a more complete subsequent version, together with further evaluation across teacher and student configurations.

  1. Repeated-run statistics

Thank you for recommending repeated-run statistics. Table 1 now reports means and standard deviations for the VideoSwin-S/EK100 baseline, comparison methods, and MARD using seeds 42, 43, and 44. MARD achieves 39.79±0.57% action Top-1, compared with 34.33±1.36% for the baseline and 38.95±0.37% for PAT.

These additional statistics cover VideoSwin-S/EK100. Repeated runs for SSv2/X3D-S remain incomplete, so the 0.25-point margin over RLD should be interpreted cautiously. We will supplement the remaining settings in a subsequent version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.