MARD: Multi-level Adaptive Representation Distillation across Heterogeneous Architectures for Egocentric Action Recognition

Deploying powerful pretrained video models for egocentric action recognition is computationally expensive, motivating the transfer of their knowledge to compact students. However, heterogeneous teacher-student architectures organize video representations differently, making direct feature imitation unnecessarily restrictive. Beyond this architectural mismatch, our analysis further shows that higher teacher-student representation similarity does not necessarily lead to better recognition accuracy. Motivated by these observations, we propose Multi-level Adaptive Representation Distillation (MARD), which transfers architecture-agnostic structures from fixed features extracted by a general-purpose teacher while preserving the student's native representation. A lightweight adapter makes these fixed teacher features responsive to the target task. MARD then performs distillation at three complementary levels: Global Similarity Distillation matches the distributions of cross-video relations, Task Conditioned Distillation preserves how relational strength is allocated across verb-noun semantic groups, and Temporal Rhythm Distillation aligns temporal changes across sequences of different lengths. Since these objectives are constructed independently within the native teacher and student spaces, MARD requires neither matched feature dimensions nor aligned temporal tokens. On EPIC-KITCHENS-100, MARD improves action top-1 accuracy over the non-distilled VideoSwin-S and X3D-S baselines by 9.19 and 6.03 percentage points, respectively; on Something-Something V2, the corresponding top-1 gains are 9.54 and 5.30 percentage points. Across both datasets, MARD outperforms representative logit- and feature-based distillation methods with both students and requires no task-specific fine-tuning of the general-purpose teacher.