- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose pi-Ego, a poly-modal-induced Ego encoder learned through a multi-teacher distillation framework (i.e., ProxyDist). It learns from representation-specific proxies that translate heterogeneous knowledge into a unified Ego representation space to mitigate representation gaps. Then, Selective Proxy Distillation (SPD), which selectively distills from the most reliable proxies for each training sample. The proposed method achieves SOTA performance on multiple downstream tasks on three commonly adopted Ego datasets, which further demonstrates its effectiveness.
Critical strengths
1/ The paper is well-motivated and clearly identifies two key challenges of multi-teacher distillation in egocentric video representation learning: representational gaps and conflicting gradients.
2/ The paper is well-written and easy to follow.
3/ The proposed method achieves strong performance across multiple mainstream egocentric benchmarks and different downstream tasks. Extensive experiments further demonstrate the effectiveness of the proposed method.
4/ The proposed method is efficient at inference, as it only requires egocentric RGB videos without relying on additional data, such as exocentric videos or other modalities.
Critical weaknesses
1/ My primary concern is whether the proposed method is specifically designed for Ego scenes and tasks. Although the method achieves promising performance, it remains unclear which technical designs are tailored to the unique challenges of egocentric videos. The proposed framework appears to be a general multi-teacher distillation approach rather than a method specifically designed to address egocentric challenges, such as limited field of view and occlusion.
2/ The authors should further clarify the definition of "poly-modal". For example, egocentric and exocentric RGB videos represent different viewpoints rather than different modalities. A clearer distinction among these concepts would help justify the use of the term "poly-modal".
3/ The proposed framework relies on multiple modalities and viewpoints during training. Although several Ego-Exo datasets have been introduced, not all of them provide the complete set of modalities required by the teacher models. It remains unclear how the proposed method handles missing modalities during training. It would be helpful if the authors could provide further discussion or analysis on this issue.
4/ Have the authors considered the EgoMe dataset? It would be helpful if the authors could conduct experiments on EgoMe to further evaluate the generalizability of the proposed method.
Overall, despite the remaining concerns, the paper is well-motivated, clearly written, and demonstrates strong experimental results. Therefore, my initial rating is 3: Weak Accept. I will consider adjusting my rating according to the authors’ rebuttal and other reviewers’ comments.
Questions for authors
See weaknesses.
4 comments
Response to Reviewer WMis [Part 2/2]
Q4. Experiment on EgoMe dataset
We thank the reviewer for this suggestion. Following the reviewer’s suggestion, we evaluated -EGO on EgoMe [4], which contains 7,902 paired egocentric–exocentric videos spanning 184 action categories. Following GCEAN [5] and Ego-PMOVE [6], we retain only correctly imitated pairs (Video type = True), yielding 3,709 training, 800 validation, and 1,643 test pairs under the official split. We compare -EGOEgo against an egocentric RGB TimeSformer baseline and direct multi-teacher distillation. We also report the mean ± standard deviation of accuracy across the nine proxies.
| Method | Accuracy |
|---|---|
| RGB Baseline | 80.9 |
| Nine proxies (mean ± std) | 82.1 ± 0.4 |
| Multi-teacher Distillation | 82.2 |
| -EGO | 84.4 |
-EGO outperforms the RGB baseline by 3.5% and direct multi-teacher distillation by 2.2%. These results provide additional evidence that the benefits of our framework extend across datasets. We have revised the manuscript to include this dataset. [L344-L347; L357-L367]
[4] H. Qiu, Z. Shi, L. Wang, H. Xiong, X. Li, and H. Li, "EgoMe: A new dataset and challenge for following me via egocentric view in real world," 2025, arXiv:2501.19061.
[5] Z. Shi, H. Qiu, L. Wang, Q. Wu, F. Meng, and H. Li, "Unsupervised ego- and exo-centric dense procedural activity captioning via gaze consensus adaptation," in Proc. 33rd ACM Int. Conf. Multimedia (MM), Dublin, Ireland, 2025, pp. 3731–3740, doi: 10.1145/3746027.3755167.
[6] H. Qiu, L. Wang, T. Zhao, Z. Shi, X. Li, L. Xu, and H. Li, "Ego-PMOVE: Prompt-aware mixture of view experts network for egocentric gaze prediction," in Proc. AAAI Conf. Artif. Intell., vol. 40, no. 10, 2026, pp. 8565–8573, doi: 10.1609/aaai.v40i10.37808.
Thanks for your rebuttal. I'd like to keep my positive rating.
Thank you for feedback and positive score.
We thank the reviewer again for the valuable feedback and for the positive score.
Response to Reviewer WMis [Part 1/2]
W1/Q1. Motivation and Design for Egocentric Video Understanding
We thank the reviewer for raising this concern. Our design is motivated by the partial observations inherent to egocentric videos: limited field of view and occlusion make complementary modalities and exocentric views valuable sources of privileged supervision during training. Our proxies transfer this heterogeneous knowledge into egocentric RGB models, while sample-dependent proxy selection accommodates variation in each source’s relevance across observations. The final student requires only egocentric RGB at inference, without additional sensors or exocentric views. Therefore, egocentric video understanding provides a particularly suitable testbed for our method, which addresses observation constraints through privileged knowledge transfer. Its potential applicability beyond this setting does not diminish the intellectual merit in our contribution and offers opportunities for broader use.
W2/Q2. Clarification on the term “poly-modal”
We thank the reviewer for this question. We agree that egocentric and exocentric RGB videos represent different viewpoints, not different modalities. In our paper, “poly-modal” describes a pool of privileged teachers that varies along three distinct axes:
Input modality: RGB and skeleton.
Viewpoint: egocentric and exocentric.
Representation: geometry-oriented features from Depth Anything, visual features from DINOv2, and vision-language features from SigLIP.
As described in Lines 46-48, we treat each teacher, i.e. its input together with its pretrained encoder, as a virtual sensor providing a potentially complementary signal about the same action. We therefore use “poly-modal” as an umbrella term for teacher heterogeneity, without equating viewpoints or learned representations with distinct physical modalities.
This terminology is motivated by the broad definition of modality as “a way in which a natural phenomenon is perceived or expressed” [1] and related work incorporating derived signals: CMC [2] treats depth and optical flow as distinct views, while π-VAD [3] uses “poly-modal” for signals including depth, pose, and optical flow extracted from RGB. These precedents contextualize our usage, while the distinction among modalities, viewpoints, and representations remains important.
[1] P. P. Liang, A. Zadeh, and L.-P. Morency, “Foundations & trends in multimodal machine learning: Principles, challenges, and open questions,” ACM Comput. Surv., vol. 56, no. 10, Art. no. 264, 2024, doi: 10.1145/3656580.
[2] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Glasgow, U.K., Aug. 2020, pp. 776–794, doi: 10.1007/978-3-030-58621-8_45.
[3] S. Majhi, G. D’Amicantonio, A. Dantcheva, Q. Kong, L. Garattoni, G. Francesca, E. Bondarev, and F. Brémond, “Just dance with π! A poly-modal inductor for weakly-supervised video anomaly detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2025, pp. 24265–24274.
Q3.Addressing Missing Modalities during training
We thank the reviewer for this question. When a training sample lacks a modality or viewpoint, such as an exocentric video (for example, 94 out of 1776 videos in EgoExoFitness), we mask the corresponding teacher-distillation loss in Level-I. The associated proxy still learns from action classification supervision on the egocentric RGB input. Level-II requires only the trained proxies and egocentric RGB inputs, so it does not require the original modalities or viewpoints to be available.
Furthermore, Table 10 (Appendix E) evaluates the case where exocentric videos are entirely unavailable. Using only ego-view teachers, π-EGO still improves over the baseline by 2.8% on EEF, 1.5% on A101, and 0.8% on EE4D. These results demonstrate that the framework remains effective without the complete teacher set. We have revised Section 4.1 to make this process explicit [L313 - L314].