- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper proposes AnchorMAE for ego–exo video representation learning, aiming to capture both task semantics and fine-grained temporal correspondence. The framework uses a visual pathway and a training-only task-conditioned pathway that share a temporal encoder. It combines masked visual reconstruction, clip-level alignment, and same-video frame-level semantic transfer. Only the visual pathway is used at inference. The method is evaluated on AE2 and Ego-Exo4D, with component ablations and representation diagnostics.
Critical strengths
- The paper raises a relevant question within ego–exo representation learning: how to incorporate task semantics while preserving fine-grained temporal information. It proposes a concrete framework for investigating this question and reports useful empirical improvements, supported by evaluations and component ablations.
Critical weaknesses
-
The specific limitation of prior work is not clearly established. It is difficult to determine whether existing methods fail to capture both task identity and temporal phase, or whether they capture both but combine their supervision inadequately. Since the paper acknowledges prior approaches incorporating semantic and temporal learning, this distinction is essential to understanding what problem remains unresolved and what constitutes the proposed advance.
-
The connection between this problem and the architectural design is insufficiently explained. The two pathways differ in access to task text; they do not explicitly separate task identity from temporal phase. It remains unclear why transferring task-conditioned frame representations through this structure should achieve the desired balance. In particular, the paper could better explain how semantic transfer preserves phase distinctions rather than encouraging similarity across frames of the same activity. The ablations report benefits from the proposed components, but do not sufficiently clarify the reasoning that connects the design to the motivating limitation.
Questions for authors
-
Would it be possible to include qualitative examples like Figure 3 for a wider range of labels in the main paper? Such examples, with more visible temporal progression cues, would help readers see the motivating problem and how the framework distinguishes task separation, temporal structure, and cross-view alignment.
-
If visual-only inference is one of the contributions, could you indicate in the results tables which compared methods support visual-only inference and which require additional inputs at inference time? This would help clarify how much of the reported advantage comes from the framework itself rather than from differences in the inference setting.
3 comments
Follow-up assessment
Thank you for the revisions. The explanation of how frame-level semantic transfer works alongside visual reconstruction clarifies the design. The revised Figure 3 addresses my question about temporal progression, and I appreciate the clarification that visual-only inference is a deployment property of AnchorMAE.
My remaining concern is that the paper does not yet clearly show where prior methods’ clip-level supervision falls short and how frame-level transfer addresses that limitation. Most of my initial concerns have been addressed, so I am raising my rating from 1 to 2 (Weak reject).
Follow-up on Clip-Level vs Frame-Level Supervision
Thank you for the follow-up and for raising the rating. We strengthened the remaining clip-level-vs-frame-level supervision point both conceptually and experimentally.
The revision now states explicitly that clip-level alignment constrains only the pooled video representation, whereas same-video transfer supervises each visual token from its task-conditioned counterpart at the same frame index.
Main-paper Table 6 tests this directly: clip-level-only/no-transfer gives 0.8096/0.7826/0.9072/0.9211 (F1/mAP@10/R²/τ), pooled broadcast gives 0.8145/0.7894/0.8993/0.9747, shuffled-index transfer drops to 0.7984/0.7554/0.8881/0.6947, and same-index transfer is strongest on all four metrics at 0.8687/0.8264/0.9247/0.9966.
This directly separates exposure to task semantics from preservation of frame-specific correspondence.
Response to Reviewer gj1S
Thank you for identifying that the original paper did not make the problem-to-design connection sufficiently explicit. We revised the framing, mechanism explanation, diagnostics, and qualitative evidence around task identity versus temporal phase.
W1 — What remains unresolved relative to prior work? The revision no longer suggests that prior methods lack temporal learning or semantics. Instead, the Introduction and Related Work explicitly acknowledge BYOV, MILES/TGM, VRS, PRISM, and feature distillation. Figure 1 now summarizes the narrower remaining interface question: how can task-level semantics supervise individual deployable visual frames while dense reconstruction remains pre-semantic, synchronized ego–exo frame matching is not assumed, and text is confined to training?
W2 — Why should the architecture address this limitation? We now state explicitly that AnchorMAE does not mathematically disentangle task identity and temporal phase. The visual path retains masked reconstruction against detached pre-semantic targets, while the training-only semantic path supplies task semantics; detached same-video frame transfer connects the two at matching frame indices within each video. Figure 2 shows this separation directly. Table 6 tests transfer removal, reconstruction-target type, and negative handling; Table 9 separately measures task/semantic and temporal/viewpoint diagnostics. We retain the modest temporal-distractor result (75.04% versus 74.97% Recon-only) rather than overclaiming it.
Q1 — Wider labels and visible temporal progression. We added a dedicated main-paper qualitative analysis in Fig. 3. It uses three Tennis Forehand ego queries from Early (Backswing), Mid (Hit), and Late (Front Swing) phases, each with its actual Top-3 ego->exo retrievals. The activity and retrieval direction remain fixed while the query phase changes, making the progression directly visible. Table 8 reports all four AE2 activities separately for Recon+Align and AnchorMAE. Supplement Fig. 7 further broadens the qualitative analysis to Break Eggs, Pour Liquid, Tennis Forehand, and Pour Milk in both ego->exo and exo->ego directions. Supplement Figs. 4–6 provide the model-comparison, embedding, and failure analyses.
Q2 — Visual-only inference. We revised this claim to avoid implying that competing representation-learning methods necessarily require text at inference. For AnchorMAE, visual-only inference is stated only as a deployment property: the text encoder, SAA modules, semantic forward pass, masked decoders, and training losses are removed, leaving the unconditioned visual pathway and downstream heads. This is stated in Sec. 3, illustrated in Fig. 2, and reiterated in the Ego-Exo4D Table 3 caption.
Together, Figs. 1–3, Tables 5–9, and Supplement Figs. 4–7 make the intended gap -> mechanism -> evidence chain explicit while keeping the qualitative claims appropriately scoped.