- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduced an atomic-action representation learning framework called InvAct. InvAct involves a dedicated scene-aware contrastive objective that reduces scene dependence while maintaining viewpoint invariance. A structured token commitment is proposed to restrict each video token to reading from either of clsass token or register token. InvAct is trained using atomic-action supervision and evaluated on atomic and keystep retrieval, transfer to unseen datasets, and robot-policy pretraining. The paper addresses a relevant limitation of cross-view action understanding. However, I have concerns about the novelty of the contrastive objective and temporal aggregation, and whether the comparisons sufficiently distinguish the proposed contributions from the benefits of the underlying representation and training setup.
Critical strengths
- InvAct achieves the best average atomic retrieval results among the evaluated methods in Table 1, outperforming prior ego-video pretrained models, robotic-centric methods and domain agnostic approaches.
- The distinction between viewpoint invariance and scene invariance is meaningful. Matching synchronized ego–exo views can exploit shared environments, and evaluating action retrieval across scenes helps expose this limitation.
- In Figures 4 and 5, the proposed token re-routing shows good separation of action and background clues.
Critical weaknesses
-
The novelty of the scene-aware contrastive objective is insufficiently established relative to EgoNCE. EgoVLP (Lin et al., NeurIPS 2022) already motivates action-aware positive sampling and scene-aware hard-negative sampling by the same two challenges: matching the same action across different scenarios and distinguishing different actions within the same scenario. InvAct only differs in using video–video alignment and explicit action/scene identities, but the conceptual overlap is substantial.
-
The comparisons do not adequately isolate improvements over existing methods from differences in inputs, pretraining, and supervision. Table 1 compares released checkpoints with different architectures, modalities, and training data. More tellingly, the Table 13 base configuration, which already uses DINOv3 and flow but excludes scene repulsion and token commitment, achieves 30.17/29.12 cross-scene/cross-view atomic retrieval—already exceeding the corresponding averages of all existing methods in Table 1. This suggests that a substantial advantage exists before adding these two proposed components. The controlled ablations are valuable, but they do not replace comparisons against established objectives trained with the same backbone, modalities, supervision, and budget.
-
The goal of adding flow in this work is not clear, while most prior ego-video pretrained approaches are RGB-only.
-
L191-194: In figure 3, the video features and flow features should be extracted separately. The current figure shows video features -> tokenizer, which is quite confusing.
-
Two minor issues: 5.1 averaging atomic embeddings for keystep retrieval provides limited methodological novelty. Aggregating short-clip representations into a longer-video representation is a longstanding approach, including segment-consensus methods such as early TSN (Wang et al., 2017). Nevertheless, average pooling itself is a conventional operation and does not model temporal order or action composition. The paper should distinguish this empirical transfer result from a new compositional mechanism and compare other atomic encoders under the same segmentation and pooling protocol.
5.2 The main unseen-dataset comparison should include Charades-Ego. Table 2 presents SSV2 and NTU, whereas Charades-Ego appears only in the appendix. Table 11 reports 16.45 for InvAct versus 17.98 for Rosetta, and nearly identical unseen-verb results, 39.85 versus 39.91. The paper acknowledges these results, but including them in the main comparison would provide a more balanced account of generalization and make the tradeoff between suppressing scene shortcuts and retaining useful contextual information more visible.
Questions for authors
- What's the difference between the proposed objective and EgoNCE? Refer to Q1 for details.
- Can you perform joint training by combining atomic action-level supervision and keystep-level supervision?
- In Table 12, what does InvAct + DINOv3 mean? If I understood correctly, InvAct already contains DINOv3 visual features.
4 comments
Reply for oaLo [2/2]
[W2] Comparison with the same backbone and budget
"they do not replace comparisons against established objectives trained with the same backbone, modalities, supervision, and budget."
We compare with the backbone fixed in two settings, one CLIP-based and one DINOv3-based.
CLIP-based (Tab 4, Sec 4.3). The submission already ablated our components on a CLIP-based model that, like Viewpoint Rosetta, starts from text and view alignment. We now add SSF@1 to make the effect on scene bias clear (single run, losses in App B.2 and B.3).
| Tab 4 (CLIP and flow) | CS ↑ | CV ↑ | SSF@1 ↓ |
|---|---|---|---|
| Text alignment | 25.61 | 25.45 | 0.81 |
| + view invariance | 26.41 | 27.16 | 0.88 |
| + action attraction | 26.93 | 25.45 | 0.83 |
| + scene repulsion | 28.20 | 27.14 | 0.63 |
| + token commitment | 28.85 | 27.54 | 0.59 |
Aligning views raises SSF@1 from 0.81 to 0.88, and our components then add 2.44 / 0.38 points and lower it to 0.59.
DINOv3-based (new Tab 6, Sec 4.3, top block). Every row trains our encoder on frozen DINOv3 features and flow with the same clips and schedule over three seeds, and the established objectives use a tuned learning rate and temperature. The Rosetta-style row uses only Rosetta's own text and view losses.
| Objective | CS ↑ | CV ↑ | SSF@1 ↓ |
|---|---|---|---|
| Rosetta-style | 29.62 ± 0.14 | 30.83 ± 0.16 | 0.90 ± 0.01 |
| SupCon | 30.52 ± 0.13 | 29.80 ± 0.10 | 0.86 ± 0.01 |
| EgoExoNCE | 30.21 ± 0.16 | 29.81 ± 0.12 | 0.89 ± 0.01 |
| Our loss | 31.57 ± 0.10 | 31.05 ± 0.11 | 0.58 ± 0.02 |
| Our loss + token commitment | 32.30 ± 0.07 | 31.50 ± 0.07 | 0.53 ± 0.01 |
All established objectives stay scene-dependent (SSF@1 ≥ 0.86), and our loss leads the best of them by 1.05 points across scenes (standard errors ≤ 0.16). The Rosetta-style objective stays close across views (0.22 points, within noise), since view alignment is its goal, and with token commitment the gap grows to 0.67. On the base of your Table 13 (now Tab 15), our components add 2.13 / 2.38 points over three seeds (p < 0.03) and lower SSF@1 from 0.79 to 0.53. Tab 1, in turn, compares released systems as a whole, as Viewpoint Rosetta and BYOV also do. Its caption and Sec 4.1 clarify that.
[W4] Fig 3
We agree. The Fig 3 caption now names RAFT flow and DINOv3 as the video features, and Sec 3.1 states that flow forms the video tokens while DINOv3 enters only through cross-attention. The final version will draw the two streams separately.
[5.1] Averaging for keysteps
We agree that average pooling is conventional, as in TSN. We therefore present keystep retrieval as an empirical transfer result and not as a compositional mechanism. All Tab 1 baselines use the same segmentation and pooling as InvAct, and Sec 4.2 now says so and notes that averaging ignores order (see also App G).
[5.2] Charades-Ego
Tab 2 (Sec 4.2) now includes Charades-Ego. Rosetta leads there (17.98 vs 16.45) and ties on unseen verbs (39.91 vs 39.85), likely because static scene context helps with many Charades-Ego labels. Adding a static DINOv3 feature to InvAct raises it to 19.72, above Rosetta, at a cost on Ego-Exo4D (32.16 → 29.70, Tab 13 in App C.4). Sec 4.2 now states this tradeoff.
[Q2] Joint atomic and keystep training
Yes. With keystep labels added, InvAct matches Viewpoint Rosetta on keystep retrieval across views, and its lead across scenes grows (App D.7, Tab 20).
| Model and labels | Atomic CS | Atomic CV | Keystep CS | Keystep CV |
|---|---|---|---|---|
| InvAct, atomic | 32.53 | 31.79 | 41.47 | 35.09 |
| InvAct, atomic and keystep | 32.41 | 31.62 | 42.10 | 37.64 |
| Viewpoint Rosetta | 21.66 | 21.43 | 39.12 | 37.35 |
Atomic retrieval drops by only 0.12 / 0.17 points.
[Q3] InvAct + DINOv3
You are right that InvAct already reads DINOv3, but only through cross-attention. That row, now "InvAct + static DINOv3 feature" in Tab 13, appends the clip's mean-pooled DINOv3 feature to the embedding. App C.4 provides further explanation.
Thank you again for the concrete questions, which made the paper clearer. We hope the controlled comparisons address your concerns, and we would be glad if you would consider raising your score.
Dear Reviewer,
Thank you again for your valuable feedback, recommendations, and corrections. Since we are approaching the end of the discussion session, we would sincerely appreciate your thoughts on our responses and are happy to address any remaining questions.
Thank you for your time and consideration.
Dear Reviewer,
Since we are nearly ending the discussion phase, we would really like to hear your response on the new experiments and clarifications, and whether they do or do not address your concerns. Thank you for your consideration.
Reply for oaLo [1/2]
Thank you for the detailed review. We are glad you find the distinction between viewpoint and scene invariance meaningful and the separation of action and background cues in Figs 4 and 5 convincing. All additions below are in the revised PDF, with edited and new content in blue. New tables shift the appendix table numbers, so all numbers below refer to the revised PDF.
[W1, Q1] Difference from EgoNCE
Our loss builds on EgoNCE, just as EgoNCE builds on InfoNCE. Extending EgoNCE for a new goal is common. HierVL (Ashutosh et al., 2023), EgoNCE++ (Xu et al., 2025) and EgoExoNCE (Xu et al., 2024) extend it for long-term intent, hand-object interactions and ego-exo captioning, and Sec 2 notes that all three keep positives from the query's own scene. Our goal is view and scene invariance, hence we change which clips may be positives. Sec 1 and Sec 3.2 now present our loss as such an extension, with three rules (Tab 10 in App B.1).
Like EgoNCE, we keep same-scene clips of other actions as negatives (scene repulsion). The new Tab 6 (Sec 4.3, bottom block) adds the rules to EgoNCE one at a time over three seeds, with our encoder, data and budget fixed.
Together the three rules add 1.69 / 1.50 points over EgoNCE and lower SSF@1 from 0.88 to 0.58. Our sampler alone helps little (0.54 / 0.16), because the synchronized view still dominates the positives. Removing same-scene positives is therefore what lowers scene bias, first the synchronized view (0.87 → 0.70) and then the remaining same-scene clips (0.70 → 0.58). Fig 2 shows how view alignment alone can still retrieve by scene.
[W3] Why flow
Flow gives the model explicit motion, while DINOv3 enters only through cross-attention (Sec 3.1). Tab 18 (App D.5) trains InvAct without flow.
Flow adds 2.73 / 2.29 points and lowers SSF@1. Even without flow, InvAct beats every prior method on the atomic averages of Tab 1, so the gain does not come from flow alone. App D.2 also compares other input combinations (Tab 16).