acceptodds
Under review as a conference paper at ICLR 2027

InvAct: Reducing Scene Bias in View-Invariant Atomic Action Learning

Abstract

View-invariant action learning methods aim to match actions from videos across ego-exo views. However, because these views also share the same environment, models can match scenes rather than actions, limiting fine-grained atomic action understanding. We address this with InvAct, an atomic action embedding model designed for viewpoint and scene invariance. To reduce reliance on scene cues, we propose structured token commitment, a hard row-wise attention mask that routes each video token to a single special-token slot. Action-relevant tokens align with the class token, while non-action tokens align with register tokens that are discarded at inference. We further introduce scene-aware contrastive objective that attracts same-action clips across scenes while separating different actions within a scene. On Ego-Exo4D, InvAct improves cross-scene and cross-view atomic retrieval while reducing same-scene bias. Importantly, despite training only on atomic actions, combining its embeddings provide strong longer action retrieval without retraining. The same checkpoint transfers to unseen datasets without finetuning and provides useful latent targets for robot-policy pretraining. These results highlight scene dependence as an important consideration in view-invariant action learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
action learning, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oaLoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduced an atomic-action representation learning framework called InvAct. InvAct involves a dedicated scene-aware contrastive objective that reduces scene dependence while maintaining viewpoint invariance. A structured token commitment is proposed to restrict each video token to reading from either of clsass token or register token. InvAct is trained using atomic-action supervision and evaluated on atomic and keystep retrieval, transfer to unseen datasets, and robot-policy pretraining. The paper addresses a relevant limitation of cross-view action understanding. However, I have concerns about the novelty of the contrastive objective and temporal aggregation, and whether the comparisons sufficiently distinguish the proposed contributions from the benefits of the underlying representation and training setup.

Critical strengths
  1. InvAct achieves the best average atomic retrieval results among the evaluated methods in Table 1, outperforming prior ego-video pretrained models, robotic-centric methods and domain agnostic approaches.
  2. The distinction between viewpoint invariance and scene invariance is meaningful. Matching synchronized ego–exo views can exploit shared environments, and evaluating action retrieval across scenes helps expose this limitation.
  3. In Figures 4 and 5, the proposed token re-routing shows good separation of action and background clues.
Critical weaknesses
  1. The novelty of the scene-aware contrastive objective is insufficiently established relative to EgoNCE. EgoVLP (Lin et al., NeurIPS 2022) already motivates action-aware positive sampling and scene-aware hard-negative sampling by the same two challenges: matching the same action across different scenarios and distinguishing different actions within the same scenario. InvAct only differs in using video–video alignment and explicit action/scene identities, but the conceptual overlap is substantial.

  2. The comparisons do not adequately isolate improvements over existing methods from differences in inputs, pretraining, and supervision. Table 1 compares released checkpoints with different architectures, modalities, and training data. More tellingly, the Table 13 base configuration, which already uses DINOv3 and flow but excludes scene repulsion and token commitment, achieves 30.17/29.12 cross-scene/cross-view atomic retrieval—already exceeding the corresponding averages of all existing methods in Table 1. This suggests that a substantial advantage exists before adding these two proposed components. The controlled ablations are valuable, but they do not replace comparisons against established objectives trained with the same backbone, modalities, supervision, and budget.

  3. The goal of adding flow in this work is not clear, while most prior ego-video pretrained approaches are RGB-only.

  4. L191-194: In figure 3, the video features and flow features should be extracted separately. The current figure shows video features -> tokenizer, which is quite confusing.

  5. Two minor issues: 5.1 averaging atomic embeddings for keystep retrieval provides limited methodological novelty. Aggregating short-clip representations into a longer-video representation is a longstanding approach, including segment-consensus methods such as early TSN (Wang et al., 2017). Nevertheless, average pooling itself is a conventional operation and does not model temporal order or action composition. The paper should distinguish this empirical transfer result from a new compositional mechanism and compare other atomic encoders under the same segmentation and pooling protocol.

5.2 The main unseen-dataset comparison should include Charades-Ego. Table 2 presents SSV2 and NTU, whereas Charades-Ego appears only in the appendix. Table 11 reports 16.45 for InvAct versus 17.98 for Rosetta, and nearly identical unseen-verb results, 39.85 versus 39.91. The paper acknowledges these results, but including them in the main comparison would provide a more balanced account of generalization and make the tradeoff between suppressing scene shortcuts and retaining useful contextual information more visible.

Questions for authors
  1. What's the difference between the proposed objective and EgoNCE? Refer to Q1 for details.
  2. Can you perform joint training by combining atomic action-level supervision and keystep-level supervision?
  3. In Table 12, what does InvAct + DINOv3 mean? If I understood correctly, InvAct already contains DINOv3 visual features.
4 comments
AuthorsOpenReview ↗

Reply for oaLo [1/2]

Thank you for the detailed review. We are glad you find the distinction between viewpoint and scene invariance meaningful and the separation of action and background cues in Figs 4 and 5 convincing. All additions below are in the revised PDF, with edited and new content in blue. New tables shift the appendix table numbers, so all numbers below refer to the revised PDF.


[W1, Q1] Difference from EgoNCE

"The novelty of the scene-aware contrastive objective is insufficiently established relative to EgoNCE."

Our loss builds on EgoNCE, just as EgoNCE builds on InfoNCE. Extending EgoNCE for a new goal is common. HierVL (Ashutosh et al., 2023), EgoNCE++ (Xu et al., 2025) and EgoExoNCE (Xu et al., 2024) extend it for long-term intent, hand-object interactions and ego-exo captioning, and Sec 2 notes that all three keep positives from the query's own scene. Our goal is view and scene invariance, hence we change which clips may be positives. Sec 1 and Sec 3.2 now present our loss as such an extension, with three rules (Tab 10 in App B.1).

RulePairEgoNCEOurs
1Same action, other scenepositive if sampledpositive in every batch (our sampler)
2Synchronized viewpositiveexcluded, aligned by a separate view loss
3Other clip, same action and scenepositiveexcluded

Like EgoNCE, we keep same-scene clips of other actions as negatives (scene repulsion). The new Tab 6 (Sec 4.3, bottom block) adds the rules to EgoNCE one at a time over three seeds, with our encoder, data and budget fixed.

ObjectiveCS ↑CV ↑SSF@1 ↓
EgoNCE29.88 ± 0.1229.55 ± 0.150.88 ± 0.01
+ Rule 1 (our sampler)30.42 ± 0.1129.71 ± 0.130.87 ± 0.01
+ Rule 2 (no synchronized-view positive)30.95 ± 0.1230.40 ± 0.140.70 ± 0.02
+ Rule 3 (our loss)31.57 ± 0.1031.05 ± 0.110.58 ± 0.02
+ token commitment32.30 ± 0.0731.50 ± 0.070.53 ± 0.01

Together the three rules add 1.69 / 1.50 points over EgoNCE and lower SSF@1 from 0.88 to 0.58. Our sampler alone helps little (0.54 / 0.16), because the synchronized view still dominates the positives. Removing same-scene positives is therefore what lowers scene bias, first the synchronized view (0.87 → 0.70) and then the remaining same-scene clips (0.70 → 0.58). Fig 2 shows how view alignment alone can still retrieve by scene.


[W3] Why flow

"The goal of adding flow in this work is not clear"

Flow gives the model explicit motion, while DINOv3 enters only through cross-attention (Sec 3.1). Tab 18 (App D.5) trains InvAct without flow.

InputCS ↑CV ↑SSF@1 ↓
DINOv329.5729.210.66
DINOv3 and flow32.3031.500.53

Flow adds 2.73 / 2.29 points and lowers SSF@1. Even without flow, InvAct beats every prior method on the atomic averages of Tab 1, so the gain does not come from flow alone. App D.2 also compares other input combinations (Tab 16).

AuthorsOpenReview ↗

Reply for oaLo [2/2]

[W2] Comparison with the same backbone and budget

"they do not replace comparisons against established objectives trained with the same backbone, modalities, supervision, and budget."

We compare with the backbone fixed in two settings, one CLIP-based and one DINOv3-based.

CLIP-based (Tab 4, Sec 4.3). The submission already ablated our components on a CLIP-based model that, like Viewpoint Rosetta, starts from text and view alignment. We now add SSF@1 to make the effect on scene bias clear (single run, losses in App B.2 and B.3).

Tab 4 (CLIP and flow)CS ↑CV ↑SSF@1 ↓
Text alignment25.6125.450.81
+ view invariance26.4127.160.88
+ action attraction26.9325.450.83
+ scene repulsion28.2027.140.63
+ token commitment28.8527.540.59

Aligning views raises SSF@1 from 0.81 to 0.88, and our components then add 2.44 / 0.38 points and lower it to 0.59.

DINOv3-based (new Tab 6, Sec 4.3, top block). Every row trains our encoder on frozen DINOv3 features and flow with the same clips and schedule over three seeds, and the established objectives use a tuned learning rate and temperature. The Rosetta-style row uses only Rosetta's own text and view losses.

ObjectiveCS ↑CV ↑SSF@1 ↓
Rosetta-style29.62 ± 0.1430.83 ± 0.160.90 ± 0.01
SupCon30.52 ± 0.1329.80 ± 0.100.86 ± 0.01
EgoExoNCE30.21 ± 0.1629.81 ± 0.120.89 ± 0.01
Our loss31.57 ± 0.1031.05 ± 0.110.58 ± 0.02
Our loss + token commitment32.30 ± 0.0731.50 ± 0.070.53 ± 0.01

All established objectives stay scene-dependent (SSF@1 ≥ 0.86), and our loss leads the best of them by 1.05 points across scenes (standard errors ≤ 0.16). The Rosetta-style objective stays close across views (0.22 points, within noise), since view alignment is its goal, and with token commitment the gap grows to 0.67. On the base of your Table 13 (now Tab 15), our components add 2.13 / 2.38 points over three seeds (p < 0.03) and lower SSF@1 from 0.79 to 0.53. Tab 1, in turn, compares released systems as a whole, as Viewpoint Rosetta and BYOV also do. Its caption and Sec 4.1 clarify that.


[W4] Fig 3

We agree. The Fig 3 caption now names RAFT flow and DINOv3 as the video features, and Sec 3.1 states that flow forms the video tokens while DINOv3 enters only through cross-attention. The final version will draw the two streams separately.


[5.1] Averaging for keysteps

We agree that average pooling is conventional, as in TSN. We therefore present keystep retrieval as an empirical transfer result and not as a compositional mechanism. All Tab 1 baselines use the same segmentation and pooling as InvAct, and Sec 4.2 now says so and notes that averaging ignores order (see also App G).


[5.2] Charades-Ego

Tab 2 (Sec 4.2) now includes Charades-Ego. Rosetta leads there (17.98 vs 16.45) and ties on unseen verbs (39.91 vs 39.85), likely because static scene context helps with many Charades-Ego labels. Adding a static DINOv3 feature to InvAct raises it to 19.72, above Rosetta, at a cost on Ego-Exo4D (32.16 → 29.70, Tab 13 in App C.4). Sec 4.2 now states this tradeoff.


[Q2] Joint atomic and keystep training

Yes. With keystep labels added, InvAct matches Viewpoint Rosetta on keystep retrieval across views, and its lead across scenes grows (App D.7, Tab 20).

Model and labelsAtomic CSAtomic CVKeystep CSKeystep CV
InvAct, atomic32.5331.7941.4735.09
InvAct, atomic and keystep32.4131.6242.1037.64
Viewpoint Rosetta21.6621.4339.1237.35

Atomic retrieval drops by only 0.12 / 0.17 points.


[Q3] InvAct + DINOv3

You are right that InvAct already reads DINOv3, but only through cross-attention. That row, now "InvAct + static DINOv3 feature" in Tab 13, appends the clip's mean-pooled DINOv3 feature to the embedding. App C.4 provides further explanation.


Thank you again for the concrete questions, which made the paper clearer. We hope the controlled comparisons address your concerns, and we would be glad if you would consider raising your score.

Dear Reviewer,

Thank you again for your valuable feedback, recommendations, and corrections. Since we are approaching the end of the discussion session, we would sincerely appreciate your thoughts on our responses and are happy to address any remaining questions.

Thank you for your time and consideration.

Dear Reviewer,

Since we are nearly ending the discussion phase, we would really like to hear your response on the new experiments and clarifications, and whether they do or do not address your concerns. Thank you for your consideration.

Reviewer_MRW4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies scene bias in view-invariant atomic action representation learning. The authors propose InvAct, which combines a scene-aware contrastive objective with structured token commitment to encourage action-centric representations. Experiments on Ego-Exo4D demonstrate improvements in cross-scene and cross-view action retrieval, with additional evaluations on longer actions, unseen datasets, and robot-policy pretraining.

Critical strengths
  1. The paper addresses a meaningful limitation of existing ego–exo representation learning methods: viewpoint alignment does not necessarily prevent reliance on scene-specific cues. The proposed approach is intuitive and well motivated.
  2. InvAct achieves strong atomic action retrieval performance across viewpoints and scenes. The controlled multi-seed ablations support the effectiveness of the proposed components, while the additional transfer experiments demonstrate their potential beyond the primary benchmark.
Critical weaknesses
  1. Insufficient evidence for scene bias reduction. While InvAct achieves lower SSF than existing methods, it remains unclear whether this reduction is specifically caused by the proposed components, since the controlled ablations report retrieval accuracy but not SSF. Furthermore, the paper uses take IDs as scene labels, although different takes may share the same physical environment. Consequently, the current evaluation establishes reduced same-take preference more directly than invariance to distinct visual scenes.
  2. Limited attribution of the performance gains. The substantial improvements over existing methods in Table 1 may partly stem from stronger pretrained features and explicit Ego-Exo4D atomic-action supervision, which the released baseline checkpoints do not share. Although Table 13 demonstrates the effectiveness of the proposed components, the Base model already achieves strong performance without them. More comparable training conditions would help clarify the contribution of the proposed method.
Questions for authors
  1. Could the authors report SSF for the Base, Base + Scene Repulsion, and full InvAct configurations in Table 13 to directly verify whether the proposed components reduce scene bias?
  2. Could the authors clarify the cross-scene retrieval protocol and evaluate scene dependence using physical-setting identifiers rather than take IDs?
  3. Could the authors provide a representative baseline trained with comparable atomic-action supervision and pretrained features to better isolate the contribution of the proposed method?
4 comments
AuthorsOpenReview ↗

Reply for MRW4

Thank you for the positive and constructive review. We are glad you find the approach intuitive and well motivated and the multi-seed ablations convincing. All additions below are in the revised PDF, with edited and new content in blue. New tables shift the appendix table numbers, so all numbers below refer to the revised PDF.


[W1, Q1] SSF for the controlled configurations

"Could the authors report SSF for the Base, Base + Scene Repulsion, and full InvAct configurations in Table 13"

Your Table 13 is now Tab 15 (App D.1), and it reports SSF@k over the same three seeds.

ConfigurationSSF@1 ↓SSF@5 ↓SSF@10 ↓SSF@25 ↓
Base0.79 ± 0.010.950.970.98
+ scene repulsion0.58 ± 0.020.880.930.95
+ token commitment0.53 ± 0.010.870.920.94

The encoder is the same in every row, hence the drop comes from our components. Without them, the encoder is more scene-dependent than LAPA (0.60), the lowest prior method in Tab 3, and the longer-trained final model reaches 0.52. Most of the drop comes from scene repulsion, while token commitment mainly makes CLS react less to the background, by 21% (App D.6, Tab 19).


[W1, Q2] Retrieval protocol and physical settings

"Could the authors clarify the cross-scene retrieval protocol and evaluate scene dependence using physical-setting identifiers rather than take IDs?"

Protocol (App A.1). Cross-scene retrieval ranks all clips of the query's view except the query, including clips from its own take. Cross-view retrieval ranks all clips of the other view except the synchronized one. A query is a hit when one of its top-10 clips shares its verb.

Physical settings (App C.5). Takes recorded at the same site and in the same room now form one physical setting. Tab 14 reports SSF@1 by setting, and cross-scene hit@10 with the query's whole setting removed from the gallery.

MethodCS, full galleryCS, setting removedSSF@1 by takeSSF@1 by setting
SUM-L26.4622.310.940.96
LAPA25.8823.100.600.68
Rosetta21.6617.200.990.99
InvAct32.5331.020.520.61

InvAct stays the least scene-dependent by setting, and it loses the least when its setting is removed (1.51 points, against 2.78 for LAPA and 4.46 for Rosetta). With the encoder fixed, our components lower SSF@1 by setting from 0.86 to 0.67 and 0.63. Even with the query's whole category removed, InvAct finds the query verb in its top 5 for 17.1% of queries, against 10.5% for Rosetta (App C.5, Fig 11).


[W2, Q3] A baseline with comparable supervision and pretrained features

"Could the authors provide a representative baseline trained with comparable atomic-action supervision and pretrained features"

We added established objectives trained with InvAct's own encoder (new Tab 6, Sec 4.3). Every row uses frozen DINOv3 features and flow with the same clips and schedule over three seeds, and the objective is what changes. EgoNCE keeps its original random batches, and the Rosetta-style row uses only Rosetta's own text and view losses. The bottom block adds our three rules to EgoNCE one at a time, and App B.1 (Tab 10) explains every row.

Objective (mean ± standard error)CS ↑CV ↑SSF@1 ↓
Rosetta-style29.62 ± 0.1430.83 ± 0.160.90 ± 0.01
SupCon30.52 ± 0.1329.80 ± 0.100.86 ± 0.01
EgoExoNCE30.21 ± 0.1629.81 ± 0.120.89 ± 0.01
EgoNCE29.88 ± 0.1229.55 ± 0.150.88 ± 0.01
+ Rule 1 (our sampler)30.42 ± 0.1129.71 ± 0.130.87 ± 0.01
+ Rule 2 (no synchronized-view positive)30.95 ± 0.1230.40 ± 0.140.70 ± 0.02
+ Rule 3 (our loss)31.57 ± 0.1031.05 ± 0.110.58 ± 0.02
+ token commitment32.30 ± 0.0731.50 ± 0.070.53 ± 0.01

With comparable supervision and features, all established objectives stay scene-dependent (SSF@1 ≥ 0.86). Our loss leads the best of them by 1.05 points across scenes, and by 1.78 with token commitment, while standard errors are at most 0.16. The Rosetta-style objective stays close across views, since view alignment is its goal, but its SSF@1 stays at 0.90. The same holds with CLIP features, which Rosetta also uses. In Tab 4 (single run), our components add 2.44 / 0.38 points over text and view alignment and lower SSF@1 from 0.88 to 0.59. Tab 1, in turn, compares released systems as a whole, as Viewpoint Rosetta and BYOV also do. Its caption and Sec 4.1 clarify that.


Thank you again for the concrete questions, which made the paper clearer. We would be glad if you would consider raising your score.

Dear Reviewer,

Thank you again for your valuable feedback, recommendations, and corrections. Since we are approaching the end of the discussion session, we would sincerely appreciate your thoughts on our responses and are happy to address any remaining questions.

Thank you for your time and consideration.

Thanks for the responses and additional experiments. Most of my concerns are addressed with detailed experiments. I will therefore maintain my positive rating.

Dear Reviewer,

Thank you for participating in the discussion phase and suggesting new experiments to make the submission stronger. We are glad that the concerns are resolved and the rating is positive.

Reviewer_HPQHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors addresses a meaningful weakness for cross-view alignment problem, which it’s easier for the model to match a shared environment than simply an action. In specific, can action representations get the semantic view or rely on the scene view between egocentric and exocentric. The previous method can successfully align different camera viewpoints but mainly rely on background or other external information rather than focus on actions.

They introduce InvAct, combining a contrastive objective with structured token, a parameter-free attention mask routing each video token to one special token in [CLS,REG], to encourages the model to match the same actions across different scenes while separating different action within the same scene. By evaluating on Ego-Exo4D, InvAct improves atomic action retrieval performance by 6.07 and also 5.74 compared with previously baseline on cross-scene and cross-view. In addition to that, they also show the performance can be transfer to keystep actions without further training. They also conduct zero-shot experiment on SSV2 and NTU by using InvAct’s embedding on LIBERO-10, which shows improvement on mean success percentage.

Critical strengths

• InvAct combines RoPE+RAFT+Dinov3 to output a CLS embedding, which its main contributions are a scene-focused contrastive objective and a structured token commitment. They restrict the video token to read from the specific token they created. The problem is clear that the motivation for solving such question is needed. More importantly, the atomic-retreival result proofs that their InvAct gain sufficient improvement over the previously baselines and the keystep result shows their checkpoint reaches the best cross-scene average.

• The paper conducts extensive experiments on other tasks to show its improvement. Other than atomic embeddings, they also show their improvement on unseen datasets, including SSV2 and NTU, and LIBERO-10 so it’s effectively speaking that the learned representation can also be useful for other downstream tasks.

Critical weaknesses

• The routing process improves the retrieval performance. However, the token re-routing process seems not to separate the information clear enough. “With Eq. (3), each video token must pick a single read source from [CLS,REG] at every layer. Without this restriction, non-action tokens pull in a fraction of CLS’s action-aligned content each layer and start to look action-relevant even when they carry no real signal. Forcing one read source blocks this contamination” The author claims that the video token must pick a read source from the special key source at very layer. Additionaly, “each video token reads from exactly one special token,” for the attention are not always true since there are no design for blocking CLS read every video token. As a result, assigning a single video token to RED doesn’t conclude that all the video tokens are inaccessible to CLS, so the inherent communication might still possible when doing cross-attention.

• The comparison with pretrained foundation models in Table 1 is not fully controlled. According to the author’s Appendix A.5, models used as comparison like CLIP, DINOv3, V-JEPA 2, and other foundation models etc, are simply using their released pretrained checkpoints without extra finetuning, meanwhile InvAct receives additional supervision on Ego-Exo4D, which it’s clear that InvAct also use pretrained DINOv3 features as part of its pipeline. It’s hard to separate the result that the improvements over those baselines might rely on task-specific training rather than those contributions author made.

• There are some minor issues for Table 1 that the calculation for EgoVLPv2 atomic cross-view results seems to has some errors, The average shown is 23.59 while it’s larger than both g->x and x->g.

Questions for authors

• Can the author try to test by doing more ablation such like only changing the background information while keep the action the same then measure the result CLS-embedding change with or without token commitment to clarify the improvement?

• Can the authors provide a supervised adaptation baseline using those visual foundation models like DINOv3) with the exact same Ego-Exo4D atomic-action training data and training budget to help clarify whether InvAct’s improvements is not simply from its backbone?

3 comments
AuthorsOpenReview ↗

Reply for HPQH

Thank you for the thoughtful review. We are glad you find the motivation clear and the transfer to SSV2, NTU and LIBERO-10 useful. All additions below are in the revised PDF, with edited and new content in blue. New tables shift the appendix table numbers, so all numbers below refer to the revised PDF.


[W1] Does token re-routing separate the information?

"the token re-routing process seems not to separate the information clear enough"

You are right that the mask does not hide REG-routed tokens from CLS. It only limits what each video token reads (Eq. 3). The submission overstated this, and Sec 3.3 now states "Forcing one read source removes this direct path, although CLS can still read every video token." The mask removes the path where a non-action token reads CLS at every layer and starts to look action-relevant. We therefore measured whether CLS still uses these tokens, with the models of Tab 15 (new App D.6, Tab 19).

Last layer, 115 clipsWithout commitmentWith commitment
Video tokens routed to REG (%)6471
CLS attention on these tokens (%) ↓415

With commitment, 71% of the tokens are routed to REG, yet they get only 5% of the CLS attention. Hence CLS can read them but almost ignores them. Cross-attention, the path you mention, is not masked. In the final model (Fig 12), the actor covers only 3% of ego and 11% of exo frames, yet it gets 16 to 32% and 42 to 58% of the CLS cross-attention across layers. Since the actor is small in ego frames, most ego cross-attention still lands outside it, which fits the reduced but non-zero scene bias in Tab 12. A column-wise mask that hides REG-routed tokens from CLS performs much worse (24.14 / 23.06 against 32.28 / 31.50, Tab 17, App D.3).


[Q1] Changing only the background

"only changing the background information while keep the action the same then measure the result CLS-embedding change with or without token commitment"

We ran this test (App D.6, Tab 19, Fig 13). We paste each clip's actor onto an inpainted, actor-free take from another category, and we recompute flow and DINOv3 features.

Cosine distance of CLS embeddings, 93 clipsWithout commitmentWith commitment
Original to composite (± 95% bootstrap interval) ↓0.29 ± 0.020.23 ± 0.02
Reference, clips with different verbs0.800.81

With commitment, CLS moves 21% less when only the background changes, while clips with different verbs stay equally far apart. Thus commitment does not simply shrink all distances. SSF@1 falls less, from 0.58 to 0.53 (Tab 15), because clips of one take also share the person and objects, and this test keeps them.


[W2, Q2] A supervised adaptation baseline with DINOv3

"Can the authors provide a supervised adaptation baseline using those visual foundation models like DINOv3) with the exact same Ego-Exo4D atomic-action training data and training budget"

We added such baselines (new Tab 6, Sec 4.3). Each one keeps InvAct's frozen DINOv3 features, flow input, model, 30,615 clips and schedule over three seeds, and the objective is what changes. EgoNCE keeps its original random batches, and the Rosetta-style row uses only Rosetta's own text and view losses. SupCon is the most direct baseline, as it pulls together every pair of same-verb clips. App B.1 describes each objective.

Mean ± standard errorCS ↑CV ↑SSF@1 ↓
Rosetta-style29.62 ± 0.1430.83 ± 0.160.90 ± 0.01
SupCon30.52 ± 0.1329.80 ± 0.100.86 ± 0.01
EgoExoNCE30.21 ± 0.1629.81 ± 0.120.89 ± 0.01
EgoNCE29.88 ± 0.1229.55 ± 0.150.88 ± 0.01
Our loss31.57 ± 0.1031.05 ± 0.110.58 ± 0.02
Our loss + token commitment32.30 ± 0.0731.50 ± 0.070.53 ± 0.01

The supervised baselines remain scene-dependent (SSF@1 ≥ 0.86). Over SupCon, our loss adds 1.05 / 1.25 points and cuts SSF@1 from 0.86 to 0.58, and token commitment adds another 0.73 / 0.45. The Rosetta-style objective stays close across views (0.22 points, within noise), since view alignment is its goal, but its SSF@1 stays at 0.90. The gains also hold with CLIP features. With CLIP and our flow input (Tab 4, single run), our components add 2.44 / 0.38 points and lower SSF@1 from 0.88 to 0.59. Tab 1, in turn, compares released systems as a whole, as Viewpoint Rosetta and BYOV also do. Its caption and Sec 4.1 clarify that.


[W3] EgoVLPv2 average

You are right. The correct cross-view average is 22.71, the mean of 21.88 and 23.54, which moves EgoVLPv2 from fourth to seventh in that column. We also fixed the BYOV keystep cross-view average (20.77), two averages in Tab 16 and three best-result marks in Tabs 1 and 2. None of these changes any conclusion.


Thank you again. We hope these results address your concerns, and we would be glad if you would consider raising your score.

Dear Reviewer,

Thank you again for your valuable feedback, recommendations, and corrections. Since we are approaching the end of the discussion session, we would sincerely appreciate your thoughts on our responses and are happy to address any remaining questions.

Thank you for your time and consideration.

Dear Reviewer,

Since we are nearly ending the discussion phase, we would really like to hear your response on the new experiments and clarifications, and whether they do or do not address your concerns. Thank you for your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment about changes and new experiments

We thank all reviewers and the AC for their careful reading. We uploaded a revised PDF with edited and new content in blue. New tables shift the appendix table numbers, so all numbers below refer to the revised PDF. Details are in our reply to each reviewer.


[Our contribution] View-invariant methods match synchronized ego and exo clips. These also share the scene, so a model can match them by the scene alone. This scene overfitting has been overlooked.

  1. We identify and measure the overfitting. The same-scene fraction SSF@k counts how often retrieval returns a clip from the query's own scene. The view-invariant baselines of Tab 3 do so at top-1 for 94 to 99% of queries, against 52% for InvAct (Tab 3, Sec 4.3).
  2. We adapt InfoNCE and EgoNCE with three important changes, so that matching the scene no longer pays off. EgoNCE counts the synchronized view and other same-scene clips as positives, so the cheapest way to lower its loss is to match the scene. Our three rules remove these positives and put a cross-scene positive in every batch (Sec 3.2, Tab 10 in App B.1). With everything else fixed, this lowers SSF@1 from 0.88 to 0.58 (Tab 6, Sec 4.3).
  3. Structured token commitment is a parameter-free attention mask. It keeps non-action video tokens from absorbing the class token's content, so the class token relies less on the scene (Sec 3.3).

[What the new experiments show]

  • With our encoder, data and budget fixed, established objectives stay scene-dependent. On top of EgoNCE, each of our additions (rules) helps. All rows are means over three seeds (new Tab 6 in Sec 4.3), and App B.1 explains every row.
Objective (Tab 6)CS ↑CV ↑SSF@1 ↓
Rosetta-style29.62 ± 0.1430.83 ± 0.160.90 ± 0.01
Supervised contrastive (SupCon)30.52 ± 0.1329.80 ± 0.100.86 ± 0.01
EgoExoNCE30.21 ± 0.1629.81 ± 0.120.89 ± 0.01
EgoNCE29.88 ± 0.1229.55 ± 0.150.88 ± 0.01
+ Rule 1 (our sampler)30.42 ± 0.1129.71 ± 0.130.87 ± 0.01
+ Rule 2 (no synchronized-view positive)30.95 ± 0.1230.40 ± 0.140.70 ± 0.02
+ Rule 3 (= our loss)31.57 ± 0.1031.05 ± 0.110.58 ± 0.02
+ token commitment32.30 ± 0.0731.50 ± 0.070.53 ± 0.01
  • The components reduce scene bias in the controlled runs. Scene repulsion and token commitment lower SSF@1 from 0.79 to 0.58 and 0.53 (Tab 15, App D.1). InvAct stays the least scene-dependent when scenes are grouped by physical setting (Tab 14, App C.5).
  • Token commitment changes what CLS uses. Tokens routed to REG get 5% of last-layer CLS attention with commitment against 41% without, and CLS moves 21% less when only the background changes (App D.6, Tab 19).
  • The gains do not depend on DINOv3. With CLIP features and our flow input, our components add 2.44 / 0.38 points in a single run and lower SSF@1 from 0.88 to 0.59 (Tab 4, Sec 4.3).
  • Flow helps but is not required. It adds 2.73 / 2.29 points, and InvAct with DINOv3 but no flow still beats every prior method on the atomic averages of Tab 1 (Tab 18, App D.5).
  • With keystep labels added, InvAct also matches Viewpoint Rosetta on keystep retrieval across views, and its lead across scenes grows (Tab 20, App D.7).

[Clarifications to the PDF]

  • Tab 1 is a system-level comparison. Its caption and Sec 4.1 now say that it compares released checkpoints as a whole, as Viewpoint Rosetta and BYOV also do. Such a table shows which available representation best serves the task.
  • Tabs 4, 6 and 15 are the component-level ablations (Sec 4.3 and App D.1). Each keeps its encoder, data and budget fixed. Tab 6 is new, Tab 15 now reports SSF@k over three seeds, and Tab 4 now reports SSF@1.
  • EgoNCE comparison. Sec 3.2 now states how our loss extends EgoNCE with three rules, and Tab 6 adds them to EgoNCE one at a time. Our contributions (Sec 1) and Sec 2 now credit EgoNCE and its extensions.
  • Sec 3.3 now states that CLS can still read every video token, and reports how little attention it gives to tokens routed to REG.
  • Sec 4.2 adds Charades-Ego to Tab 2 and states that all methods average the same atomic segments for keystep retrieval.
  • Appendix additions
    • B.1, the objectives of Tab 6 and the pairs each one forms (Tab 10)
    • C.5, physical settings and setting-removed retrieval (Tab 14, Fig 11)
    • D.5, training without flow (Tab 18)
    • D.6, what reaches the class token (Tab 19, Figs 12 and 13)
    • D.7, joint keystep training (Tab 20)
  • Small fixes to a few averages and best-result marks in Tabs 1, 2 and 16 (see our reply to HPQH).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.