acceptodds
Under review as a conference paper at ICLR 2027

EgoExo-ViewCL: View-Labeled Continual Action Learning with Stable Analytic Anchors and View-Factorized Adaptation

Abstract

Continual video learning is usually evaluated as a sequence of new action classes, yet an embodied learner also moves between first- and third-person views. A model may preserve action semantics within an observed view yet lose cross-view transfer, or adapt by overwriting useful pretrained structure. We introduce EgoExo-ViewCL, a view-labeled continual action benchmark built from EgoMe, Assembly101, and Charades-Ego. Its three protocols isolate view shift under a fixed label space, joint class-and-view expansion, and sequential paired-view arrival. The benchmark preserves native single- and multi-label targets and evaluates both final recognition and continual dynamics, including view-conditioned interference, view-order robustness, and paired-view consistency. We further propose VFCT, which combines an order-stable analytic anchor, bounded semantic adaptation, and cumulative class–view prototype score mixing. Controlled evaluation keeps the observation set fixed when measuring view order, averages five class orders, and crosses paired-view arrival direction with true and pseudo pairing. Across the three datasets, VFCT consistently improves recognition under view switches, joint class–view expansion, and paired-view transitions. Continual and protocol-specific diagnostics show that retaining shared action evidence and adapting view-dependent cues are complementary throughout the stream. These results establish viewpoint order, pairing, and their interaction with class arrival as important axes for continual action learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Video Continual Learning, Egocentric--Exocentric Video Understanding, Viewpoint-Aware Learning, Representation Factorization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Um7sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a continual action learning benchmark, which is named EgoExo-ViewCL. The benchmark treats ego-exo viewpoint as an explicit dimension of continual change. The paper also proposes VFCT, which combines an order-stable analytic anchor, bounded semantic adaptation, and class-view prototype memory to balance stability and view-dependent adaptation. Experiments show that VFCT achieves the best reported performance.

Critical strengths
  • The paper explicitly studies the ego-exo viewpoint in continual learning,
  • The paper proposes three comprehensive protocols that cover view-order shift, joint class-view expansion, and sequential cross-view arrival.
  • Experiments show that VFCT consistently outperforms the compared baselines across the three datasets.
Critical weaknesses
  • The task construction may be overly artificial. Each task contains only one viewpoint, so viewpoint changes are strongly aligned with task boundaries. For example, in Protocol B, class arrival and viewpoint change are also entangled, making it difficult to isolate the source of forgetting.

  • It is unclear why the method should preserve view-specific differences instead of learning view-invariant representations. VFCT explicitly keeps view-specific residuals and class-view prototypes, but the paper does not sufficiently justify why retaining these differences is preferable to aligning ego/exo features.

  • The method largely combines an analytic continual learner, bounded residual adaptation, and prototype memory. Most of the ablation gain also comes from the shared analytic anchor, while the explicitly view-specific modules provide relatively modest improvements.

  • The current ablation does not analyze module interactions.

  • The appendix defines Forgetting and BWT, but I could not find explicit numerical results for these metrics.

  • The method relies on view labels at inference. VFCT uses the ego or exo label to select the view-specific residual head, but the paper does not analyze robustness when view labels are unavailable, predicted, or noisy.

  • Protocol C does not clearly demonstrate the importance of true pairing. True and pseudo pairing yield similar results, while arrival direction appears to have a larger effect. This weakens the motivation for emphasizing paired-view correspondence.

  • Important hyperparameters lack sensitivity analysis.

  • The paper analyzes persistent-state memory, but computational efficiency is less clear.

Questions for authors

See above.

Reviewer_2eEQOpenReview ↗
Rating
3
Confidence
4(+2)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual action recognition across ego and exo views. It introduces EgoExo-ViewCL with protocols for view-order learning, joint class–view incremental learning, and sequential paired-view arrival. VFCT combines analytic classifiers, bounded semantic adaptation, and class–view prototype memory on frozen video features. Evaluation on EgoMe, Assembly101, and Charades-Ego reports consistent gains over nine baselines.

Critical strengths
  1. Treating viewpoint as an explicit factor in continual learning helps study cross-view forgetting that standard class-incremental evaluation does not isolate;
  2. Protocol A compares task schedules using the same observation–label–view multiset, providing a well-controlled view-order evaluation;
  3. The experiments cover three datasets, nine baselines, and both single- and multi-label action recognition. VFCT achieves the best main-table result in all nine dataset–protocol combinations, showing consistent performance in the evaluated settings.
Critical weaknesses
  1. Protocol B introduces new classes and viewpoints together, with each class block assigned to one view. Test-time view labels also provide class priors. The results do not separate class-incremental forgetting from view interference. It is therefore unclear how much of VFCT’s gain comes from handling view shifts rather than ordinary class-incremental learning;
  2. In Figure 4, true and pseudo pairing give similar results, while arrival order has a clearer effect. The results do not show a clear benefit from instance-level pairing for cross-view transfer. Pseudo pairing also changes task composition, so it does not isolate the effect of correspondence. It remains unclear whether the paired-view transfer instability in C is distinct from the view interference already studied in A;
  3. The semantic adapter is shared across views, and its losses do not separate action semantics from view-specific features. Bounding the residual limits the size of the feature update, but does not make the update view-specific. It is unclear why this module should preserve shared action semantics while adapting only view-dependent information;
  4. The prototype branch combines ego and exo cosine scores using support-based weights, without conditioning on the query’s view. When the two view prototypes are poorly aligned, the other-view prototype can lower the correct-class score and cause negative transfer. Separate prototype storage therefore does not itself prevent cross-view interference during fusion. The fusion rule needs a clearer justification when the view gap is large.
Questions for authors
  1. How does Protocol B distinguish class-incremental forgetting from view interference? How much do the class priors supplied by test-time view labels affect the results?
  2. Does training in Protocol C use instance correspondence, for example through paired replay? Given the similar true/pseudo results, what evidence supports a paired-view transfer effect beyond ordinary view interference?
  3. Which part of the adapter objective makes the update view-specific while preserving shared action semantics?
  4. Why does the prototype branch use fixed-weight voting across both views when the query’s view is known? How is negative transfer avoided when the view prototypes are poorly aligned?
Reviewer_2dCMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses continual action learning under viewpoint shifts between egocentric and exocentric video perspectives. It introduces a view-labeled continual action learning benchmark EGOExo-VIEWC from using 50-class subsets of EgoMe, Assembly101, and Charades-Ego. It defines three protocols: view-order continual learning, joint class-and-view incremental learning, and sequential paired-view continual learning. The proposed VFCT method combines an order-stable analytic ridge anchor with view-specific score residuals, a bounded semantic adapter, and class-view prototype memory. Experiments compare against ten baselines and report strong gains across nine dataset-protocol cells, plus diagnostics on view-order robustness, stability-plasticity, and paired-view consistency.

Critical strengths
  1. This paper introduces a relevant and well-motivated continual-learning benchmark that explicitly controls view order, class-view expansion, and paired-view arrival.
  2. The empirical evaluation is broad, including multiple orders, arrival directions, true/pseudo pairing controls, view-resolved results, and diagnostics beyond final accuracy.
Critical weaknesses
  1. A major concern is that the main results may benefit from additional method state (analytic statistics and prototypes) that is not matched across baselines. Appendix D shows VFCT loses to RanPAC at 16 MiB, so the advantage is budget-dependent.
  2. Full component analysis is only shown for Assembly101; a simple view-conditioned RanPAC or per-view analytic baseline is missing, it is not very clear whether the gain comes from view factorization or extra capacity.
Questions for authors
  1. Can you provide matched persistent-state comparisons for all datasets and all three protocols, not only Assembly101 A/C?
  2. How does VFCT compare against a simple view-conditioned RanPAC baseline or separate ego/exo ridge heads with shared fallback?
  3. Can you provide full ablations across all datasets and protocols, including the view-residual and memory components?
  4. How sensitive are results to having view labels at inference, and how much would performance drop with noisy or unavailable test-time view labels?
  5. Will you release code, class lists, split files, protocol builders, and evaluation scripts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.