- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ActionRefiner, a plug-in framework for improving egocentric action recognition, particularly for actions involving inverse verbs that share the same noun but differ in temporal dynamics. The method introduces a Causal Temporal Generation Process (CTGP) to learn verb- and noun-specific latent temporal representations using VAE-based models conditioned on action-level context. A residual classification module then generates corrective logits, incorporating posterior averaging, negative verb–noun swapping, and top-k masked fusion to refine predictions from a frozen action recognition baseline. Experiments on EPIC-Kitchens-100 and EGTEA Gaze+ demonstrate improvements across multiple baselines, including LaViLa, AVION, TIM, and MTCN. The method achieves 57.7% and 82.7% Top-1 accuracy on the two datasets, respectively. Additional ablation studies and inverse-verb analyses are presented to support the effectiveness of the proposed components.
Critical strengths
-
Consistent improvements across multiple baselines and datasets. ActionRefiner demonstrates consistent performance gains when integrated with different action recognition models, including TIM, AVION, LaViLa, and MTCN. For example, it improves Top-1 accuracy by 1.3% over TIM and 2.5% over LaViLa on EPIC-Kitchens-100. These results support the practical effectiveness and general applicability of the proposed refinement framework.
-
Well-motivated problem formulation with targeted empirical validation. The paper addresses the challenging problem of distinguishing actions involving inverse verbs, which often share similar visual appearances but differ in temporal dynamics. The analysis on EGTEA Gaze+ shows larger improvements on inverse-verb actions than on other actions, providing empirical support for the proposed motivation. This task-specific analysis strengthens the practical significance of the work beyond overall accuracy improvements.
Critical weaknesses
-
Insufficient justification for the causal interpretation of CTGP. The paper presents Causal Temporal Generation Process (CTGP) as a central contribution. However, the proposed transition distributions primarily characterize action-conditioned temporal dependencies, and it remains unclear what distinguishes them from conventional conditional temporal modeling. The paper does not sufficiently establish the causal interpretation or identifiability of the learned latent representations. Although comparisons with Mamba and vanilla VAE demonstrate performance improvements, they do not necessarily validate the claimed causal mechanism. This weakens the theoretical foundation and novelty of the paper's main contribution.
-
Insufficient experimental evidence to isolate the contribution of CTGP. ActionRefiner introduces additional verb/noun representations extracted from powerful visual encoders, including V-JEPA 2.1 Gigantic (2B). While the reported improvements are encouraging, the existing ablations do not fully establish whether these gains primarily arise from the proposed CTGP or from additional visual representations, supervision, and model capacity. More controlled comparisons with simpler action-conditioned temporal models using identical features and training objectives would help establish the necessity of the proposed framework. This limits the strength of the empirical evidence supporting its methodological contribution.
Questions for authors
-
Clarification of the causal interpretation. What specifically makes the proposed CTGP a causal temporal generation process rather than a conventional action-conditioned temporal model? Could the authors clarify the causal assumptions underlying the transition distributions and whether any identifiability guarantees or empirical evidence support the causal interpretation of the learned latent variables? This distinction is important for assessing the theoretical contribution and novelty of the method.
-
Isolating the contribution of CTGP. While the comparisons with Mamba and vanilla VAE are useful, they do not fully isolate the contribution of the proposed temporal mechanism. Could the authors provide a comparison with a simpler action-conditioned temporal model (e.g., a conditional GRU, Transformer, or VAE) using the same verb/noun features, supervision, and comparable model capacity? Such a comparison would help determine whether the improvements arise specifically from CTGP or from additional conditioning and representation learning.
-
Statistical significance of inverse-verb improvements. In Table 5, the improvements on inverse-verb actions are relatively modest compared to those on other actions (e.g., +1.1% versus +0.8% for LaViLa). Could the authors report the sample sizes of these subsets and provide uncertainty estimates or statistical significance analyses? This would help establish whether the proposed method specifically addresses inverse-verb confusion, rather than providing a general improvement in action recognition.
Reply
We thank the reviewer for suggestions and comments.
Causal interpretation. Conventional conditional temporal models learn one unconstrained joint mapping to produce the whole vector zt from its history and context, there is no structure about how individual components of zt relate to others. CTGP employs a factorized component-wise transition. Each coordinate of zt is generated from the noise term, lagged history and action context, this is the content in nonlinear ICA in causal literature. The underlying assumptions are that verb and noun latent variables are influenced by time-delayed history, context and exogenous action features.
The identifiability theorems in Yao et al. (2022) are proven under the assumption that the domain variable is given and exogenous, i.e. a known, discrete domain/regime index. However, there are no such indices in action recognition task/dataset, this is the reason we use a continuous GRU. The assumption in Yao et al. (2022) does not hold once domain vector is replaced by a value estimated by a network, since it is no longer independent of the estimation process the theorem is meant to prove. Therefore the identifiability experiments in Yao et al. (2022) (e.g., Mean Correlation Coefficient) is not feasible on egocentric video. Li et al. (2024a) also use a continuous GRU to estimate domain variables with the assumption that they can be learned empirically and do not provide theoretical proof on identifiability. CTGP essentially is inline with this assumption and we do not claim identifiability either. We will revise the manuscript to state this directly, i.e. ActionRefiner uses transition-prior VAE as a structured conditioning mechanism, not as a source of identifiability guarantees.
Isolating CTGP contribution. We additionally test GRU and transformer to compare with CTGP. We replace CTGP with GRU or transformer and keep the rest components and settings the same as ActionRefiner. Concretely, verb feature from visual encoder are concatenated with action feature and fed to verb GRU/Transformer. The same concatenation is applied to noun GRU/Transformer. The model sizes are on the same level as CTGP.
Both GRU and transformer are on par with the base model LaViLa. The results confirm the contribution of CTGP and show the necessity of CTGP.
Statistical test. The total test set has 2022 videos, the inverse verb subset has 1381 videos and the reset subset has 641 videos. We run McNemar’s test on both subsets with LaViLa.
For the inverse verb subset, p=0.078 and the rest subset has p=0.18. On inverse verb subset it is approaching significance while not significant on the rest set.