- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Q-ViSTA, a framework for Generalized Video Skill Discovery (GVSD) that jointly models semantic identity and localized transition dynamics. The method employs a frozen VideoMAEv2 backbone and introduces two complementary representation branches. The semantic branch captures global appearance and contextual information, while the dynamic ator to modbranch uses recurrent Action Queries to extract localized trajectories and a multiscale transition operel temporal changes. These representations are integrated through a nonlinear fusion module and optimized using several representation-learning objectives. The framework supports both non-parametric clustering and parametric classification. Experiments are conducted on six protocols covering human actions and robot manipulation skills. The results show improvements over several GCD baselines, particularly on the RoboCasa365 Atomic-to-Composite protocol, suggesting the potential benefits of combining semantic context with localized temporal information for video category discovery.
Critical strengths
This paper investigates the joint modeling of semantic information and localized temporal dynamics for Generalized Video Skill Discovery, addressing a meaningful research problem. The authors propose a dual-branch framework that integrates global semantic representations with localized transition modeling. The overall design is reasonable, as it considers both contextual information and temporal interactions in videos. Furthermore, the proposed method is evaluated on six protocols covering human actions and robotic manipulation skills, achieving performance improvements over existing baselines. In particular, the results on the Atomic-to-Composite task are encouraging, demonstrating the potential effectiveness and applicability of the proposed framework.
Critical weaknesses
The main limitation of this paper lies in its relatively limited methodological novelty, as its core components largely rely on existing attention mechanisms, temporal modeling techniques, and feature fusion strategies, without introducing a sufficiently distinctive technical contribution. Moreover, the experimental results do not adequately demonstrate the necessity of several key components. For example, Table 4 shows that multiscale transition modeling provides only marginal improvements on several datasets and even decreases novel-class accuracy in some cases, raising questions about the effectiveness of this relatively complex design. In addition, the ability of Action Queries to capture interaction-relevant regions is mainly supported by qualitative visualizations, without sufficient quantitative validation. Regarding the Atomic-to-Composite task, the current experiments do not adequately isolate the effects of object appearance and scene context, making it difficult to determine whether the observed improvements genuinely reflect compositional generalization. Overall, although the proposed method achieves encouraging results, the experimental evidence remains insufficient to convincingly establish its methodological novelty and the effectiveness of its key components.
Questions for authors
-
How can the authors demonstrate that the proposed framework genuinely captures temporal dynamics rather than relying primarily on appearance and contextual information? I suggest conducting additional experiments involving temporal shuffling or frame reversal to verify whether the model truly depends on temporal transitions for category discovery.
-
The results in Table 4 show that multiscale transition modeling provides only marginal improvements on several datasets, while Table 13 indicates that the shortest temporal scale dominates the learned mixture weights. Could the authors further explain the necessity of the multiscale design and provide stronger experimental evidence to justify its additional complexity?
-
The effectiveness of Action Queries is mainly demonstrated through qualitative attention visualizations, which are insufficient to establish their ability to capture meaningful interaction regions. Could the authors provide quantitative evaluations of localization accuracy and temporal consistency to demonstrate that Action Queries genuinely track action-related regions rather than simply selecting discriminative appearance features?
-
The Atomic-to-Composite experiment is presented as important evidence of the model's compositional generalization ability. However, the current experimental protocol does not adequately control variations in object appearance and scene context. How can the authors ensure that the reported improvements result from learning novel combinations of atomic skills rather than exploiting semantic or visual differences between categories?
-
Although the proposed method achieves improvements over existing baselines, it introduces additional trainable modules and optimization objectives. Could the authors provide more controlled comparisons with similar model capacity and computational budgets? Without such comparisons, it remains unclear whether the improvements come from the proposed query-localized transition modeling or simply from increased architectural complexity.