acceptodds
Withdrawn from ICLR 2027

Q-ViSTA: Semantic Identity and Query-Localized Transition Dynamics for Generalized Video Skill Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generalized category discovery (GCD) enables open-world systems to organize known and novel categories using partial supervision. Extending this capability from images to videos requires preserving not only semantic identity, but also the localized interactions that define an operation. The same object can participate in different skills, yet global video aggregation can dilute the regions where their defining changes occur. To address this problem, we propose Q-ViSTA, which combines Semantic Identity with Query-Localized Transition Dynamics. Following a where-before-how design, recurrent Action Queries first collect regional trajectories, and a multiscale transition operator then describes their evolution. A parallel semantic branch retains global context, while nonlinear fusion brings both representations into a shared discovery space after temporal modeling. This representation supports both non-parametric clustering and a jointly learned parametric classifier without requiring region annotations. Experiments across six human-action and robot-skill protocols demonstrate improved domain-level discovery over SimGCD for both variants under a shared frozen backbone, with pronounced gains on unlabeled multi-step tasks that combine familiar atomic skills.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generalized category discovery, Generalized Video skill discovery, Video representation learning, Semi-supervised learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f3bsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Q-ViSTA, a framework for Generalized Video Skill Discovery (GVSD) that jointly models semantic identity and localized transition dynamics. The method employs a frozen VideoMAEv2 backbone and introduces two complementary representation branches. The semantic branch captures global appearance and contextual information, while the dynamic ator to modbranch uses recurrent Action Queries to extract localized trajectories and a multiscale transition operel temporal changes. These representations are integrated through a nonlinear fusion module and optimized using several representation-learning objectives. The framework supports both non-parametric clustering and parametric classification. Experiments are conducted on six protocols covering human actions and robot manipulation skills. The results show improvements over several GCD baselines, particularly on the RoboCasa365 Atomic-to-Composite protocol, suggesting the potential benefits of combining semantic context with localized temporal information for video category discovery.

Critical strengths

This paper investigates the joint modeling of semantic information and localized temporal dynamics for Generalized Video Skill Discovery, addressing a meaningful research problem. The authors propose a dual-branch framework that integrates global semantic representations with localized transition modeling. The overall design is reasonable, as it considers both contextual information and temporal interactions in videos. Furthermore, the proposed method is evaluated on six protocols covering human actions and robotic manipulation skills, achieving performance improvements over existing baselines. In particular, the results on the Atomic-to-Composite task are encouraging, demonstrating the potential effectiveness and applicability of the proposed framework.

Critical weaknesses

The main limitation of this paper lies in its relatively limited methodological novelty, as its core components largely rely on existing attention mechanisms, temporal modeling techniques, and feature fusion strategies, without introducing a sufficiently distinctive technical contribution. Moreover, the experimental results do not adequately demonstrate the necessity of several key components. For example, Table 4 shows that multiscale transition modeling provides only marginal improvements on several datasets and even decreases novel-class accuracy in some cases, raising questions about the effectiveness of this relatively complex design. In addition, the ability of Action Queries to capture interaction-relevant regions is mainly supported by qualitative visualizations, without sufficient quantitative validation. Regarding the Atomic-to-Composite task, the current experiments do not adequately isolate the effects of object appearance and scene context, making it difficult to determine whether the observed improvements genuinely reflect compositional generalization. Overall, although the proposed method achieves encouraging results, the experimental evidence remains insufficient to convincingly establish its methodological novelty and the effectiveness of its key components.

Questions for authors
  1. How can the authors demonstrate that the proposed framework genuinely captures temporal dynamics rather than relying primarily on appearance and contextual information? I suggest conducting additional experiments involving temporal shuffling or frame reversal to verify whether the model truly depends on temporal transitions for category discovery.

  2. The results in Table 4 show that multiscale transition modeling provides only marginal improvements on several datasets, while Table 13 indicates that the shortest temporal scale dominates the learned mixture weights. Could the authors further explain the necessity of the multiscale design and provide stronger experimental evidence to justify its additional complexity?

  3. The effectiveness of Action Queries is mainly demonstrated through qualitative attention visualizations, which are insufficient to establish their ability to capture meaningful interaction regions. Could the authors provide quantitative evaluations of localization accuracy and temporal consistency to demonstrate that Action Queries genuinely track action-related regions rather than simply selecting discriminative appearance features?

  4. The Atomic-to-Composite experiment is presented as important evidence of the model's compositional generalization ability. However, the current experimental protocol does not adequately control variations in object appearance and scene context. How can the authors ensure that the reported improvements result from learning novel combinations of atomic skills rather than exploiting semantic or visual differences between categories?

  5. Although the proposed method achieves improvements over existing baselines, it introduces additional trainable modules and optimization objectives. Could the authors provide more controlled comparisons with similar model capacity and computational budgets? Without such comparisons, it remains unclear whether the improvements come from the proposed query-localized transition modeling or simply from increased architectural complexity.

Reviewer_hvKZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Q-ViSTA for generalized video skill discovery by combining Semantic Identity with Query-Localized Transition Dynamics. Specifically, recurrent Action Queries are employed to collect regional trajectories, followed by a multiscale transition operator to characterize their temporal evolution. The resulting representations are fused for category discovery. Experiments on six human-action and robot-skill protocols report improvements over several GCD baselines, particularly on the Atomic-to-Composite setting.

Critical strengths

(1) The paper combines Semantic Identity with Query-Localized Transition Dynamics to capture both global semantic information and localized temporal changes. The framework does not require additional region-level annotations, reducing the need for extra supervision. (2) The method is evaluated on six human-action and robot-skill protocols and demonstrates encouraging overall performance across different settings. The improvements on the Atomic-to-Composite protocol are particularly notable, while the ablation results provide further evidence for the effectiveness of the main components.

Critical weaknesses

(1) The proposed framework combines established techniques, including query-based feature extraction, recurrent temporal modeling, and multiscale transition encoding. However, the technical differences from previous query-based representation learning and video category discovery methods could be explained more clearly. (2) The improvements on novel categories vary across datasets. For example, the non-parametric Q-ViSTA achieves lower New accuracy than SimGCD on UCF101 and K400, while Q-ViSTA-Param also performs slightly worse than NC-GCD on K400. Further discussion would help explain these results and clarify the role of temporal modeling in novel-category discovery.

Questions for authors

(1) The recurrent Action Queries are used to capture localized temporal dynamics. It would be helpful to explain how the recurrent updates improve temporal consistency compared with extracting Queries independently at each tubelet. (2) The temporal offsets in the multiscale transition operator are set to {1, 2, 4}. Is there a particular reason for this choice? It would be useful to see how different offset combinations affect the results. (3) According to Table 4, multiscale modeling brings relatively small improvements on SSv2 and Atomic65. Could the authors discuss the benefits of adaptive multiscale routing compared with fixed-weight fusion? (4) In Table 1, the non-parametric Q-ViSTA achieves lower New accuracy than SimGCD on UCF101 and K400, although it shows improvements in overall performance on some datasets. Further discussion of these differences would help clarify the results. (5) Some key ablation experiments are reported using a single random seed. Reporting the mean and standard deviation over multiple runs would help assess the stability of the results. (6) MCCL is included in Appendix F, but the experimental settings are not entirely consistent. It would be helpful to clarify these differences or provide a comparison under more consistent settings.

Reviewer_vixLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Q-ViSTA, a framework for Generalized Video Skill Discovery (GVSD) that aims to discover both known and novel action or skill categories from partially labeled video datasets. The method combines two representations: Semantic Identity, which captures global video context, and Query-Localized Transition Dynamics, which models localized temporal changes. Specifically, recurrent Action Queries extract regional observations over time, while a multiscale transition operator models temporal changes using predefined offsets of {1, 2, 4}. The resulting semantic and dynamic representations are fused for category discovery using either non-parametric clustering or a parametric classifier. Experiments across six human-action and robot-skill protocols report improvements over existing generalized category discovery baselines, particularly for discovering novel composite robot task categories. The paper also provides component ablations, sensitivity analyses, and qualitative visualizations to investigate the proposed architecture.

Critical strengths
  • The paper studies Generalized Video Skill Discovery (GVSD), extending generalized category discovery to video data. The motivation for incorporating both semantic context and localized temporal changes is reasonable, particularly for robot manipulation tasks involving similar objects but different operations. The Atomic-to-Composite protocol also provides an interesting evaluation setting for discovering previously unlabeled multi-step robot task categories.

  • The authors evaluate Q-ViSTA across six protocols, and the results demonstrate improvements in human-action and robot-skill domain averages over SimGCD, with particularly substantial gains on the Atomic-to-Composite protocol. The evaluation considers both non-parametric clustering and parametric classification, demonstrating the applicability of the proposed representations to different category discovery mechanisms. The paper further provides component ablations, query-capacity and supervision sensitivity analyses, and qualitative visualizations. These experiments offer useful evidence supporting the effectiveness of the complete framework, while the detailed implementation and evaluation information improves reproducibility.

Critical weaknesses
  • Insufficient validation of the dual-representation learning framework.

One of the central contributions of the paper is the separate learning of Semantic Identity and Query-Localized Transition Dynamics, followed by nonlinear fusion. However, the core ablation study in Table 4 retains the Semantic branch in all configurations and primarily investigates the effects of query-based localization and multiscale temporal modeling. Although a Semantic-only baseline is included, the paper does not report a Transition-only baseline or systematically evaluate the benefits of the proposed fusion mechanism. Furthermore, the paper does not sufficiently investigate whether the separate learning objectives and stop-gradient operation in the fusion module are necessary. Given that dual-representation learning is one of the paper's main methodological contributions, these missing controls make it difficult to establish whether the proposed dual-representation learning strategy offers specific benefits over simpler alternatives.

  • Insufficient justification of the proposed recurrent query and multiscale transition mechanisms. The paper motivates recurrent Action Queries as a mechanism for collecting regional trajectories before modeling their temporal transitions. However, Equation (4) aggregates the four query observations into a single representation at each tubelet using weights computed separately at each temporal position. Consequently, the aggregated sequence may be dominated by different queries at different time steps, meaning that the transition descriptors in Equation (5) do not necessarily characterize changes along a consistent query-specific regional trajectory. Although these descriptors are constructed from states derived from the aggregated sequence, query-specific regional correspondence is not explicitly preserved. While such aggregation may still be effective for category discovery, its relationship to the proposed "where-before-how" motivation requires further clarification. As stated in Section 3.3, the frozen video encoder and video-conditioned query initialization already provide temporal context. It is therefore unclear what additional benefit is provided by recurrent query updates. Although Table 4 demonstrates improvements from query-based selection over global pooling, it does not isolate the contribution of recurrence. Moreover, the qualitative visualizations do not expose individual query trajectories. A controlled ablation that reuses the video-conditioned initial queries at every tubelet without GRU updates, while preserving the remaining architecture and training objectives, would help establish the necessity of recurrence. In addition, the multi-scale transition operator uses predefined offsets of {1, 2, 4}. Although Table 4 includes single-scale comparisons, the improvements from multiscale modeling are relatively modest across most evaluated datasets. Appendix E.2 further shows that the shortest scale receives more than 93% of the effective mixture coefficients across the examined datasets. While these coefficients do not directly measure each scale's contribution to accuracy, the results raise questions about whether the additional multiscale routing complexity is sufficiently justified.

Because recurrent query updates and multiscale transition modeling are presented as core elements of the "where-before-how" design, the limited evidence isolating their individual contributions weakens support for the paper's central claims beyond the demonstrated benefits of query-based selection alone.

Questions for authors

Q1. Dual-representation learning (W1): Could the authors clarify whether they have evaluated Transition-only representations or alternative fusion strategies, particularly without the semantic stop-gradient operation? Such results would help determine whether the proposed dual-representation learning framework provides benefits beyond simpler alternatives.

Q2. Recurrent queries and multiscale modeling (W2): Could the authors clarify how temporal correspondence is maintained after the time-varying query aggregation in Equation (4)? Additionally, are there results comparing recurrent queries with a non-recurrent variant that reuses the video-conditioned initial queries at each tubelet? Such evidence would help establish the necessity of the recurrent updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.