- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper primarily addresses the task of zero-shot temporal action localization (ZS-TAL), aiming to recognize and localize unseen action classes in untrimmed videos. The authors propose a hierarchical video–text representation framework that decomposes complex actions into temporally ordered sub-actions. Specifically, an LLM is employed to generate hierarchical textual descriptions of actions, while corresponding video trees are constructed by recursively partitioning predicted action intervals into temporal segments. An Action–Sub-action Attention (ASA) module is introduced to aggregate information across sub-actions and their parent actions in both video and text representations. In addition, taxonomy-based supervision is incorporated to encourage semantically related action classes to share similar representations, and an attribute-based duration prior is used to refine localization confidence during inference. Experimental results show improvements over the Ti-FAD baseline in unseen-action localization performance.
Critical strengths
The proposed method achieves consistent improvements over the Ti-FAD baseline on zero-shot temporal action localization benchmarks, demonstrating its effectiveness in localizing unseen action classes.
Critical weaknesses
- The Introduction does not clearly establish the primary task of zero-shot temporal action localization (ZS-TAL), particularly what the model is expected to predict and what makes this task challenging. Instead, the authors formulate four objectives involving action localization, video tree construction, linguistic labeling of tree nodes, and action taxonomy construction. However, the relationships among these objectives are not adequately motivated. In particular, it remains unclear whether the latter three objectives represent independent prediction tasks or intermediate representations designed to improve ZS-TAL. The Introduction should clearly distinguish the actual research task from the proposed methodological components.
- The first paragraph of the Introduction states that the only annotation available for a video is a set of possible action names, which could suggest a weakly supervised temporal action localization setting. However, the proposed method uses action-class labels and temporal boundary annotations for seen classes during training. Although the authors may be referring to the test-time inputs, this distinction is not explicitly established in the Introduction. The paper should clearly differentiate training supervision from inference-time information and explain the differences between the strict zero-shot and coarse-supervision protocols. 3.The authors motivate their approach by representing complex actions as temporally ordered sub-actions, but provide limited analysis of why existing temporal action localization methods cannot effectively exploit such hierarchical structures. In fact, action decomposition and sub-action modeling have been extensively investigated in prior work, including weakly supervised temporal action localization. For example, Action Unit Memory Network for Weakly Supervised Temporal Action Localization (CVPR 2021) explicitly models action units to improve localization. Other studies have also investigated sub-action composition and hierarchical category supervision. The paper should provide a more comprehensive discussion of these closely related approaches and clearly distinguish its contributions from existing hierarchical action modeling methods.
- The discussion of related methods appears relatively late in the Introduction and mainly distinguishes previous approaches based on their structural configurations, rather than analyzing their fundamental limitations. For example, the authors point out that PDA employs three action phases, but do not explain why this decomposition is insufficient or why recursively decomposing an action into more sub-actions would necessarily produce better representations. Moreover, the authors make a strong claim regarding recursive action composition at arbitrary depths without sub-action annotations. However, Appendix A.4 indicates that THUMOS14 and ActivityNet use only one sub-action level, while FineGym and FineDiving use at most two levels. The experiments therefore provide limited evidence for the claimed generality of arbitrary-depth recursive composition.
- The main contributions involve hierarchical representation construction, Action–Sub-action Attention (ASA), and taxonomy-based auxiliary supervision. However, the paper does not clearly explain which specific limitations of existing ZS-TAL methods are addressed by each component. Consequently, the overall framework appears to combine several hierarchical modeling and supervision techniques without sufficiently establishing why their joint design is necessary for the target task.
- According to Section 3.2.2, the video branch constructs its hierarchical structure by uniformly partitioning each predicted action interval into a predefined number of equal-length temporal segments. Deeper sub-action levels are constructed through the same procedure. Therefore, the temporal tree structure is determined by a predefined partitioning strategy rather than learned sub-action boundaries or semantic transitions. Furthermore, there is no explicit supervision or evaluation establishing whether the resulting video segments correspond to the sub-actions described in the generated text trees. The authors also acknowledge this limitation in the Conclusion. 7.Appendix A.4 indicates that THUMOS14 and ActivityNet use a fixed number of ten sub-actions for each action category. However, different complex actions naturally consist of different numbers of meaningful sub-actions, making this fixed decomposition potentially inappropriate. More importantly, if the proposed decomposition mainly provides finer temporal sampling, it is unclear how the resulting representation fundamentally differs from conventional multi-scale temporal feature modeling. The authors should investigate whether the improvements originate from meaningful sub-action semantics and temporal composition, rather than simply using additional textual descriptions and temporal features.
Questions for authors
See Weakness
Thank you, and withdrawal of the submission
Dear Area Chair and Reviewers,
Thank you for the reviews and the discussion.
Reviewer ciUW, thank you for confirming that our responses resolve your concerns, for your constructive suggestions, and for the kindness of your comments and your recognition of the work throughout the discussion.
Reviewer Jwus, thank you for the depth of your review, for your suggestions on how to improve the work, and for responding promptly to every one of our replies across both rounds.
Given the current scores, we have decided to withdraw the submission. The comments we received in this process have shaped the work for the better, and we are grateful to all reviewers and the Area Chair for their time and care.