acceptodds
Withdrawn from ICLR 2027

Learning Joint Video and Text Representations and Taxonomy of Actions in Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The video of a complex action can be decomposed into a temporally ordered sequence of simpler sub-actions, each of which can, in turn, be recursively decomposed to form a video tree. Given a video whose only annotation is a set of names of possible complex action contained in it, this paper aims to simultaneously segment the video into clips of the complex actions, learn a video tree of sub-actions for each segment, and assign linguistic names to its nodes. Each video tree has a co-existing text tree, formed by the text names of the video tree nodes. We refer to both tress together as the definition tree. In the absence of annotations, we generate text trees using LLMs. To construct video trees, we recursively divide estimated action segments in video into ordered temporal sub-actions. For both trees we compute representation features bottom-up, starting from the leaf actions. Action–sub-action attention (ASA) lets neighboring sub-actions exchange information before they are combined into a parent action in both trees. As a second objective, we use taxonomic supervision to draw semantically related actions close to each other to form a taxonomy of the complex actions in the input video. Compared with Ti-FAD, our video features better preserve the temporal order of sub-actions for unseen actions (+16.7%), yield better taxonomic grouping (+18.2%), and improve zero-shot localization mAP by 9.0-10.8% on THUMOS14 and 2.3-3.9% on ActivityNet.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
hierarchical video representation, zero-shot temporal action localization, action hierarchy, video understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JwusOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper primarily addresses the task of zero-shot temporal action localization (ZS-TAL), aiming to recognize and localize unseen action classes in untrimmed videos. The authors propose a hierarchical video–text representation framework that decomposes complex actions into temporally ordered sub-actions. Specifically, an LLM is employed to generate hierarchical textual descriptions of actions, while corresponding video trees are constructed by recursively partitioning predicted action intervals into temporal segments. An Action–Sub-action Attention (ASA) module is introduced to aggregate information across sub-actions and their parent actions in both video and text representations. In addition, taxonomy-based supervision is incorporated to encourage semantically related action classes to share similar representations, and an attribute-based duration prior is used to refine localization confidence during inference. Experimental results show improvements over the Ti-FAD baseline in unseen-action localization performance.

Critical strengths

The proposed method achieves consistent improvements over the Ti-FAD baseline on zero-shot temporal action localization benchmarks, demonstrating its effectiveness in localizing unseen action classes.

Critical weaknesses
  1. The Introduction does not clearly establish the primary task of zero-shot temporal action localization (ZS-TAL), particularly what the model is expected to predict and what makes this task challenging. Instead, the authors formulate four objectives involving action localization, video tree construction, linguistic labeling of tree nodes, and action taxonomy construction. However, the relationships among these objectives are not adequately motivated. In particular, it remains unclear whether the latter three objectives represent independent prediction tasks or intermediate representations designed to improve ZS-TAL. The Introduction should clearly distinguish the actual research task from the proposed methodological components.
  2. The first paragraph of the Introduction states that the only annotation available for a video is a set of possible action names, which could suggest a weakly supervised temporal action localization setting. However, the proposed method uses action-class labels and temporal boundary annotations for seen classes during training. Although the authors may be referring to the test-time inputs, this distinction is not explicitly established in the Introduction. The paper should clearly differentiate training supervision from inference-time information and explain the differences between the strict zero-shot and coarse-supervision protocols. 3.The authors motivate their approach by representing complex actions as temporally ordered sub-actions, but provide limited analysis of why existing temporal action localization methods cannot effectively exploit such hierarchical structures. In fact, action decomposition and sub-action modeling have been extensively investigated in prior work, including weakly supervised temporal action localization. For example, Action Unit Memory Network for Weakly Supervised Temporal Action Localization (CVPR 2021) explicitly models action units to improve localization. Other studies have also investigated sub-action composition and hierarchical category supervision. The paper should provide a more comprehensive discussion of these closely related approaches and clearly distinguish its contributions from existing hierarchical action modeling methods.
  3. The discussion of related methods appears relatively late in the Introduction and mainly distinguishes previous approaches based on their structural configurations, rather than analyzing their fundamental limitations. For example, the authors point out that PDA employs three action phases, but do not explain why this decomposition is insufficient or why recursively decomposing an action into more sub-actions would necessarily produce better representations. Moreover, the authors make a strong claim regarding recursive action composition at arbitrary depths without sub-action annotations. However, Appendix A.4 indicates that THUMOS14 and ActivityNet use only one sub-action level, while FineGym and FineDiving use at most two levels. The experiments therefore provide limited evidence for the claimed generality of arbitrary-depth recursive composition.
  4. The main contributions involve hierarchical representation construction, Action–Sub-action Attention (ASA), and taxonomy-based auxiliary supervision. However, the paper does not clearly explain which specific limitations of existing ZS-TAL methods are addressed by each component. Consequently, the overall framework appears to combine several hierarchical modeling and supervision techniques without sufficiently establishing why their joint design is necessary for the target task.
  5. According to Section 3.2.2, the video branch constructs its hierarchical structure by uniformly partitioning each predicted action interval into a predefined number of equal-length temporal segments. Deeper sub-action levels are constructed through the same procedure. Therefore, the temporal tree structure is determined by a predefined partitioning strategy rather than learned sub-action boundaries or semantic transitions. Furthermore, there is no explicit supervision or evaluation establishing whether the resulting video segments correspond to the sub-actions described in the generated text trees. The authors also acknowledge this limitation in the Conclusion. 7.Appendix A.4 indicates that THUMOS14 and ActivityNet use a fixed number of ten sub-actions for each action category. However, different complex actions naturally consist of different numbers of meaningful sub-actions, making this fixed decomposition potentially inappropriate. More importantly, if the proposed decomposition mainly provides finer temporal sampling, it is unclear how the resulting representation fundamentally differs from conventional multi-scale temporal feature modeling. The authors should investigate whether the improvements originate from meaningful sub-action semantics and temporal composition, rather than simply using additional textual descriptions and temporal features.
Questions for authors

See Weakness

Reviewer_ciUWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for learning joint video-text representations and an action taxonomy using a "definition tree" that recursively decomposes complex actions. An LLM automatically generates a textual tree of sub-actions, and on the video side, a video tree is built by dividing the estimated action segment into equal parts, which allows training without sub-action labels. On datasets such as THUMOS14 and ActivityNet, the proposed method outperforms existing state-of-the-art methods (e.g., Ti-FAD) in zero-shot temporal action localization (TAL) accuracy, as well as in temporal ordering and taxonomy grouping for unseen classes.

Critical strengths
  • The approach of automatically generating the sub-action structure (the definition tree) with an LLM and using it to guide video representations is original, and it removes the dependence on hierarchical labels, which are costly to annotate.
  • On zero-shot TAL, a challenging task, the method improves mAP by up to 10.8% on THUMOS14 and up to 3.9% on ActivityNet, giving clear quantitative evidence of its practical effectiveness.
  • Beyond plain classification, the introduction of Taxonomy Supervision yields a representation space that remains semantically consistent even for unseen actions.
Critical weaknesses
  • The video tree is built by splitting the action segment into equal-length chunks. In real actions, sub-actions rarely have uniform durations, so this oversimplified assumption likely becomes a bottleneck for the representation.
  • The LLM-generated text trees are not validated by humans. If they contain hallucinations, the model risks learning an incorrect structure, which raises concerns about whether the representations are properly grounded.
  • Among the components of the proposed method, the Attribute Similarity Prior (ASP) yields a much smaller accuracy gain than the other components (ASA and TS), so its contribution appears limited.
  • Figures should be embedded in vector format so that their quality does not degrade when enlarged.
Questions for authors
  • Do you have quantitative results on how equal-length splitting affects videos in which sub-action durations vary widely? Have you considered introducing a dynamic segmentation method to address this?
  • Would it be possible to conduct a qualitative human evaluation on a small number of samples to assess how well the LLM-generated sub-action decompositions match the actual video content?
  • The number of child nodes K, a hyperparameter, is set separately for each dataset. Could you provide a sensitivity analysis showing how much variation in K affects the final mAP?
Reviewer_3FVrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a method to generate action representations in a hierarchical manner. Their method consists of a visual and textual branch, each of which decomposes a video of actions into a series of sub-actions. These sub-actions are grouped into an action-tree, which are then passed through an "Action–Sub-action Attention" to obtain a final feature representation. It is trained without temporal localizations, but with a coarse action and taxonomy label for the entire video. They apply this approach to the task of zero-shot temporal action localization, on datasets Thumos and ActivityNet.

Critical strengths
  • The approach is a novel method for computing a video representation. It seems relatively general, in that it mainly requires a video-level annotation to train rather than needing more fine-grained annotations which can be costly to obtain.
Critical weaknesses
  • The method relies on a VLM to produce reasonable text trees which are temporally coherent. From the text tree the action can be classified, but one question would be if one could directly predict the temporal bounds of the action with an additional LLM call on the text tree? It would be coarse, but this would be a reasonable baseline to compare against on this task.
  • Building on the above question, it is not clear if this representation is necessarily a better representation than the VLM on which it relies. Since a necessary component of this approach is the VLM, then to prove the importance or viability of the proposed representation learning algorithm, there should be some aspect of the video which the new representation captures which the given VLM cannot. The current experimental backing does not show strong enough evidence that the representation generated by the whole system is necessarily stronger than a component of the system.
Questions for authors
  • How robust is the approach relative to the value of K. From the appendix, K is selected based on dataset statistics, but does this granularity brought by changing K improve or worsen results?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you, and withdrawal of the submission

Dear Area Chair and Reviewers,

Thank you for the reviews and the discussion.

Reviewer ciUW, thank you for confirming that our responses resolve your concerns, for your constructive suggestions, and for the kindness of your comments and your recognition of the work throughout the discussion.

Reviewer Jwus, thank you for the depth of your review, for your suggestions on how to improve the work, and for responding promptly to every one of our replies across both rounds.

Given the current scores, we have decided to withdraw the submission. The comments we received in this process have shaped the work for the better, and we are grateful to all reviewers and the Area Chair for their time and care.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.