acceptodds
Withdrawn from ICLR 2027

Decoupling Temporal Action Ownership and Actor Geometry for Spatio-Temporal Action Localization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatio-temporal action localization must identify what action occurs, when it starts and ends, and where each actor is in every frame. We introduce YOLO-ST, a detector that separates temporal action ownership from spatial actor geometry. A Kinetics-pretrained VideoMAE-L pathway supplies semantics, a frozen (2+1)D pathway that sees every frame supplies motion near likely actors, and a frozen COCO-pretrained YOLO11-L pyramid supplies frame appearance. Dense per-frame heads own geometry, tube queries own actor identity and temporal extent, and at inference a training-free snap moves each linked tube box toward the dense detection of the same class. Evaluating on UCF101-24, we find that published frame mAP is not one metric: on identical predictions, the conventions in use span 6.6 points, from 88.47 under the all-frame evaluator that ROAD recommends to 95.10 under our re-implementation of the one released with YOWOFormer. Under the latter our checkpoint exceeds YOWOFormer-L's reported 93.35; under ROAD's two all-frame evaluators, whose protocol BMViT cites, it scores 88.47 and 89.30, below BMViT's 90.7, because it fires on frames outside the annotated actions, a failure that gating by the tubes' own temporal extent does not repair. Video mAP is 89.82 / 73.39 / 34.97 at spatio-temporal overlap 0.2, 0.5, and 0.5:0.95, level with STAR-L on the first two and 0.83 below it on the third. Substantial gaps to the strongest results remain on JHMDB-21 (91.5 frame), AVA v2.2 (33.23) and MultiSports (42.11 frame).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatio-temporal action localization, video action-detection, actor tubes, video transformers, motion representation learning, VideoMAE, dense prediction, cross-dataset transfer, evaluation protocol, UCF101-24

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uz4tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces YOLO-ST, combining VideoMAE, a frozen YOLO11 backbone, and a frozen motion pathway, with dense heads predicting frame geometry and tube queries predicting actor ownership and temporal extent. It evaluates the framework on four benchmarks.

Critical strengths
  1. Scoring the same predictions under different conventions produces frame mAP ranging from 88.47 to 95.10, exposing consequential differences in frame selection, AP integration, and bounding-box conventions.

  2. Geometry snap improves UCF101-24 test video AP50 and AP50:95 by 0.46 and 0.79 points, respectively, providing concrete evidence that dense detections can improve tube localization.

Critical weaknesses

This paper is obviously low quality and should be rejected. Without any innovation or technical contribution shown in paper.

  1. The central benefit of ownership–geometry decoupling is not established. TubeR [1] and STAR [2] already provide explicit tube modeling, so this paper must demonstrate the additional value of its division of responsibilities through controlled comparisons against dense-only and fully shared dual-head alternatives. Tube-extent gating fails to improve action-free-frame detection, while JHMDB and MultiSports ultimately use linked dense detections because these outperform tube queries (section 5.3), leaving the necessity of the central ownership branch unproven. Also, the performance comparisons do not compensate for the missing methodological evidence.

  2. The training objective is underspecified at precisely the points supporting the main claim. In this paper, equation (3) does not mathematically define key terms or specify the Hungarian matching cost and supervision of unmatched queries. For AVA, the paper explains only dense-loss masking, leaving unclear how the tube branch’s softmax classification and interval/boundary supervision accommodate sparse, multilabel annotations; consequently.

  3. The frozen motion trunk was trained using labels from the Tune and Confirm partitions, removing it reduces Confirm frame mAP by 3.13 points but test frame mAP by only 0.30, exposing severely inflated validation evidence (section 4.2, Table 2). The selection protocol is contaminated, and retrospective freezing cannot repair it.

  4. Geometry snap lacks an identity-preserving association mechanism. Section 3.4 moves each tube box 75% toward the highest-IoU detection of the same class, without describing actor-identity constraints or one-to-one assignment across tubes. This rule permits identity switches or multiple tubes snapping toward the same detection when same-class actors overlap, and no corresponding error analysis is provided.

In the experiment, the ablations do not justify the architectural complexity. Removing the task adapters and pyramid adapter improves Confirm frame mAP by 0.30 and 0.40 points, respectively, while removing cross-clip memory has essentially no effect; even within the authors’ contaminated validation protocol, these results do not support retaining those components. Their video-mAP ablations are omitted, preventing assessment of whether they improve temporal ownership sufficiently to justify their inclusion.

[1] Zhao, J., Zhang, Y., Li, X., Chen, H., Shuai, B., Xu, M., Liu, C., Kundu, K., Xiong, Y., Modolo, D., Marsic, I., Snoek, C.G.M. and Tighe, J., 2022, June. TubeR: Tubelet transformer for video action detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 13598–13607).

[2] Gritsenko, A.A., Xiong, X., Djolonga, J., Dehghani, M., Sun, C., Lucic, M., Schmid, C. and Arnab, A., 2024, June. End-to-end spatio-temporal action localisation with video transformers. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 18373–18383).

Questions for authors
  1. What operational definition of “decoupling” is used, and what binds dense geometry to tube identity?

  2. Under identical backbones, data, and training budgets, how do dense-only, fully shared dual-head, and partially isolated models compare?

  3. Does the tube branch improve temporal boundary error, temporal-only IoU, or false positives on action-free frames?

  4. How does geometry snap prevent identity switches and many-to-one associations? What are the identity-error rates before and after snapping?

  5. What are the exact definitions of all losses in Equation (3), the matching cost, and unmatched-query supervision?

  6. What ensures that the two interval queries within an actor slot represent the same person?

  7. Is the tube branch enabled on AVA? How are multilabel classification and missing interval annotations handled?

Reviewer_4FuSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes YOLO-ST, which combines semantic, appearance, and motion pathways with dense frame heads and tube queries. Dense predictions supply spatial geometry, while tube queries model identity and temporal extent; geometry snap combines their outputs at inference. The paper also audits UCF101-24 frame-mAP conventions and reports results on four benchmarks.

Critical strengths
  1. The proposed division of responsibilities is concrete, and geometry snap provides measured improvements in video localization.
  2. The UCF101-24 evaluation audit documents specific implementation differences beyond frame selection.
Critical weaknesses
  1. The experimental support remains incomplete. On JHMDB-21 and MultiSports, the reported video results use dense detection linking because it outperformed tube queries (Section 5.3). The paper does not explain this dataset-dependent behavior, leaving it unclear under what conditions the proposed decomposition is beneficial. This limits the experimental support for the benefit and scope of the central design.
Questions for authors

Could you clarify the dataset-dependent performance differences between tube-query prediction and dense detection linking? Section 5.3 states that dense linking outperforms tube queries on JHMDB-21 and MultiSports, whereas section 3.4 says tube queries are used for inference.

Reviewer_GnDyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates spatio-temporal action localization by proposing YOLO-ST, a decoupled framework separating frame-level actor geometry prediction and temporal action ownership modeling. The model adopts a three-branch multi-scale feature pyramid with semantic, appearance and motion pathways. Dense prediction heads handle per-frame bounding box estimation, while tube queries track actor identities and action time intervals. A training-free Geometry Snap module is introduced at inference to refine tube box coordinates. The work also reveals inconsistent evaluation protocols on UCF101-24, showing that different assessment rules can cause up to 6.6-point fluctuation in frame mAP. Extensive experiments are carried out across four datasets. The authors report comparable video mAP against STAR-L on UCF101-24 but observe clear performance gaps on more challenging benchmarks including AVA and MultiSports. A key unresolved defect remains: the model consistently produces false positive detections outside annotated action timestamps, and multiple attempted mitigation strategies fail to eliminate this temporal false alarm issue.

Critical strengths
  1. Valuable community-oriented benchmark analysis: The paper systematically quantifies how evaluation settings affect reported metrics on UCF101-24, highlighting risks of unfair cross-paper comparison and raising awareness of benchmark protocol chaos for spatio-temporal action localization research.
  2. Clear model decomposition design: The work explicitly decouples spatial box prediction and temporal action tube modeling, providing an intuitive architecture to analyze spatial and temporal failures separately.
  3. Comprehensive ablation studies exploring multiple remedies for temporal false alarms: A large set of candidate strategies including temporal gating, training with action-free clips and boundary erosion are tested and documented, offering useful negative results for follow-up research.
Critical weaknesses
  1. Critical core limitation remains unsolved: The proposed decoupled architecture cannot suppress temporal false alarms where detections appear outside annotated action periods. The tube query branch fails to constrain the dense prediction heads, and Geometry Snap only refines box locations without fixing false positives. This fundamental defect undermines the core practical value of spatio-temporal localization.
  2. Severe data leakage contaminates key ablation results: The frozen (2+1)D motion backbone has seen samples from the held-out tuning split before ablation. The large performance gain observed on the tuning set overestimates the actual contribution of the motion pathway; only test-set results reflect its real marginal benefit.
  3. Limited empirical improvement over state-of-the-art methods: The model only matches STAR-L on UCF101-24 under specific evaluation protocols and falls substantially behind leading SOTA on AVA, JHMDB-21 and MultiSports. The unified performance gain brought by the proposed architectural innovation is insufficiently demonstrated.
Questions for authors

This paper provides useful observations on benchmark evaluation pitfalls and maintains high transparency on experimental records. However, it suffers from fundamental unresolved temporal false alarm issues, contaminated ablation results due to data leakage, and insufficient performance advantages over existing SOTA on challenging benchmarks. The core architectural contribution cannot deliver consistent performance gains across datasets, and key analyses of model efficiency and internal failure mechanisms are missing. The identified problems require major rework of model design and large-scale supplementary experiments, which exceed the scope of normal revision. Therefore, a rejection is proposed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.