acceptodds
Under review as a conference paper at ICLR 2027

Semantic Frequency based Query Relations for Temporal Action Detection

Abstract

Accurately identifying the boundaries of actions is crucial for temporal action detection (TAD), but this task remains challenging as sudden visual changes do not always indicate action boundaries. Existing frequency-domain TAD methods can capture the global temporal changes in videos, but they ignore that some of these changes do not necessarily correspond to action boundaries but are caused by background changes, thereby affecting the accuracy of action boundary localization. During the decoding stage, existing query-based detectors update queries based on the temporal features of the video, but a large number of queries corresponding to different action boundaries may temporally overlap or cover different parts of the same action instance, resulting in redundancy or fragmentation between the predicted action segments. To solve this problem, we propose Semantic Frequency based Query Relations Model for Temporal Action Detection (SQRAct). Specifically, we first design a global frequency-domain semantic filter in the encoder, which dynamically generates a filter bank based on the action and background semantic information of the video, enhancing the ability to distinguish action boundaries through semantically guided frequency filtering. Moreover, considering the redundancy and fragmentation of different queries for the same action instance, we present an inter-layer query relations refinement (QRR) module in the decoder. The QRR module analyzes the containment or adjacency relationships between queries with similar features and further refines the features and temporal boundaries of the queries. SQRAct simultaneously models both the global frequency-domain semantic features and the explicit query relationships to enhance the ability to distinguish action boundaries and refine redundant and fragmented action predictions, thereby significantly improving the accuracy of action boundary localization. Extensive experiments on THUMOS14, HACS-Segment, and ActivityNet v1.3 show that the proposed SQRAct model outperforms the existing state-of-the-art methods, which validates that enhancing boundary distinguishability and explicitly modeling instance-level query relationships is crucial for TAD.

Primary area:
learning on time series and dynamical systems
Keywords:
temporal action detection, action boundary localization, frequency-domain modeling, query relations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Yf9COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

SQRAct proposes a temporal action detector combining semantic-conditioned frequency filtering, multiscale feature enhancement, and inter-layer query refinement. Building on DiGIT’s decoder, it uses containment and adjacency relationships between predicted segments to refine their representations and boundaries, targeting duplicate and fragmented detections.

Critical strengths

The paper addresses a meaningful distinction between genuine action boundaries and feature changes caused by internal motion or background variation. Explicitly examining relationships between partial predictions is a reasonable direction for improving temporal localization.

Critical weaknesses
  1. After reading your manuscript, I think that the claimed foreground/background semantics are unsubstantiated. As shown in Section 3.3 that introduces an action probability P(t)P(t) without specifying its supervision, loss, or evidence that it identifies foreground. If P(t)P(t) is constant, both weighted averages in Eq. (1) reduce to the same global feature mean before their separate projections; no stated constraint excludes this degenerate case. Consequently, the central claim of semantic foreground/background conditioning rests on an unverified interpretation of learned weights.

  2. Also, in the methodology section, the frequency mechanism does not establish selective suppression of background changes. A single video-conditioned H(f)H(f) applies the same gain to action and background contributions occupying the same frequency component, so pooled semantic conditioning alone does not resolve their ambiguity. Moreover, Eq. (2) gives Hh(f)∈(1,3)H_h(f)\in(1,3), while Hb(f)H_b(f) approaches 2 when its Gaussian term vanishes: these branches amplify rather than individually suppress out-of-band content. Neither the unspecified filter combination nor the experiments demonstrate how this construction preserves genuine boundaries while rejecting visually similar nuisance transitions.

  3. QRR substitutes semantic similarity for instance identity without validating that assumption. Consecutive repetitions of the same action can be highly similar yet belong to distinct instances, while different phases of one action can have dissimilar features; the similarity gate therefore does not reliably distinguish fragmentation from legitimate instance separation. Equation (10) permits only expansion within the adjacency update, creating a concrete risk of bridging distinct instances. Containment refinement also moves predictions toward a query selected by classification confidence, although classification confidence does not establish localization quality, a distinction explicitly addressed by ReAct [2], and no evidence shows that this reduces duplicates rather than making them more alike.

  4. Another question is that section 3.4 does not formally define the containment, proximity, and adjacency functions, and the paper omits the complete training objective, essential hyperparameters, and a reproducible specification of SFE. Equations (8)–(9) also associate aj−a_j^{-} with j+j^{+}, despite the prose identifying j−j^{-} as the predecessor, leaving the directional convention inconsistent. The appendix promised for essential parameterization is absent from the supplied PDF, so these omissions directly obstruct verification. The central algorithm cannot be reproduced reliably.

Some issues regarding your experimental part.

The main experimental claims are internally inconsistent and use an unexplained weak baseline. Section 4.1 claims 41.5% ActivityNet mAP and a 0.3-point improvement, whereas Table 1 labels the proposed method “FRQAct” and reports 39.2% versus 37.8%, implying a 1.4-point improvement. The original DiGIT reports 42.0% with InternVideo2, or 41.3% without NMS [1], making the unexplained 37.8% reproduction a serious comparison problem. Together with the missing HACS results, these discrepancies leave the claimed superiority across datasets unsupported.

Furthermore, no experiment establishes which proposed mechanism actually contributes. The supplied paper contains no component ablations, repeated-run variability, or direct measurements of boundary errors, duplication, and fragmentation. Given the inherited DiGIT decoder [1], existing relational action queries [2], and prior semantic-adaptive frequency filtering [3], aggregate mAP cannot establish the specific contribution of the proposed combination. A 0.9-point THUMOS14 gain without controlled ablations or capacity and runtime comparisons is inadequate evidence for the necessity of this substantially expanded architecture.

[1] Kim, H.-J., Lee, Y., Hong, J.-H. and Lee, S.-W., 2025, June. DiGIT: Multi-dilated gated encoder and central-adjacent region integrated decoder for temporal action detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24286–24296).

[2] Shi, D., Zhong, Y., Cao, Q., Zhang, J., Ma, L., Li, J. and Tao, D., 2022, October. ReAct: Temporal action detection with relational queries. ECCV 2022 (pp. 105–121). Springer.

[3] Huang, Z., Zhang, Z., Lan, C., Zha, Z.-J., Lu, Y. and Guo, B., 2023, October. Adaptive frequency filters as efficient global token mixers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 6049–6059).

Questions for authors
  1. What supervises P(t)P(t), and what evidence excludes constant or semantically incorrect foreground/background assignments? Can GFSF distinguish true boundaries from background changes with overlapping frequency content? Also, why does Hb(f)H_b(f) approach 2 outside its target band, and how are the filter responses combined and constrained?

  2. How does QRR distinguish adjacent instances of the same class from fragments of one instance? and why should classification confidence identify the query with better boundaries, and does containment refinement actually reduce duplicate detections?

  3. What are the exact relation functions, thresholds, self-pair masks, and rules for queries without eligible neighbors? Also, which predecessor/successor indexing in Eqs. (8)–(10) is correct, and how are conflicting reverse updates resolved?

  4. What are the complete training losses, SFE operations, FFT conventions, and training/inference settings?

  5. Which ActivityNet results are correct, and why does the reproduced DiGIT fall substantially below its original results? Are feature extraction, external classification scores, and NMS settings identical across the relevant comparisons? What do individual and combined module ablations show, including variation across seeds and computational overhead?

Reviewer_pdqcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a semantic frequency-based query relation framework for temporal action detection. The authors aim to integrate both feature similarity and temporal relationships to improve the quality of queries. To be specific, they first design global frequency-domain semantic filters in the encoder to generate a set of filtering banks to distinguish action boundaries. Furthermore, they introduce an inter-layer query relations refinement module in the decoder to analyze the containment or adjacency relationships between queries and features to further refine the features and temporal boundaries of the queries. They validate the effectiveness of the proposed designs on three widely used benchmarks.

Critical strengths

None.

Critical weaknesses

Overall, I feel that the paper was written in a rush, and the experiments are incomplete, with many important elements missing. In its current form, the paper is not yet ready for publication.

  1. The main weakness of this paper is its writing. The issues are widespread and may be difficult to address substantially during the rebuttal period. The writing is overly complicated, making the paper difficult to follow. In particular, the abstract and introduction devote considerable space to discussing the limitations of existing methods without clearly and logically identifying the central problem. As a result, the paper’s motivation remains unclear, making it difficult to assess the novelty and rationale of the proposed approach.
  2. Similarly, the related work section lists many methods but does not conclude by clearly identifying the main improvements and differences of the proposed approach compared with previous methods.
  3. Although the method section includes many detailed equations to aid understanding, it lacks high-level explanations of what these equations accomplish and how they contribute to the proposed design. More importantly, the paper lacks ablation studies, making it difficult to verify the effectiveness of these design choices.
  4. The authors currently report only overall performance on the datasets, without providing a comprehensive evaluation of the proposed method through both qualitative and quantitative results. In addition, the experimental results on HACS-Segment are missing.
Questions for authors

Please refer to critical weaknesses.

Reviewer_mbvkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses temporal action detection (TAD) in untrimmed videos. Existing frequency‑domain methods rely on fixed filters and often misinterpret background or intra‑action visual changes as genuine action boundaries. Query‑based detectors also produce redundant or fragmented predictions, where multiple queries cover partial segments of one action and prior work mostly discards overlapping queries. The authors propose SQRAct: its encoder applies a semantics‑guided adaptive frequency‑domain filter to suppress spurious boundary signals, alongside cross‑scale feature enhancement for fine‑grained boundary details. A plug‑in QRR decoder module explicitly models containment and adjacency relations among queries, refining query features and temporal boundaries instead of merely filtering duplicates. Experiments on THUMOS14, ActivityNet v1.3 and HACS‑Segments yield competitive performance.

Critical strengths
  1. Unlike prior fixed frequency filters, this paper generates filters guided by foreground‑background semantics to suppress spurious boundary signals from background or intra‑action motion, targeting a key pain point of TAD.

  2. Instead of discarding overlapping queries by tIoU, the plug‑in QRR module models containment and adjacency relations to refine query features and boundaries, alleviating redundant and fragmented predictions.

Critical weaknesses
  1. Lack of comparison against modern multimodal video LLMs. This paper only compares traditional temporal‑action‑detection baselines, without benchmarking recent MLLMs such as Qwen‑VL. It remains unclear how competitive the proposed SQRAct is against end‑to‑end multimodal large‑language‑model alternatives for temporal action detection.

  2. Insufficient analysis of hyper‑parameter sensitivity for QRR. The QRR module relies on multiple hand‑set thresholds for containment, adjacency and expansion. The paper does not provide sensitivity ablation, leaving it unclear how performance degrades when these key hyper‑parameters shift.

  3. Limited quantitative analysis for fragmented/redundant predictions. Though QRR is designed to mitigate fragment and duplicate outputs, the paper only reports standard mAP metrics. It lacks dedicated quantitative statistics to directly verify the actual reduction of fragmented or redundant predictions.

Questions for authors

Q1. Modern multimodal video MLLMs like Qwen‑VL can perform temporal action detection‑style tasks. Why are these models absent in your comparisons? Could you clarify how SQRAct compares against such MLLM‑based solutions?

Q2. QRR depends on multiple manually‑defined hyper‑parameters and targets to reduce fragmented and redundant predictions. Can you provide hyper‑parameter sensitivity results, and dedicated quantitative statistics measuring the actual reduction of fragmented / redundant predictions caused by QRR?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.