Semantic Frequency based Query Relations for Temporal Action Detection

Accurately identifying the boundaries of actions is crucial for temporal action detection (TAD), but this task remains challenging as sudden visual changes do not always indicate action boundaries. Existing frequency-domain TAD methods can capture the global temporal changes in videos, but they ignore that some of these changes do not necessarily correspond to action boundaries but are caused by background changes, thereby affecting the accuracy of action boundary localization. During the decoding stage, existing query-based detectors update queries based on the temporal features of the video, but a large number of queries corresponding to different action boundaries may temporally overlap or cover different parts of the same action instance, resulting in redundancy or fragmentation between the predicted action segments. To solve this problem, we propose Semantic Frequency based Query Relations Model for Temporal Action Detection (SQRAct). Specifically, we first design a global frequency-domain semantic filter in the encoder, which dynamically generates a filter bank based on the action and background semantic information of the video, enhancing the ability to distinguish action boundaries through semantically guided frequency filtering. Moreover, considering the redundancy and fragmentation of different queries for the same action instance, we present an inter-layer query relations refinement (QRR) module in the decoder. The QRR module analyzes the containment or adjacency relationships between queries with similar features and further refines the features and temporal boundaries of the queries. SQRAct simultaneously models both the global frequency-domain semantic features and the explicit query relationships to enhance the ability to distinguish action boundaries and refine redundant and fragmented action predictions, thereby significantly improving the accuracy of action boundary localization. Extensive experiments on THUMOS14, HACS-Segment, and ActivityNet v1.3 show that the proposed SQRAct model outperforms the existing state-of-the-art methods, which validates that enhancing boundary distinguishability and explicitly modeling instance-level query relationships is crucial for TAD.