acceptodds
Under review as a conference paper at ICLR 2027

SpaTAD: Preserving Spatial Structure in Temporal Action Detection

Abstract

Temporal action detection (TAD) requires modeling how visual evidence evolves over time. Most detectors, however, pool each frame or snippet into a single fea- ture vector before temporal modeling. This is effective when action evidence can be summarized globally, but can become restrictive when an action is defined by where evidence occurs and how it moves, such as the path a vehicle takes through an intersection, or when concurrent actions rely on different regions of the frame. We therefore ask how far into a temporal action detector explicit spatial structure should be kept. We introduce SpaTAD, a TAD model that keeps spatial tokens separate until action scores are computed. Rather than pooling each frame into a single vector, SpaTAD maintains a causal, location-bound recurrent state for each spatial token, predicts action scores at individual tokens, and only then aggregates them across space into frame-level action scores. On MultiTHUMOS, Charades, ATARS, and ROAD-Waymo, SpaTAD improves over the strongest matched base- line by an average of 11.6 Seg-mAP and 6.5 Det-mAP. Pooling before class pre- diction hurts performance on all four benchmarks, while collapsing spatial struc- ture before temporal modeling causes an additional 5.07 Det-mAP and 10.41 Seg- mAP drop on ATARS. These results suggest that dense temporal modeling alone is not always sufficient: preserving spatial organization through class prediction can be beneficial, and location-bound temporal memory appears especially useful when actions depend on how evidence moves over time.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Temporal Action Detection, Spatiotemporal Representation Learning, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kn7bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the SpaTAD framework for temporal action detection, which aims to retain localized action evidence and spatially dependent motion histories that may be lost through early spatial pooling aggregation. There are three parts in SpaTAD. First, multi-granularity tokenizer is designed to preserve spatial structure at different granularities. Second, spatial-temporal encoder with per-token temporal memory focuses on preserving spatial structure during temporal modeling. Third, token-wise action prediction further preserves spatial structure by predicting actions at individual spatial tokens before spatial aggregation. SpaTAD achieves SOTA results on four datasets. In the ablation experiment part, the authors further investigate the impact of spatial aggregation position and demonstrate that pooling before class prediction consistently degrades performance.

Critical strengths
  1. SpaTAD achieves SOTA results over baselines across four datasets.
  2. The ablation confirms that pooling before class prediction consistently degrades performance across all four datasets, and preserving location-bound memory provides additional benefits
Critical weaknesses
  1. SpaTAD mainly combines existing techniques, including spatial-temporal modeling, self-attention, and token-wise prediction combined with spatial pooling. Thus, its novelty seems limited, since its primary contribution is the integration of established components rather than the development of a fundamentally new approach.
  2. The contributions of the multi-granularity tokenizer, within-frame attention, and LSE pooling have not been adequately evaluated individually. Furthermore, the observed performance differences could stem from the use of different pooling operators rather than solely from the position at which spatial aggregation is performed.
Questions for authors
  1. Additional ablation studies are needed to better evaluate the contributions of the key components. Specifically, how does the performance change when (1) using only single-scale tokens instead of the multi-granularity tokenizer, (2) removing the within-frame attention module, and (3) replacing the per-token SSM with alternative temporal modeling modules?
  2. The authors argue that per-token recurrent states help preserve motion histories. However, since these states are tied to fixed spatial locations rather than moving objects, how does the model effectively track motion information across locations as objects move?
3 comments
AuthorsOpenReview ↗

Response to Reviewer kn7b (1/2)

Thank you for the review. Our answers follow point by point. Numbers are test-set means ± standard deviations over runs; our general response summarises the revision.

W1. "Its primary contribution is the integration of established components rather than the development of a fundamentally new approach."

Our contribution is a design principle and its experimental study: how far into a temporal action detector the spatial structure should be kept. Temporal action detectors typically reduce each frame to one feature vector before temporal modelling; to our knowledge, SpaTAD is the first to keep the spatial structure through temporal modelling and class prediction, and to measure what this contributes. It builds on established components; its contribution is to isolate the stage of spatial aggregation and to examine its effect across temporal architectures.

Table 3 (Sec. 4.3): token-wise vs. pooled prediction across temporal modules. All rows receive the same patch tokens (the ConvLSTM on the fine token grid); the two rows of a module differ only in whether the tokens are kept until class prediction or pooled by learned attention just before it. Det-mAP / Seg-mAP, mean ± std, under the shared ablation settings (Appendix E; levels not comparable with Table 1).

MultiTHUMOSCharadesATARSROAD-Waymo
Temporal moduleDetSegDetSegDetSegDetSeg
Mamba-2 (SpaTAD)36.03±0.7351.08±0.3834.09±0.0238.80±0.3524.18±1.8141.07±2.1417.41±0.1748.93±0.07
Mamba-2 w/ pooled prediction33.79±0.2647.38±0.6327.35±0.3632.06±0.2220.83±2.4038.19±3.8216.27±0.4246.72±0.47
GRU35.49±0.2950.36±0.2033.58±0.4738.69±0.2617.77±2.5834.06±3.6917.15±0.3147.94±0.10
GRU w/ pooled prediction31.50±0.5845.82±0.5828.23±0.1133.53±0.1916.14±1.0831.84±0.7816.08±0.5546.52±1.12
Temporal attention33.29±0.1849.21±0.3332.48±0.1637.89±0.0422.72±3.2639.06±4.2816.95±0.1548.27±0.31
Temporal attention w/ pooled prediction28.81±0.6044.24±1.1526.14±0.2631.93±0.0718.79±1.4435.29±3.0315.75±0.1446.12±0.10
ConvLSTM27.22±0.3844.50±0.1727.54±0.1731.85±0.0420.43±0.4439.33±0.5716.98±0.1447.64±0.64
ConvLSTM w/ pooled prediction28.20±0.6543.56±0.4025.04±0.2230.30±0.1521.12±0.5840.14±0.3315.57±0.0546.32±0.21

Keeping the tokens through temporal modelling is not enough: the pooled-prediction rows do so, yet with Mamba-2, a GRU or temporal attention they trail token-wise prediction on all four datasets (by 1.1 to 6.7 Det-mAP and 1.4 to 6.7 Seg-mAP); with a ConvLSTM encoder, token-wise prediction is ahead on Charades and ROAD-Waymo and within one point on the other two. Its benefit thus extends beyond Mamba-2, to a degree that depends on the temporal module.

W2. The tokenizer, within-frame attention and LSE pooling "have not been adequately evaluated individually", and "the observed performance differences could stem from the use of different pooling operators rather than solely from the position at which spatial aggregation is performed".

To disentangle the pooling operator from the aggregation stage, the revised Table 2 applies mean, max and LSE at both stages. Keeping the tokens until class prediction helps most when the function can select: with LSE, aggregating after class prediction instead of before it gains 2.8 / 3.3 / 1.8 / 1.0 Det-mAP on MultiTHUMOS / Charades / ATARS / ROAD-Waymo, whereas with mean the two stages are close. The individual components are evaluated under Q1.

From Table 2 (Sec. 4.3): each pooling function applied before class prediction, to token features ("Pooled prediction w/ ...", first four rows), or after it, to token class scores ("Mean / Max aggregation"). Change relative to SpaTAD (LSE aggregation).

MultiTHUMOSCharadesATARSROAD-Waymo
VariantΔDetΔSegΔDetΔSegΔDetΔSegΔDetΔSeg
Pooled prediction w/ attention pooling-2.25±0.99-3.70±0.68-6.74±0.38-6.73±0.13-3.35±3.12-2.88±3.59-1.14±0.31-2.21±0.46
Pooled prediction w/ mean pooling-1.06±0.17-1.85±0.41-4.92±0.03-4.53±0.30-0.70±1.05-0.54±1.54-0.83±0.20-1.37±0.41
Pooled prediction w/ LSE pooling-2.83±0.31-2.84±0.41-3.30±0.04-2.55±0.13-1.84±3.44-1.07±4.03-1.00±0.40-1.46±0.38
Pooled prediction w/ max pooling-3.23±0.89-3.08±0.78-3.45±0.28-2.80±0.14-0.09±1.41+1.10±1.64-1.02±0.44-1.78±0.65
Mean aggregation-1.47±1.74-1.92±1.18-5.19±0.26-4.89±0.11-1.86±0.74-1.74±1.78-0.99±0.37-1.77±0.88
Max aggregation-0.47±0.91-0.47±0.41-0.61±0.20-0.43±0.28-1.83±1.92-2.56±3.39+0.10±0.39+0.25±0.66

Δ = variant minus SpaTAD, in mAP points (negative: worse than SpaTAD). Max aggregation is within 0.6 points of SpaTAD on MultiTHUMOS, Charades and ROAD-Waymo, so LSE is not the only effective function. On ATARS the differences among functions are within the variation over runs.

AuthorsOpenReview ↗

Response to Reviewer kn7b (2/2)

Q1. "How does the performance change when (1) using only single-scale tokens instead of the multi-granularity tokenizer, (2) removing the within-frame attention module, and (3) replacing the per-token SSM with alternative temporal modeling modules?"

All three are now in the paper. Each row changes one component of SpaTAD and is compared with SpaTAD under identical settings.

One component changed; change relative to SpaTAD. Rows 1 and 2: from Table 2 (Sec. 4.3); rows 3 and 4: from Table 8 (Appendix E).

MultiTHUMOSCharadesATARSROAD-Waymo
VariantΔDetΔSegΔDetΔSegΔDetΔSegΔDetΔSeg
Single-scale tokens-0.49±0.17-0.40±0.14-0.23±0.31-0.04±0.21-0.48±1.69-0.61±1.87+0.09±0.37+0.03±1.03
No within-frame attention-3.03±0.48-3.46±0.18-3.84±0.20-3.56±0.35-0.38±1.30-0.71±1.33-0.27±0.30-1.92±0.86
GRU temporal module-0.77±0.61-0.85±0.59-0.51±0.49-0.11±0.09-5.98±2.86-7.15±3.91-0.22±0.38-0.79±0.62
Temporal attention module-2.82±0.42-1.86±0.13-1.61±0.18-0.91±0.30-0.79±3.71-1.88±4.76-0.40±0.14-0.54±0.50

Δ = variant minus SpaTAD, in mAP points (negative: worse than SpaTAD).

  • (1) "Single-scale tokens" uses only the fine token grid. The main gains do not rely on multi-granularity tokenization: accuracy changes by at most 0.61 mAP points (Sec. 4.3).
  • (2) "No within-frame attention" removes the attention of every spatial module and keeps its feed-forward network: about 3 to 4 points lower on MultiTHUMOS and Charades, smaller effects on ATARS and ROAD-Waymo.
  • (3) The per-token Mamba-2 is replaced, with the same normalisation and residual connection, by a GRU or by a causal Transformer layer along time, in which each token location attends to its own earlier frames within the training window. Among the evaluated temporal modules, Mamba-2 achieves the highest mean Det-mAP and Seg-mAP on all four datasets; the GRU loses most on ATARS, temporal attention on MultiTHUMOS.

Q2. "Since these states are tied to fixed spatial locations rather than moving objects, how does the model effectively track motion information across locations as objects move?"

SpaTAD propagates spatially structured temporal information without explicit object tracking or identity supervision: each temporal module keeps a history at every image location, and within-frame attention between the temporal modules lets information move across locations (Sec. 3.2). It does not assign identities to objects. The evidence is consistent with this mechanism: removing within-frame attention lowers accuracy (Q1); permuting the recurrent states across locations lowers it most where trajectories define the classes (ATARS; Table 12, Appendix G); and on ROAD-Waymo, recorded from a moving vehicle, SpaTAD's advantage over TriDetPlus is larger on moving-camera frames than on static ones:

From Table 11 (Appendix G): ROAD-Waymo, frame-level Seg-mAP on test frames with a static and with a moving camera.

Test framesFramesSpaTADTriDetPlus w/ patch tokens
Camera static4,45352.68±0.1149.11±0.56
Camera moving18,38147.64±0.4840.71±1.07

"Camera static": the test frames whose ego-vehicle label is "stopped"; "camera moving": all other test frames (a proxy for camera motion). Both evaluated models perform worse on the moving-camera subset.

Thank you for the response. I still have concerns about the limited novelty. However, the additional experiments provide stronger evidence for the effectiveness of the method and address my previous concerns about the experiments. Therefore, I raise my score from 2 to 3.

Reviewer_PgDGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpaTAD, a temporal action detection model that preserves spatial tokens through temporal modeling and class prediction. The method maintains a recurrent state for each spatial location, uses within-frame attention to exchange spatial information, and predicts class scores at individual tokens before aggregating them into frame-level predictions. Experiments on MultiTHUMOS, Charades, ATARS, and ROAD-Waymo show improvements over existing detection methods. Controlled ablations examine the effect of aggregating spatial information at different stages.

Critical strengths
  1. The paper addresses a clear question: how long should spatial structure be preserved in temporal action detection?
  2. The proposed design is well motivated, particularly for concurrent actions and trajectory‑dependent events.
  3. The distinction from ESTF‑SSM and DST‑Mamba is meaningful, as SpaTAD retains spatial information through class prediction rather than only during temporal feature extraction.
  4. The aggregation ablations directly support the paper's main claim and are more convincing than standard component removal experiments.
  5. Performance gains are substantial across four datasets, particularly on ATARS, where spatial trajectories are important.
  6. The paper reports variability across runs and includes additional streaming inference analysis.
Critical weaknesses
  1. Spatial recurrent states are related to ConvLSTM, ConvGRU, and TrajGRU, while token‑level prediction followed by spatial aggregation resembles multiple‑instance learning. The paper should more clearly establish what is new beyond combining these ideas.
  2. The comparisons are not fully matched in representation capacity. Most major baselines receive pooled SigLIP 2 features, whereas SpaTAD uses dense spatial tokens. Although the internal ablations are useful, the cross‑model improvements may partly reflect access to richer features.
  3. The pooling ablation is described inconsistently. Section 4.3 refers to mean pooling, while Appendix E describes learned attention pooling. This makes it difficult to determine the exact experimental control.
  4. The contribution of aggregation order and pooling formulation is not fully separated. SpaTAD uses log‑sum‑exp aggregation after token‑level prediction, while the pooled variants use a different aggregation operation.
  5. The improvement on ROAD‑Waymo Det‑mAP is marginal. SpaTAD achieves 16.29 versus 15.90 for TriDetPlus, suggesting limitations when multiple instances of the same action overlap temporally.
  6. The robustness of location‑bound memory remains unclear. Large object movements and camera motion may weaken the association between spatial locations and the corresponding action history.
Questions for authors
  1. Section 4.3 describes the aggregation ablations as using mean pooling, while Appendix E specifies learned attention pooling. Which implementation was used for Table 2? Could the authors clarify whether the conclusions hold under both pooling strategies?
  2. In Table 1, most competing detectors use pooled SigLIP 2 frame embeddings, while SpaTAD receives dense spatial tokens. Could the authors provide a stronger baseline that preserves spatial tokens but uses an existing TAD architecture, to better separate the benefits of dense input features from the proposed modeling strategy?
  3. SpaTAD combines late spatial aggregation with log‑sum‑exp pooling. How much of the improvement comes from delaying aggregation rather than from the pooling function itself? Could the authors compare alternative aggregation functions under otherwise identical settings?
  4. On ROAD‑Waymo, the Det‑mAP improvement over TriDetPlus is only 0.39 points, despite a larger Seg‑mAP gain. How frequently does the current decoder merge overlapping same‑class action instances, and how much does this limitation affect detection performance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PgDG (1/2)

Thank you for the detailed and constructive review. Our answers follow point by point. Numbers are test-set means ± standard deviations over runs; our general response summarises the revision.

W1. "Spatial recurrent states are related to ConvLSTM, ConvGRU, and TrajGRU, while token-level prediction followed by spatial aggregation resembles multiple-instance learning": what is new beyond combining these ideas?

Our contribution is to investigate, and to isolate experimentally, how the stage of spatial aggregation affects temporal action detection: SpaTAD keeps location-bound temporal representations until class prediction, whereas detectors typically operate on one feature vector per frame. ConvLSTM and ConvGRU (spatially structured recurrent states), TrajGRU (motion-adaptive recurrent connections) and MIL (aggregation of instance-level evidence) are the relevant precedents for the building blocks (Sec. 2); here MIL is applied over the spatial tokens of a frame inside a fully supervised detector, not over time under weak labels. Two findings follow. First, keeping the spatial structure until class prediction improves accuracy; the advantage of this late aggregation is clearest with LSE, while max shows a similar overall trend (W4 / Q3). Second, the benefit is not specific to Mamba-2:

Table 3 (Sec. 4.3): token-wise vs. pooled prediction across temporal modules. All rows receive the same patch tokens (the ConvLSTM on the fine token grid); the two rows of a module differ only in whether the tokens are kept until class prediction or pooled by learned attention just before it. Det-mAP / Seg-mAP, mean ± std, under the shared ablation settings (Appendix E; levels not comparable with Table 1).

MultiTHUMOSCharadesATARSROAD-Waymo
Temporal moduleDetSegDetSegDetSegDetSeg
Mamba-2 (SpaTAD)36.03±0.7351.08±0.3834.09±0.0238.80±0.3524.18±1.8141.07±2.1417.41±0.1748.93±0.07
Mamba-2 w/ pooled prediction33.79±0.2647.38±0.6327.35±0.3632.06±0.2220.83±2.4038.19±3.8216.27±0.4246.72±0.47
GRU35.49±0.2950.36±0.2033.58±0.4738.69±0.2617.77±2.5834.06±3.6917.15±0.3147.94±0.10
GRU w/ pooled prediction31.50±0.5845.82±0.5828.23±0.1133.53±0.1916.14±1.0831.84±0.7816.08±0.5546.52±1.12
Temporal attention33.29±0.1849.21±0.3332.48±0.1637.89±0.0422.72±3.2639.06±4.2816.95±0.1548.27±0.31
Temporal attention w/ pooled prediction28.81±0.6044.24±1.1526.14±0.2631.93±0.0718.79±1.4435.29±3.0315.75±0.1446.12±0.10
ConvLSTM27.22±0.3844.50±0.1727.54±0.1731.85±0.0420.43±0.4439.33±0.5716.98±0.1447.64±0.64
ConvLSTM w/ pooled prediction28.20±0.6543.56±0.4025.04±0.2230.30±0.1521.12±0.5840.14±0.3315.57±0.0546.32±0.21

With Mamba-2, a GRU or temporal attention, token-wise prediction is ahead of pooled prediction on all four datasets (by 1.1 to 6.7 Det-mAP and 1.4 to 6.7 Seg-mAP). With a ConvLSTM encoder it is ahead on Charades and ROAD-Waymo and within one point on the other two; this encoder is also below SpaTAD on all four datasets.

W2 / Q2. Most baselines receive pooled features: "Could the authors provide a stronger baseline that preserves spatial tokens but uses an existing TAD architecture?"

We address this with two complementary controls. (i) Existing TAD architectures with the tokens (Table 1): their temporal models take one vector per frame, so we adapted each detector with a jointly trained attention-pooling adapter over the patch tokens. The tokens are thus used but pooled before temporal modelling. Under this matched comparison, SpaTAD achieves the highest mean Det-mAP and Seg-mAP on all four datasets, with 21 to 32% fewer trainable parameters than TriDetPlus w/ patch tokens. (ii) Tokens preserved or not, inside one model (Table 2): "Global temporal state" applies learned attention pooling before temporal modelling, "Pooled prediction" applies it after temporal modelling, just before class prediction; everything else is unchanged. Keeping the tokens until class prediction is better than either: both controls lower Det-mAP and Seg-mAP on all four datasets (the pooled-prediction rows are quoted under W4 / Q3).

W3 / Q1. Mean pooling (Sec. 4.3) or learned attention pooling (Appendix E): "Which implementation was used for Table 2? Could the authors clarify whether the conclusions hold under both pooling strategies?"

Thank you for identifying this inconsistency. Table 2 used learned attention pooling, as Appendix E states; "mean pooling" in Sec. 4.3 was an incorrect description and has been corrected. The conclusion holds under both strategies: pooled prediction is below SpaTAD with attention pooling and with mean pooling (rows "w/ attention pooling" and "w/ mean pooling" of the table under W4 / Q3, part 2/2).

AuthorsOpenReview ↗

Response to Reviewer PgDG (2/2)

W4 / Q3. "How much of the improvement comes from delaying aggregation rather than from the pooling function itself?"

Keeping the tokens until class prediction helps, and the benefit is largest when the aggregation function can select the salient token-level evidence. To disentangle operator and stage, the revised Table 2 applies mean, max and LSE at both stages. With LSE, aggregating after class prediction instead of before it gains 2.8 / 3.3 / 1.8 / 1.0 Det-mAP on MultiTHUMOS / Charades / ATARS / ROAD-Waymo; with mean, the two stages are close.

From Table 2 (Sec. 4.3): each pooling function applied before class prediction, to token features ("Pooled prediction w/ ...", first four rows), or after it, to token class scores ("Mean / Max aggregation"). Change relative to SpaTAD (LSE aggregation).

MultiTHUMOSCharadesATARSROAD-Waymo
VariantΔDetΔSegΔDetΔSegΔDetΔSegΔDetΔSeg
Pooled prediction w/ attention pooling-2.25±0.99-3.70±0.68-6.74±0.38-6.73±0.13-3.35±3.12-2.88±3.59-1.14±0.31-2.21±0.46
Pooled prediction w/ mean pooling-1.06±0.17-1.85±0.41-4.92±0.03-4.53±0.30-0.70±1.05-0.54±1.54-0.83±0.20-1.37±0.41
Pooled prediction w/ LSE pooling-2.83±0.31-2.84±0.41-3.30±0.04-2.55±0.13-1.84±3.44-1.07±4.03-1.00±0.40-1.46±0.38
Pooled prediction w/ max pooling-3.23±0.89-3.08±0.78-3.45±0.28-2.80±0.14-0.09±1.41+1.10±1.64-1.02±0.44-1.78±0.65
Mean aggregation-1.47±1.74-1.92±1.18-5.19±0.26-4.89±0.11-1.86±0.74-1.74±1.78-0.99±0.37-1.77±0.88
Max aggregation-0.47±0.91-0.47±0.41-0.61±0.20-0.43±0.28-1.83±1.92-2.56±3.39+0.10±0.39+0.25±0.66

Δ = variant minus SpaTAD, in mAP points (negative: worse than SpaTAD). Max aggregation is within 0.6 points of SpaTAD on MultiTHUMOS, Charades and ROAD-Waymo, so LSE is not the only effective function. On ATARS the differences among functions are within the variation over runs.

W5 / Q4. "How frequently does the current decoder merge overlapping same-class action instances, and how much does this limitation affect detection performance?"

ROAD-Waymo exposes a limit of class-level temporal detection: several agents perform the same action at once, and a detector outputs class, start and end, without assigning segments to agents. On the test set, SpaTAD merges 4.5% of the instances with a concurrent one, against 20.9% for TriDetPlus w/ patch tokens.

ROAD-Waymo test set: scores (Table 1) and merging analysis (Table 10, Appendix G).

ROAD-WaymoSpaTADTriDetPlus w/ patch tokens
Det-mAP17.62±0.0416.43±0.68
Seg-mAP49.87±0.6842.75±0.69
Merged instances (%)4.5±0.220.9±0.2
Det-mAP gain of oracle duplication (diagnostic)3.6±0.18.2±0.3

92% of the test instances overlap in time with another instance of their class. An instance counts as merged if it is missed although a predicted segment covers it (tIoU of at least 0.5), i.e. that segment is already matched to a concurrent instance. The last row is a diagnostic that uses the ground truth: each predicted segment is duplicated for every further concurrent instance it covers. Resolution is a second factor: 76% of the annotated agents are smaller than one patch token, and with a finer token grid (19x29 instead of 13x19) SpaTAD improves from 17.67 / 50.18 to 18.15 / 50.81 Det-mAP / Seg-mAP, the baseline from 15.91 / 41.92 to 16.05 / 41.79 (one run each; Appendix G).

W6. "Large object movements and camera motion may weaken the association between spatial locations and the corresponding action history."

This is a legitimate concern for any representation indexed by image location: under camera motion, a fixed location no longer follows an object. In SpaTAD, within-frame attention between the temporal modules lets locations exchange what they have accumulated, so the history is not confined to one location, and the measurements indicate that its advantage persists under camera motion.

From Table 11 (Appendix G): ROAD-Waymo, frame-level Seg-mAP on test frames with a static and with a moving camera.

Test framesFramesSpaTADTriDetPlus w/ patch tokens
Camera static4,45352.68±0.1149.11±0.56
Camera moving18,38147.64±0.4840.71±1.07

"Camera static": the test frames whose ego-vehicle label is "stopped"; "camera moving": all other test frames (a proxy for camera motion). Both models are lower on the moving-camera frames, and SpaTAD's advantage is larger there (6.9 against 3.6 Seg-mAP). The state-permutation test (Table 12, Appendix G) agrees: the binding of states to locations matters most with a fixed camera (ATARS) and least on ROAD-Waymo.

Thanks for the response. The new experiments help clear up most of my concerns, especially the ones about pooling and the temporal modules. I still have some concerns about the novelty and how the method handles complex motion and overlapping actions, but overall, I think the response is reasonable. I'll keep my rating.

Reviewer_YeziOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper preserves spatial tokens until the action prediction stage and maintains an independent Mamba-2 temporal state for each spatial location. The research question is clear, and the method is evaluated on MultiTHUMOS, Charades, ATARS, and ROAD-Waymo, with improvements reported across all four datasets. The central concern is that SpaTAD uses approximately 336 dense SigLIP2 patch tokens, whereas most baselines use pooled frame embeddings. Therefore, it is unclear whether the gains come from the proposed spatial-temporal modeling or simply from using a stronger spatially detailed input representation.

Critical strengths

A useful design question supported by consistent internal evidence. Table 2 reports paired changes across four datasets: pooling before class prediction reduces both metrics throughout, while the additional effect of removing spatially distributed temporal states depends on the dataset.

Critical weaknesses

The paper lacks strong baselines using the same dense tokens, such as ActionFormer, TriDet, ConvLSTM, or a simple Mamba model. The description of the pooling controls in Table 2 is also inconsistent: the main text refers to mean pooling, while the appendix describes learned attention pooling. In addition, the causal claim is too strong because end-offset refinement uses future-frame information. The model may also have difficulty representing multiple concurrent instances of the same class, and the conversion between different output formats and evaluation metrics may introduce additional bias.

Questions for authors

Does the advantage remain with class-specific attention pooling before classification, using comparable capacity and the same losses and decoder?

Evaluation clarification: How many independent source clips underlie the ATARS test set after grouping geometric variants?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Yezi (1/2)

Thank you for the careful and constructive review. Our answers follow point by point. Numbers are test-set means ± standard deviations over runs; our general response summarises the revision.

W1 (first part). "It is unclear whether the gains come from the proposed spatial-temporal modeling or simply from using a stronger spatially detailed input"; baselines such as ActionFormer and TriDet should use the same dense tokens.

You raise an important question: do richer spatial features alone explain the performance differences? To answer it, we evaluated ActionFormer, TriDet and TriDetPlus with the same SigLIP 2 patch tokens as SpaTAD, keeping their temporal architectures and adding a jointly trained attention-pooling adapter. Under this matched comparison, SpaTAD achieves the highest mean Det-mAP and Seg-mAP on all four datasets (on average +2.4 Det-mAP and +5.9 Seg-mAP over the best baseline of each dataset, which for Seg-mAP on ROAD-Waymo is MS-TCT w/ patch tokens; Table 1).

From Table 1 (Sec. 4.2): the detectors you name and TriDetPlus, each adapted to read the same patch tokens as SpaTAD. Det-mAP / Seg-mAP; Avg. = mean over the four datasets.

MultiTHUMOSCharadesATARSROAD-WaymoAvg.
MethodDetSegDetSegDetSegDetSegDetSeg
TriDetPlus w/ patch tokens38.28±0.0349.80±0.1634.10±0.1440.69±0.3315.86±0.5726.58±0.8516.43±0.6842.75±0.6926.1739.96
TriDet w/ patch tokens36.63±0.6049.03±0.7231.71±0.2939.00±0.3214.25±0.4724.66±0.3812.49±0.2636.79±0.4623.7737.37
ActionFormer w/ patch tokens33.78±0.4447.20±0.7132.11±0.1639.50±0.1814.05±0.3225.38±0.6412.35±0.0636.48±0.4223.0737.14
SpaTAD (ours)38.42±0.2253.16±0.1634.24±0.1541.50±0.1423.81±0.5042.76±0.6417.62±0.0449.87±0.6828.5246.82

The adapter is an added input-processing stage with its own trainable parameters and computation; SpaTAD reaches these results with 21 to 32% fewer trainable parameters than TriDetPlus w/ patch tokens (14 to 23 M against 21 to 29 M). With the same tokens inside one model, pooling the spatial structure earlier lowers accuracy (next point and Q1).

W1 (second part). The baselines should include simple models on the same dense tokens: "ConvLSTM, or a simple Mamba model".

Both were trained on the same patch tokens, and both remain below SpaTAD on all four datasets. Neither is a detector by itself, so each replaces SpaTAD's encoder and keeps its head, losses and decoder.

Change relative to SpaTAD; rows from Table 8 (Appendix E) and Table 2 (Sec. 4.3).

MultiTHUMOSCharadesATARSROAD-Waymo
VariantΔDetΔSegΔDetΔSegΔDetΔSegΔDetΔSeg
ConvLSTM encoder-8.81±0.44-6.58±0.31-6.57±0.13-7.17±0.09-3.75±1.38-1.74±1.62-0.49±0.31-1.39±0.72
Global temporal state w/ attention pooling-3.29±0.62-3.53±0.79-5.90±0.59-5.45±0.81-13.00±1.64-17.23±2.28-1.58±0.27-2.32±0.20

Δ = variant minus SpaTAD, in mAP points (negative: worse than SpaTAD). "ConvLSTM encoder" is an existing spatially structured recurrent model: three ConvLSTM layers on the fine token grid, with about SpaTAD's parameter count. "Global temporal state" is the simple Mamba model: learned attention pooling reduces the tokens of each frame to one vector before temporal modelling, and the same Mamba-2 stack runs over the frame sequence, so no per-token temporal state remains. It loses most on ATARS, where several agents act at once.

W2. "The main text refers to mean pooling, while the appendix describes learned attention pooling" (Table 2).

Thank you for identifying this inconsistency. The experiment used learned attention pooling, as the appendix states; "mean pooling" in Sec. 4.3 was an incorrect description and has been corrected. We also ran the mean-pooling control, now a sub-row of Table 2.

From Table 2 (Sec. 4.3): the token features pooled before class prediction. Change relative to SpaTAD.

MultiTHUMOSCharadesATARSROAD-Waymo
VariantΔDetΔSegΔDetΔSegΔDetΔSegΔDetΔSeg
Pooled prediction w/ attention pooling-2.25±0.99-3.70±0.68-6.74±0.38-6.73±0.13-3.35±3.12-2.88±3.59-1.14±0.31-2.21±0.46
Pooled prediction w/ mean pooling-1.06±0.17-1.85±0.41-4.92±0.03-4.53±0.30-0.70±1.05-0.54±1.54-0.83±0.20-1.37±0.41

Pooled prediction is below SpaTAD with either function.

W3. "The causal claim is too strong because end-offset refinement uses future-frame information."

You are right that the end-offset refinement uses future frames; it belongs to segment decoding. The causal claim covers the temporal modelling and the frame-level predictions it outputs: at time t they depend only on frames up to t (Sec. 3.3).

AuthorsOpenReview ↗

Response to Reviewer Yezi (2/2)

W4. "The model may also have difficulty representing multiple concurrent instances of the same class."

Temporal action detection outputs class-level segments, not agent identities, so concurrent instances of one class can be merged when their start and end times cannot be told apart; this is inherent to class-level detection rather than specific to SpaTAD. The revised paper quantifies it (Table 10, Appendix G): on ROAD-Waymo, 4.5±0.2% of the test instances are merged for SpaTAD, against 20.9±0.2% for TriDetPlus w/ patch tokens. The case is frequent only on ROAD-Waymo: 92% of its test instances overlap in time with another instance of their class (0 to 4% on the other datasets). An instance counts as merged if it is missed although a predicted segment covers it (tIoU of at least 0.5), i.e. that segment is already matched to a concurrent instance. Resolution is a second factor: 76% of the annotated agents are smaller than one patch token, and a finer token grid (19x29 instead of 13x19) improves SpaTAD from 17.67 / 50.18 to 18.15 / 50.81 Det-mAP / Seg-mAP (one run; Appendix G), at a higher cost.

W5. "The conversion between different output formats and evaluation metrics may introduce additional bias."

We checked this directly. The dense baselines output frame scores only, so their segments have to be derived: the frame scores are thresholded (threshold selected on validation) and consecutive positive frames form one segment. SpaTAD needs no conversion, because it predicts frame scores and segments natively. To rule out that the conversion rule is what separates the methods, we applied the same rule to SpaTAD's frame scores: SpaTAD remains ahead of every dense baseline (Appendix D, paragraph "Conversion check").

Det-mAP with segments obtained by thresholding the frame scores; Seg-mAP as in Table 1.

MultiTHUMOSCharadesATARSROAD-Waymo
ModelDetSegDetSegDetSegDetSeg
SpaTAD w/ frame-score thresholding27.17±0.2653.16±0.1623.81±0.1641.50±0.1415.99±1.3242.76±0.646.58±0.1049.87±0.68
MS-TCT w/ patch tokens15.01±1.4241.90±1.2517.17±0.2737.54±0.397.73±0.8426.00±0.585.68±0.2046.46±0.06
MS-Temba w/ patch tokens13.76±2.0241.43±0.8610.83±0.2031.41±0.315.53±1.1523.37±1.955.31±0.1544.82±0.46
Action-slot (patch tokens by design)8.05±0.8726.49±1.668.36±0.2622.94±0.353.25±3.5811.24±5.505.42±0.4244.00±0.88

The advantage over the dense baselines is therefore not produced by the conversion, and in Seg-mAP no conversion is involved for these methods. Segment-based detectors are converted to frame scores by one fixed rule for all methods (Appendix D).

Q1. "Does the advantage remain with class-specific attention pooling before classification, using comparable capacity and the same losses and decoder?"

Yes, the advantage remains. In this control, SpaTAD's token-wise prediction is replaced by a pooled prediction with class-specific learnable queries: each class has its own query, which attends over the tokens of a frame and pools them into one vector for that class. The encoder, head, losses and decoder are unchanged, and the capacity is comparable (17.1 M against 16.0 M parameters on MultiTHUMOS). The variant remains below SpaTAD on all four datasets.

From Table 2 (Sec. 4.3): pooling before classification, with one query for all classes or one query per class. Change relative to SpaTAD.

MultiTHUMOSCharadesATARSROAD-Waymo
VariantΔDetΔSegΔDetΔSegΔDetΔSegΔDetΔSeg
Pooled prediction w/ attention pooling-2.25±0.99-3.70±0.68-6.74±0.38-6.73±0.13-3.35±3.12-2.88±3.59-1.14±0.31-2.21±0.46
Pooled prediction w/ class-specific attention pooling-3.13±0.58-4.06±0.54-2.41±1.20-2.63±0.99-3.76±1.23-3.94±0.45-0.86±0.06-1.99±0.78

Δ = variant minus SpaTAD, in mAP points (negative: worse than SpaTAD). The class-specific readout narrows the gap on Charades, but hardly elsewhere. The takeaway: selecting features per class before classification is not equivalent to predicting classes at every token. There the selection is made before any location has a class score; in SpaTAD the class scores themselves select across space. Action-slot, a baseline that reads the patch tokens with one attention slot per class by design, is also well below SpaTAD (Table 1).

Q2. "How many independent source clips underlie the ATARS test set after grouping geometric variants?"

Six. The 48 test videos are six source recordings, each in the eight geometric variants that the dataset itself provides (four rotations, each also mirrored). The splits are grouped by source recording, so no variant of a test recording occurs in the training or validation data, and the test intersection occurs in neither (Sec. 4.1).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: additional experiments and revisions

We thank all reviewers for their careful and constructive comments; each review is answered point by point in its own reply. Two points concern all reviews.

1. Stronger baselines with matched patch-token access. To examine the role of the spatial input directly, as Reviewers Yezi and PgDG asked, we adapted the existing detectors to read the same frozen SigLIP 2 patch tokens as SpaTAD. Their temporal architectures are preserved; a jointly trained attention-pooling adapter maps the patch tokens of each frame to the frame-level representation they expect (Action-slot takes patch tokens by design). On average, the adapted baselines outperform their counterparts on pooled SigLIP 2 features, at the cost of an additional pooling stage with its own parameters and computation. Even so, SpaTAD achieves the highest mean Det-mAP and Seg-mAP on all four datasets (Table 1; on average +2.4 Det-mAP and +5.9 Seg-mAP over the best baseline of each dataset), with 21 to 32% fewer trainable parameters than the strongest adapted detector, TriDetPlus. The controlled ablations (Tables 2 and 3) show that keeping the spatial structure until class prediction improves accuracy.

2. Settings of SpaTAD. In the same round, SpaTAD's training and inference settings were tuned per dataset on the validation splits (Appendices B and C; on MultiTHUMOS this includes token dropout as a regulariser); its architecture and parameter count are unchanged. The Table 2 ablations were re-run under the shared protocol in Appendix E; the conclusions remain unchanged. For every model, the checkpoint and the decoding settings are selected on the validation split, and the test split is evaluated once.

Numbering. Section numbers, Tables 1 and 2, and Appendices A to F are unchanged; Table 3 (Sec. 4.3) and Appendix G (Limitations) are new.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.