- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the SpaTAD framework for temporal action detection, which aims to retain localized action evidence and spatially dependent motion histories that may be lost through early spatial pooling aggregation. There are three parts in SpaTAD. First, multi-granularity tokenizer is designed to preserve spatial structure at different granularities. Second, spatial-temporal encoder with per-token temporal memory focuses on preserving spatial structure during temporal modeling. Third, token-wise action prediction further preserves spatial structure by predicting actions at individual spatial tokens before spatial aggregation. SpaTAD achieves SOTA results on four datasets. In the ablation experiment part, the authors further investigate the impact of spatial aggregation position and demonstrate that pooling before class prediction consistently degrades performance.
Critical strengths
- SpaTAD achieves SOTA results over baselines across four datasets.
- The ablation confirms that pooling before class prediction consistently degrades performance across all four datasets, and preserving location-bound memory provides additional benefits
Critical weaknesses
- SpaTAD mainly combines existing techniques, including spatial-temporal modeling, self-attention, and token-wise prediction combined with spatial pooling. Thus, its novelty seems limited, since its primary contribution is the integration of established components rather than the development of a fundamentally new approach.
- The contributions of the multi-granularity tokenizer, within-frame attention, and LSE pooling have not been adequately evaluated individually. Furthermore, the observed performance differences could stem from the use of different pooling operators rather than solely from the position at which spatial aggregation is performed.
Questions for authors
- Additional ablation studies are needed to better evaluate the contributions of the key components. Specifically, how does the performance change when (1) using only single-scale tokens instead of the multi-granularity tokenizer, (2) removing the within-frame attention module, and (3) replacing the per-token SSM with alternative temporal modeling modules?
- The authors argue that per-token recurrent states help preserve motion histories. However, since these states are tied to fixed spatial locations rather than moving objects, how does the model effectively track motion information across locations as objects move?
3 comments
Response to Reviewer kn7b (2/2)
Q1. "How does the performance change when (1) using only single-scale tokens instead of the multi-granularity tokenizer, (2) removing the within-frame attention module, and (3) replacing the per-token SSM with alternative temporal modeling modules?"
All three are now in the paper. Each row changes one component of SpaTAD and is compared with SpaTAD under identical settings.
One component changed; change relative to SpaTAD. Rows 1 and 2: from Table 2 (Sec. 4.3); rows 3 and 4: from Table 8 (Appendix E).
| MultiTHUMOS | Charades | ATARS | ROAD-Waymo | |||||
|---|---|---|---|---|---|---|---|---|
| Variant | ΔDet | ΔSeg | ΔDet | ΔSeg | ΔDet | ΔSeg | ΔDet | ΔSeg |
| Single-scale tokens | -0.49±0.17 | -0.40±0.14 | -0.23±0.31 | -0.04±0.21 | -0.48±1.69 | -0.61±1.87 | +0.09±0.37 | +0.03±1.03 |
| No within-frame attention | -3.03±0.48 | -3.46±0.18 | -3.84±0.20 | -3.56±0.35 | -0.38±1.30 | -0.71±1.33 | -0.27±0.30 | -1.92±0.86 |
| GRU temporal module | -0.77±0.61 | -0.85±0.59 | -0.51±0.49 | -0.11±0.09 | -5.98±2.86 | -7.15±3.91 | -0.22±0.38 | -0.79±0.62 |
| Temporal attention module | -2.82±0.42 | -1.86±0.13 | -1.61±0.18 | -0.91±0.30 | -0.79±3.71 | -1.88±4.76 | -0.40±0.14 | -0.54±0.50 |
Δ = variant minus SpaTAD, in mAP points (negative: worse than SpaTAD).
- (1) "Single-scale tokens" uses only the fine token grid. The main gains do not rely on multi-granularity tokenization: accuracy changes by at most 0.61 mAP points (Sec. 4.3).
- (2) "No within-frame attention" removes the attention of every spatial module and keeps its feed-forward network: about 3 to 4 points lower on MultiTHUMOS and Charades, smaller effects on ATARS and ROAD-Waymo.
- (3) The per-token Mamba-2 is replaced, with the same normalisation and residual connection, by a GRU or by a causal Transformer layer along time, in which each token location attends to its own earlier frames within the training window. Among the evaluated temporal modules, Mamba-2 achieves the highest mean Det-mAP and Seg-mAP on all four datasets; the GRU loses most on ATARS, temporal attention on MultiTHUMOS.
Q2. "Since these states are tied to fixed spatial locations rather than moving objects, how does the model effectively track motion information across locations as objects move?"
SpaTAD propagates spatially structured temporal information without explicit object tracking or identity supervision: each temporal module keeps a history at every image location, and within-frame attention between the temporal modules lets information move across locations (Sec. 3.2). It does not assign identities to objects. The evidence is consistent with this mechanism: removing within-frame attention lowers accuracy (Q1); permuting the recurrent states across locations lowers it most where trajectories define the classes (ATARS; Table 12, Appendix G); and on ROAD-Waymo, recorded from a moving vehicle, SpaTAD's advantage over TriDetPlus is larger on moving-camera frames than on static ones:
From Table 11 (Appendix G): ROAD-Waymo, frame-level Seg-mAP on test frames with a static and with a moving camera.
| Test frames | Frames | SpaTAD | TriDetPlus w/ patch tokens |
|---|---|---|---|
| Camera static | 4,453 | 52.68±0.11 | 49.11±0.56 |
| Camera moving | 18,381 | 47.64±0.48 | 40.71±1.07 |
"Camera static": the test frames whose ego-vehicle label is "stopped"; "camera moving": all other test frames (a proxy for camera motion). Both evaluated models perform worse on the moving-camera subset.
Thank you for the response. I still have concerns about the limited novelty. However, the additional experiments provide stronger evidence for the effectiveness of the method and address my previous concerns about the experiments. Therefore, I raise my score from 2 to 3.
Response to Reviewer kn7b (1/2)
Thank you for the review. Our answers follow point by point. Numbers are test-set means ± standard deviations over runs; our general response summarises the revision.
W1. "Its primary contribution is the integration of established components rather than the development of a fundamentally new approach."
Our contribution is a design principle and its experimental study: how far into a temporal action detector the spatial structure should be kept. Temporal action detectors typically reduce each frame to one feature vector before temporal modelling; to our knowledge, SpaTAD is the first to keep the spatial structure through temporal modelling and class prediction, and to measure what this contributes. It builds on established components; its contribution is to isolate the stage of spatial aggregation and to examine its effect across temporal architectures.
Table 3 (Sec. 4.3): token-wise vs. pooled prediction across temporal modules. All rows receive the same patch tokens (the ConvLSTM on the fine token grid); the two rows of a module differ only in whether the tokens are kept until class prediction or pooled by learned attention just before it. Det-mAP / Seg-mAP, mean ± std, under the shared ablation settings (Appendix E; levels not comparable with Table 1).
Keeping the tokens through temporal modelling is not enough: the pooled-prediction rows do so, yet with Mamba-2, a GRU or temporal attention they trail token-wise prediction on all four datasets (by 1.1 to 6.7 Det-mAP and 1.4 to 6.7 Seg-mAP); with a ConvLSTM encoder, token-wise prediction is ahead on Charades and ROAD-Waymo and within one point on the other two. Its benefit thus extends beyond Mamba-2, to a degree that depends on the temporal module.
W2. The tokenizer, within-frame attention and LSE pooling "have not been adequately evaluated individually", and "the observed performance differences could stem from the use of different pooling operators rather than solely from the position at which spatial aggregation is performed".
To disentangle the pooling operator from the aggregation stage, the revised Table 2 applies mean, max and LSE at both stages. Keeping the tokens until class prediction helps most when the function can select: with LSE, aggregating after class prediction instead of before it gains 2.8 / 3.3 / 1.8 / 1.0 Det-mAP on MultiTHUMOS / Charades / ATARS / ROAD-Waymo, whereas with mean the two stages are close. The individual components are evaluated under Q1.
From Table 2 (Sec. 4.3): each pooling function applied before class prediction, to token features ("Pooled prediction w/ ...", first four rows), or after it, to token class scores ("Mean / Max aggregation"). Change relative to SpaTAD (LSE aggregation).
Δ = variant minus SpaTAD, in mAP points (negative: worse than SpaTAD). Max aggregation is within 0.6 points of SpaTAD on MultiTHUMOS, Charades and ROAD-Waymo, so LSE is not the only effective function. On ATARS the differences among functions are within the variation over runs.