- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Seam-DETR improves query-based temporal action detection through Transition-Aware Memory (TAM), which injects temporal change features into localization memory, and Match-Aware Ranking (MAR), which relaxes suppression of unmatched high-overlap predictions and ranks proposals by localization quality.
Critical strengths
-
TAM uses simple temporal filters and lightweight adapters, while MAR introduces no additional inference parameters or computation. Improvements on both TadTR and TE-TAD support the applicability of the approach beyond its primary detector.
-
This method improves average mAP over the reported DiGIT baseline by 1.7, 0.6, and 1.1 percentage points across three benchmarks.
Critical weaknesses
After reading this manuscript, I have some concerns regarding the methodology and experiment.
First, the ranking loss can have zero gradients despite an incorrect ordering. With the default delta=0.5, consider two proposals with IoUs 0.9 and 0.6, but scores 0.1 and 0.9: Eqs. (8) to (10) produce a positive sorting penalty, yet ordinary differentiation gives zero score gradients because the comparison functions are saturated. The original Rank & Sort method uses a custom Identity Update, whereas this manuscript does not specify whether that update or ordinary automatic differentiation is used [1]. Under ordinary differentiation, the sorting term cannot directly correct this severe inversion.
The default thresholds leave some false positives completely unconstrained by the relevant score losses. For an unmatched proposal with class-specific IoU 0.45, the default lambda=0.4 removes its negative focal loss, while tau_p=0.5 and tau_n=0.3 exclude it from both ranking sets. Its class score therefore receives no direct supervision from these terms and can remain high, although the proposal is a false positive at evaluation thresholds of 0.5 and above. This is a concrete gap in the claimed quality-aware supervision; the authors should quantify the scores and false-positive contribution of proposals in this interval.
The stated regression objective is either incomplete or materially different from the baseline. Equation (2) specifies 1-tIoU, which provides zero direct boundary-regression gradient when a matched prediction and its ground truth are strictly disjoint. However, the original DiGIT specifies GIoU and a log-ratio distance loss [2],
Also, in the methodology part, the “transition blindness” explanation is not convincingly established. Figure 3 shows that memory features differ across boundaries, but this does not demonstrate that the decoder cannot exploit those differences; the relationship between transition duration and recall in Figure 1 also leaves action duration and category uncontrolled. Although Table 7 reports a meaningful recall improvement from 0.38 to 0.46 for gradual boundaries, comparisons against ordinary temporal differences or temporal convolutions with comparable receptive fields are missing. Consequently, the evidence does not clearly distinguish the proposed mechanism from generic temporal feature enhancement.
[1] Oksuz, K., Cam, B.C., Akbas, E. and Kalkan, S., 2021, October. Rank & sort loss for object detection and instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 3009–3018). IEEE. [2] Kim, H.J., Lee, Y., Hong, J.H. and Lee, S.W., 2025, June. DiGIT: Multi-dilated gated encoder and central-adjacent region integrated decoder for temporal action detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24286–24296). IEEE.
In your experiment, the confidence-mismatch diagnosis suffers from selection bias. Figure 4 partitions predictions using test-time Hungarian matching and then compares their scores, even though the DiGIT matching cost already incorporates classification scores [1]. Higher confidence among matched predictions therefore partly reflects the grouping procedure and does not independently establish harmful ranking errors caused by one-to-one supervision. The paper should quantify how often lower-IoU predictions outrank better predictions for the same instance and how this affects post-NMS AP; increasing duplicate scores alone is insufficient evidence of better ranking.
I also find some issues in the baseline comparison, where additional training and baseline discrepancies are insufficiently controlled. The reproduced DiGIT baseline scores 72.8 in Table 3, versus 73.8 in Table 1, without adequately explaining the discrepancy. Also, some experiments give MAR an additional 20 training epochs but do not report a control that continues training TAM with the original detection loss under the same learning rate and schedule. Without this control, the contribution of MAR cannot be cleanly separated from the benefit of additional optimization.
Furthermore, comparisons with the most relevant supervision methods are incomplete. Table 9 transfers only the Align-DETR loss while retaining one-to-one matching, excluding the original method’s many-to-one supervision of intermediate layers; Rank-DETR’s ranking-oriented loss and matching designs are also absent from the experimental comparison [2,3]. These results establish an advantage over two particular loss substitutions, but they do not adequately demonstrate superiority over existing quality-aware supervision strategies.
In the experiment of the ablation study, the reliability of the reported improvements is insufficiently established. Tables 1 and 2 show average mAP gains of 1.7, 0.6, and 1.1 percentage points over DiGIT using the same features, but provide neither results across multiple random seeds nor confidence intervals for mAP. The error bars in Figure 1 describe diagnostic statistics and do not establish robustness to training variability. In particular, differences such as the 0.2-point gap between fixed and learnable filters are insufficient to support reliable conclusions about design superiority.
[1] Kim, H.J., Lee, Y., Hong, J.H. and Lee, S.W., 2025, June. DiGIT: Multi-dilated gated encoder and central-adjacent region integrated decoder for temporal action detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24286–24296).
[2] Pu, Y., Liang, W., Hao, Y., Yuan, Y., Yang, Y., Zhang, C., Hu, H. and Huang, G., 2023. Rank-DETR for high quality object detection. In Advances in Neural Information Processing Systems, 36 (pp. 16100–16113).
[3] Cai, Z., Liu, S., Wang, G., Li, Z., Ge, Z., Zhang, X. and Huang, D., 2024, November. Align-DETR: Enhancing end-to-end object detection with aligned loss. In Proceedings of the 35th British Machine Vision Conference.
Questions for authors
For the methodology part:
-
What is the substantive methodological advance beyond combining fixed temporal filters, an ignore rule, and existing rank-and-sort objectives? Does the relationship between transition duration and recall persist after controlling for action duration and category?
-
Does TAM outperform ordinary temporal differences or temporal convolutions with comparable receptive fields and parameter budgets?
-
Can confidence mismatch be demonstrated using score-independent proposal grouping, avoiding selection bias from classification-aware Hungarian matching?
-
How often does MAR correct within-instance IoU ranking errors, and how much do those corrections improve post-NMS AP?
For the experimental part:
-
What happens when TAM receives the same additional 20 epochs and learning-rate schedule as MAR, using only the original detection loss?
-
Why do the reproduced DiGIT and TadTR baselines score 72.8 and 64.7, versus 73.8 and 67.3 in Table 1?
-
Under matched training budgets, how does MAR compare with Align-DETR including its intermediate-layer matching strategy, and with Rank-DETR’s loss and matching designs?
-
Are the main improvements and small ablation differences reproducible across random seeds? The authors should report mean and standard deviation for the key comparisons.
-
Which data splits were used to select TAM’s filter scales and MAR’s hyperparameters, and were evaluation annotations involved in those choices?
-
How sensitive is MAR to the transition between training stages, and what justifies changing from a hard ignore rule on THUMOS14 to a soft rule on the other datasets?
6 comments
W5 / Methodology Q3 and Q4. Selection bias in the confidence-mismatch diagnosis
Yes, the mismatch also appears with a score-independent grouping, and we added this analysis to Appendix A.2 (new Tab. 9). We also revised the discussion of Fig. 4 in Sec. 3.2 and added a reference to this score-independent analysis. Every proposal before NMS joins the same-class instance that it overlaps most (IoU 0.5), so neither the scores nor the matching enter the grouping.
With this grouping, we call a pair of proposals of the same instance inverted when the tighter one (IoU higher by at least 0.1) scores lower. The mAP lost to the ordering is the gain of an oracle re-ranking, which permutes the scores inside each instance according to IoU and leaves the boxes unchanged:
| Metric | Base | Base + MAR | TAM | TAM + MAR |
|---|---|---|---|---|
| Inverted pairs | 21.2% | 18.4% | 21.0% | 19.2% |
| mAP lost to the ordering | 5.35 | 4.56 | 5.57 | 5.15 |
Without MAR, about one pair in five has the wrong order, and a correct ordering inside each instance would add more than 5 mAP after NMS. MAR lowers the share of inverted pairs on both detectors, and it reduces the mAP lost to the ordering from 5.35 to 4.56 on the base detector and from 5.57 to 5.15 on the TAM detector.
W6 / Experimental Q1 and Q2. Additional training and reproduced baselines
We ran this control, and it shows that the gain of MAR does not come from the additional training. Starting from the same TAM model, we continue training with only the original detection loss, at the same learning rate and for the same number of epochs as MAR (new Tab. 14):
| Method | Avg. mAP, three seeds | Mean |
|---|---|---|
| TAM | 74.77 / 74.84 / 74.63 | 74.75 |
| + further training, original loss | 74.56 / 74.58 / 74.56 | 74.57 |
| + further training, MAR (ours) | 75.53 / 75.61 / 75.49 | 75.54 |
Further training with the original loss does not improve TAM, and MAR is 0.9 to 1.0 higher than this control in every seed.
On the baselines, the 73.8 for DiGIT and the 67.3 for TadTR in Tab. 1 are the results published in the DiGIT paper, and 72.8 and 64.7 are our own runs of the official code of each detector on the same features. For DiGIT, we use its released THUMOS14 configuration, which we did not modify or re-tune. With the settings described in the DiGIT paper, the gap to the released configuration is small (table under W3), so we keep the released configuration. For TadTR, no configuration for InternVideo2 features has been released, so we run its official code and change only the input dimension and the window length. Our TE-TAD run matches its reported 70.3. Our full model exceeds both the published 73.8 (Tab. 1, +1.7) and our reproduced baseline, and all ablation gains are measured against their reproduced baselines (Sec. 4.3, Appendices A.1 and A.4).
W7 / Experimental Q3. Align-DETR with its matching, and Rank-DETR
We ran both comparisons that the reviewer asks for, Align-DETR with its many-to-one matching and Rank-DETR with its loss and matching cost, under the same training budget as MAR and with three seeds each (new Tab. 14). Both stay within 0.1 of the original loss.
For Align-DETR, we follow its official code: its loss, many-to-one matching on the encoder and the intermediate decoder layers (k = 4 and = 1.5, the defaults of its paper), and one-to-one matching on the last layer. For Rank-DETR, we use its GIoU-aware classification loss and its high-order matching cost.
| Method | Avg. mAP, three seeds | Mean |
|---|---|---|
| Original loss only | 74.56 / 74.58 / 74.56 | 74.57 |
| Align-DETR | 74.59 / 74.59 / 74.64 | 74.61 |
| Rank-DETR | 74.63 / 74.59 / 74.53 | 74.58 |
| MAR (ours) | 75.53 / 75.61 / 75.49 | 75.54 |
MAR is 0.8 to 1.0 higher than both methods in every seed. Both keep every unmatched proposal of the last layer as background, which is the supervision that MAR changes.
W8 / Experimental Q4. Reproducibility across random seeds
Yes, the improvements hold across random seeds. We repeated the component ablation of Tab. 3 with three seeds and report the mean and the standard deviation (new Tab. 10):
| Method | Avg. mAP, three seeds | Mean std |
|---|---|---|
| Base | 72.95 / 72.96 / 72.51 | 72.81 0.26 |
| + TAM | 74.77 / 74.84 / 74.63 | 74.75 0.11 |
| + MAR | 74.75 / 74.29 / 74.31 | 74.45 0.26 |
| + TAM + MAR (ours) | 75.53 / 75.61 / 75.49 | 75.54 0.06 |
Every gain holds in every seed. TAM adds 1.8 to 2.1 over the base detector. MAR adds 1.3 to 1.8 on the base detector and 0.8 to 0.9 on top of TAM. The full model adds 2.6 to 3.0, far above the standard deviation of any row. The comparisons against alternative modules and losses also use three seeds each (new Tabs. 12 and 14).
On the other two datasets, three seeds of the full model give (new Tab. 10):
| Dataset | Avg. mAP, three seeds | Mean std |
|---|---|---|
| ActivityNet-1.3 | 42.55 / 42.62 / 42.70 | 42.62 0.08 |
| HACS-Segment | 46.93 / 47.08 / 47.12 | 47.04 0.10 |
Every seed exceeds the result of DiGIT in Tabs. 1 and 2 (42.0 and 45.9).
For the filters, we do not read the 0.2 gap as an advantage of the fixed ones. Our statement in Sec. 4.3 is that making the filters learnable does not improve mAP in this ablation, so the two fixed filters suffice.
W1. Gradient of the sorting loss
Thank you for the suggestion. We use ordinary automatic differentiation through the soft step, and we now state this in Sec. 3.4. For the pair in the example, the sort loss saturates, as the reviewer notes, but the rank loss can still act on the tight proposal. With a score of 0.1, it lies within δ of every negative that scores below 0.6, so these active comparisons can raise its score. Once its score exceeds 0.4, its gap to the loose proposal falls below δ, and the sort loss becomes active.
We also tested whether such saturated pairs limit MAR. We replaced automatic differentiation with the Identity Update of Rank & Sort, which gives these pairs a gradient, and kept everything else unchanged (new Tab. 14):
The Identity Update does not raise the mean and varies more across seeds, so we keep automatic differentiation.
W2. Proposals with IoU between 0.4 and 0.5
We quantified this interval as the reviewer asks and added the analysis to Appendix A.3 (new Tab. 16). The scores of these proposals stay low without a direct loss, and their false-positive contribution is almost the same with and without MAR. The table compares the unmatched proposals of this interval in the TAM detector, where the focal loss still applies to them, and in the full detector, where no direct score loss applies:
MAR leaves these proposals out of both sets because they are too loose to serve as positives and too close to an instance to serve as clear background. The sensitivity study (Fig. 8) contains the three settings that close the interval (λ = 0.5, τp = 0.4, τn = 0.5), and none improves on the default.
W3. Regression objective
Eq. (2) is complete, and it is the objective of the baseline. The DiGIT paper describes a GIoU loss, while its official release, which we build on, uses the IoU loss 1 - tIoU and gives the log-ratio term a weight of zero on THUMOS14. We did not change the regression loss, so the base detector and all our variants share it (now stated in Appendix A.1). We also trained the baseline with the settings of the DiGIT paper (GIoU loss and a weight decay of 1e-4, its supplementary Table B):
The gap between the two settings is small, so we keep the released configuration.
Matched pairs overlap their instances from the start of training. The queries start from the top-scoring encoder proposals, and the matching cost includes the overlap. When we train the base detector, the mean IoU between matched predictions and their instances is already 0.73 in the first epoch. For the trained detector, 1.0% of the matched pairs on the training videos are disjoint.
W4 / Methodology Q1 (second part) and Q2. Support for transition blindness
The relation between transition duration and recall holds after controlling for action duration and category, and neither extra parameters nor generic temporal filters give the gain of TAM.
For the first control (Methodology Q1, second part), we split the boundaries into thirds by action duration. In every third, the base detector recalls fewer tight proposals (IoU ≥ 0.9) in the more gradual half of the boundaries (new Tab. 7):
The relation is also negative in 14 of the 16 classes, and a logistic regression with action duration, class, and boundary side as controls gives an odds ratio of 0.74 per standard deviation of transition duration (95% CI [0.68, 0.80]).
For the second (Methodology Q2), we keep the adapters and the position of TAM and change only what the adapters read, with three seeds each (new Tab. 12). The temporal convolution spans the same time steps as our direction view, so its receptive field and parameters are comparable:
The adapters alone gain 0.1 and the two generic filters gain 0.6 and 0.9, against 1.9 for TAM.