acceptodds
Under review as a conference paper at ICLR 2027

Seam-DETR: Transition- and Quality-Aware Decoder for Temporal Action Detection

Abstract

Query-based detectors have become a strong family for temporal action detection, yet their localization quality remains unsatisfactory. We trace the problem to the two sides of the decoder. On the input side, the memory attended by the decoder encodes temporal boundaries only implicitly, making it difficult to perceive temporal transitions, especially gradual ones. On the supervision side, one-to-one matching assigns the positive label to a single proposal per instance and treats its near-duplicates as background, so the learned scores only reflect the matching assignments rather than the actual overlap with the ground truth. We propose Seam-DETR, which restores localization quality from both sides. Transition-Aware Memory (TAM) introduces two parameter-free views of the memory to what the localization attention reads: one measuring how much the content changes at each moment and the other exposing what it changes into, so the transitions become perceptible. Match-Aware Ranking (MAR) prevents unmatched near-duplicates from being suppressed as background and ranks same-class proposals by their overlap with the ground truth, making confidence scores more localization-aware. The two designs improve localization quality with minimal overhead, where TAM adds only two lightweight adapters, and MAR introduces no additional parameters or inference cost. Extensive experiments demonstrate that Seam-DETR achieves state-of-the-art performance among query-based methods on THUMOS14, ActivityNet-1.3, and HACS-Segment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Temporal Action Detection, DETR, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sbvNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Seam-DETR improves query-based temporal action detection through Transition-Aware Memory (TAM), which injects temporal change features into localization memory, and Match-Aware Ranking (MAR), which relaxes suppression of unmatched high-overlap predictions and ranks proposals by localization quality.

Critical strengths
  1. TAM uses simple temporal filters and lightweight adapters, while MAR introduces no additional inference parameters or computation. Improvements on both TadTR and TE-TAD support the applicability of the approach beyond its primary detector.

  2. This method improves average mAP over the reported DiGIT baseline by 1.7, 0.6, and 1.1 percentage points across three benchmarks.

Critical weaknesses

After reading this manuscript, I have some concerns regarding the methodology and experiment.

First, the ranking loss can have zero gradients despite an incorrect ordering. With the default delta=0.5, consider two proposals with IoUs 0.9 and 0.6, but scores 0.1 and 0.9: Eqs. (8) to (10) produce a positive sorting penalty, yet ordinary differentiation gives zero score gradients because the comparison functions are saturated. The original Rank & Sort method uses a custom Identity Update, whereas this manuscript does not specify whether that update or ordinary automatic differentiation is used [1]. Under ordinary differentiation, the sorting term cannot directly correct this severe inversion.

The default thresholds leave some false positives completely unconstrained by the relevant score losses. For an unmatched proposal with class-specific IoU 0.45, the default lambda=0.4 removes its negative focal loss, while tau_p=0.5 and tau_n=0.3 exclude it from both ranking sets. Its class score therefore receives no direct supervision from these terms and can remain high, although the proposal is a false positive at evaluation thresholds of 0.5 and above. This is a concrete gap in the claimed quality-aware supervision; the authors should quantify the scores and false-positive contribution of proposals in this interval.

The stated regression objective is either incomplete or materially different from the baseline. Equation (2) specifies 1-tIoU, which provides zero direct boundary-regression gradient when a matched prediction and its ground truth are strictly disjoint. However, the original DiGIT specifies GIoU and a log-ratio distance loss [2],

Also, in the methodology part, the “transition blindness” explanation is not convincingly established. Figure 3 shows that memory features differ across boundaries, but this does not demonstrate that the decoder cannot exploit those differences; the relationship between transition duration and recall in Figure 1 also leaves action duration and category uncontrolled. Although Table 7 reports a meaningful recall improvement from 0.38 to 0.46 for gradual boundaries, comparisons against ordinary temporal differences or temporal convolutions with comparable receptive fields are missing. Consequently, the evidence does not clearly distinguish the proposed mechanism from generic temporal feature enhancement.

[1] Oksuz, K., Cam, B.C., Akbas, E. and Kalkan, S., 2021, October. Rank & sort loss for object detection and instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 3009–3018). IEEE. [2] Kim, H.J., Lee, Y., Hong, J.H. and Lee, S.W., 2025, June. DiGIT: Multi-dilated gated encoder and central-adjacent region integrated decoder for temporal action detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24286–24296). IEEE.

In your experiment, the confidence-mismatch diagnosis suffers from selection bias. Figure 4 partitions predictions using test-time Hungarian matching and then compares their scores, even though the DiGIT matching cost already incorporates classification scores [1]. Higher confidence among matched predictions therefore partly reflects the grouping procedure and does not independently establish harmful ranking errors caused by one-to-one supervision. The paper should quantify how often lower-IoU predictions outrank better predictions for the same instance and how this affects post-NMS AP; increasing duplicate scores alone is insufficient evidence of better ranking.

I also find some issues in the baseline comparison, where additional training and baseline discrepancies are insufficiently controlled. The reproduced DiGIT baseline scores 72.8 in Table 3, versus 73.8 in Table 1, without adequately explaining the discrepancy. Also, some experiments give MAR an additional 20 training epochs but do not report a control that continues training TAM with the original detection loss under the same learning rate and schedule. Without this control, the contribution of MAR cannot be cleanly separated from the benefit of additional optimization.

Furthermore, comparisons with the most relevant supervision methods are incomplete. Table 9 transfers only the Align-DETR loss while retaining one-to-one matching, excluding the original method’s many-to-one supervision of intermediate layers; Rank-DETR’s ranking-oriented loss and matching designs are also absent from the experimental comparison [2,3]. These results establish an advantage over two particular loss substitutions, but they do not adequately demonstrate superiority over existing quality-aware supervision strategies.

In the experiment of the ablation study, the reliability of the reported improvements is insufficiently established. Tables 1 and 2 show average mAP gains of 1.7, 0.6, and 1.1 percentage points over DiGIT using the same features, but provide neither results across multiple random seeds nor confidence intervals for mAP. The error bars in Figure 1 describe diagnostic statistics and do not establish robustness to training variability. In particular, differences such as the 0.2-point gap between fixed and learnable filters are insufficient to support reliable conclusions about design superiority.

[1] Kim, H.J., Lee, Y., Hong, J.H. and Lee, S.W., 2025, June. DiGIT: Multi-dilated gated encoder and central-adjacent region integrated decoder for temporal action detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24286–24296).

[2] Pu, Y., Liang, W., Hao, Y., Yuan, Y., Yang, Y., Zhang, C., Hu, H. and Huang, G., 2023. Rank-DETR for high quality object detection. In Advances in Neural Information Processing Systems, 36 (pp. 16100–16113).

[3] Cai, Z., Liu, S., Wang, G., Li, Z., Ge, Z., Zhang, X. and Huang, D., 2024, November. Align-DETR: Enhancing end-to-end object detection with aligned loss. In Proceedings of the 35th British Machine Vision Conference.

Questions for authors

For the methodology part:

  1. What is the substantive methodological advance beyond combining fixed temporal filters, an ignore rule, and existing rank-and-sort objectives? Does the relationship between transition duration and recall persist after controlling for action duration and category?

  2. Does TAM outperform ordinary temporal differences or temporal convolutions with comparable receptive fields and parameter budgets?

  3. Can confidence mismatch be demonstrated using score-independent proposal grouping, avoiding selection bias from classification-aware Hungarian matching?

  4. How often does MAR correct within-instance IoU ranking errors, and how much do those corrections improve post-NMS AP?

For the experimental part:

  1. What happens when TAM receives the same additional 20 epochs and learning-rate schedule as MAR, using only the original detection loss?

  2. Why do the reproduced DiGIT and TadTR baselines score 72.8 and 64.7, versus 73.8 and 67.3 in Table 1?

  3. Under matched training budgets, how does MAR compare with Align-DETR including its intermediate-layer matching strategy, and with Rank-DETR’s loss and matching designs?

  4. Are the main improvements and small ablation differences reproducible across random seeds? The authors should report mean and standard deviation for the key comparisons.

  5. Which data splits were used to select TAM’s filter scales and MAR’s hyperparameters, and were evaluation annotations involved in those choices?

  6. How sensitive is MAR to the transition between training stages, and what justifies changing from a hard ignore rule on THUMOS14 to a soft rule on the other datasets?

6 comments

W1. Gradient of the sorting loss

Thank you for the suggestion. We use ordinary automatic differentiation through the soft step, and we now state this in Sec. 3.4. For the pair in the example, the sort loss saturates, as the reviewer notes, but the rank loss can still act on the tight proposal. With a score of 0.1, it lies within δ\delta of every negative that scores below 0.6, so these active comparisons can raise its score. Once its score exceeds 0.4, its gap to the loose proposal falls below δ\delta, and the sort loss becomes active.

We also tested whether such saturated pairs limit MAR. We replaced automatic differentiation with the Identity Update of Rank & Sort, which gives these pairs a gradient, and kept everything else unchanged (new Tab. 14):

MethodAvg. mAP, three seedsMean
MAR with Identity Update75.58 / 75.29 / 75.0475.30
MAR with automatic differentiation (ours)75.53 / 75.61 / 75.4975.54

The Identity Update does not raise the mean and varies more across seeds, so we keep automatic differentiation.

W2. Proposals with IoU between 0.4 and 0.5

We quantified this interval as the reviewer asks and added the analysis to Appendix A.3 (new Tab. 16). The scores of these proposals stay low without a direct loss, and their false-positive contribution is almost the same with and without MAR. The table compares the unmatched proposals of this interval in the TAM detector, where the focal loss still applies to them, and in the full detector, where no direct score loss applies:

Unmatched proposals with IoU in [0.4, 0.5)TAMTAM + MAR
Mean score0.040.06
Share with score above 0.31.5%4.0%
AP at tIoU 0.5 if removed+0.4+0.6
Average mAP if removed+0.0+0.1

MAR leaves these proposals out of both sets because they are too loose to serve as positives and too close to an instance to serve as clear background. The sensitivity study (Fig. 8) contains the three settings that close the interval (λ\lambda = 0.5, τp\tau_p = 0.4, τn\tau_n = 0.5), and none improves on the default.

W3. Regression objective

Eq. (2) is complete, and it is the objective of the baseline. The DiGIT paper describes a GIoU loss, while its official release, which we build on, uses the IoU loss 1 - tIoU and gives the log-ratio term a weight of zero on THUMOS14. We did not change the regression loss, so the base detector and all our variants share it (now stated in Appendix A.1). We also trained the baseline with the settings of the DiGIT paper (GIoU loss and a weight decay of 1e-4, its supplementary Table B):

BaselineAvg. mAP, three seedsMean
Released configuration72.95 / 72.96 / 72.5172.81
Paper settings72.85 / 72.74 / 72.6972.76

The gap between the two settings is small, so we keep the released configuration.

Matched pairs overlap their instances from the start of training. The queries start from the top-scoring encoder proposals, and the matching cost includes the overlap. When we train the base detector, the mean IoU between matched predictions and their instances is already 0.73 in the first epoch. For the trained detector, 1.0% of the matched pairs on the training videos are disjoint.

W4 / Methodology Q1 (second part) and Q2. Support for transition blindness

The relation between transition duration and recall holds after controlling for action duration and category, and neither extra parameters nor generic temporal filters give the gain of TAM.

For the first control (Methodology Q1, second part), we split the boundaries into thirds by action duration. In every third, the base detector recalls fewer tight proposals (IoU ≥\ge 0.9) in the more gradual half of the boundaries (new Tab. 7):

Action durationAbrupt halfGradual half
Short (1.0 to 2.3s)0.390.28
Medium (2.3 to 5.4s)0.590.39
Long (above 5.4s)0.720.58

The relation is also negative in 14 of the 16 classes, and a logistic regression with action duration, class, and boundary side as controls gives an odds ratio of 0.74 per standard deviation of transition duration (95% CI [0.68, 0.80]).

For the second (Methodology Q2), we keep the adapters and the position of TAM and change only what the adapters read, with three seeds each (new Tab. 12). The temporal convolution spans the same time steps as our direction view, so its receptive field and parameters are comparable:

MethodAvg. mAP, three seedsMean
DiGIT72.95 / 72.96 / 72.5172.81
+ adapters on the memory itself73.41 / 72.66 / 72.7872.95
+ temporal difference73.39 / 73.13 / 73.7273.41
+ temporal convolution73.78 / 73.53 / 73.7173.67
+ TAM (ours)74.77 / 74.84 / 74.6374.75

The adapters alone gain 0.1 and the two generic filters gain 0.6 and 0.9, against 1.9 for TAM.

W5 / Methodology Q3 and Q4. Selection bias in the confidence-mismatch diagnosis

Yes, the mismatch also appears with a score-independent grouping, and we added this analysis to Appendix A.2 (new Tab. 9). We also revised the discussion of Fig. 4 in Sec. 3.2 and added a reference to this score-independent analysis. Every proposal before NMS joins the same-class instance that it overlaps most (IoU ≥\ge 0.5), so neither the scores nor the matching enter the grouping.

With this grouping, we call a pair of proposals of the same instance inverted when the tighter one (IoU higher by at least 0.1) scores lower. The mAP lost to the ordering is the gain of an oracle re-ranking, which permutes the scores inside each instance according to IoU and leaves the boxes unchanged:

MetricBaseBase + MARTAMTAM + MAR
Inverted pairs21.2%18.4%21.0%19.2%
mAP lost to the ordering5.354.565.575.15

Without MAR, about one pair in five has the wrong order, and a correct ordering inside each instance would add more than 5 mAP after NMS. MAR lowers the share of inverted pairs on both detectors, and it reduces the mAP lost to the ordering from 5.35 to 4.56 on the base detector and from 5.57 to 5.15 on the TAM detector.

W6 / Experimental Q1 and Q2. Additional training and reproduced baselines

We ran this control, and it shows that the gain of MAR does not come from the additional training. Starting from the same TAM model, we continue training with only the original detection loss, at the same learning rate and for the same number of epochs as MAR (new Tab. 14):

MethodAvg. mAP, three seedsMean
TAM74.77 / 74.84 / 74.6374.75
+ further training, original loss74.56 / 74.58 / 74.5674.57
+ further training, MAR (ours)75.53 / 75.61 / 75.4975.54

Further training with the original loss does not improve TAM, and MAR is 0.9 to 1.0 higher than this control in every seed.

On the baselines, the 73.8 for DiGIT and the 67.3 for TadTR in Tab. 1 are the results published in the DiGIT paper, and 72.8 and 64.7 are our own runs of the official code of each detector on the same features. For DiGIT, we use its released THUMOS14 configuration, which we did not modify or re-tune. With the settings described in the DiGIT paper, the gap to the released configuration is small (table under W3), so we keep the released configuration. For TadTR, no configuration for InternVideo2 features has been released, so we run its official code and change only the input dimension and the window length. Our TE-TAD run matches its reported 70.3. Our full model exceeds both the published 73.8 (Tab. 1, +1.7) and our reproduced baseline, and all ablation gains are measured against their reproduced baselines (Sec. 4.3, Appendices A.1 and A.4).

W7 / Experimental Q3. Align-DETR with its matching, and Rank-DETR

We ran both comparisons that the reviewer asks for, Align-DETR with its many-to-one matching and Rank-DETR with its loss and matching cost, under the same training budget as MAR and with three seeds each (new Tab. 14). Both stay within 0.1 of the original loss.

For Align-DETR, we follow its official code: its loss, many-to-one matching on the encoder and the intermediate decoder layers (k = 4 and τ\tau = 1.5, the defaults of its paper), and one-to-one matching on the last layer. For Rank-DETR, we use its GIoU-aware classification loss and its high-order matching cost.

MethodAvg. mAP, three seedsMean
Original loss only74.56 / 74.58 / 74.5674.57
Align-DETR74.59 / 74.59 / 74.6474.61
Rank-DETR74.63 / 74.59 / 74.5374.58
MAR (ours)75.53 / 75.61 / 75.4975.54

MAR is 0.8 to 1.0 higher than both methods in every seed. Both keep every unmatched proposal of the last layer as background, which is the supervision that MAR changes.

W8 / Experimental Q4. Reproducibility across random seeds

Yes, the improvements hold across random seeds. We repeated the component ablation of Tab. 3 with three seeds and report the mean and the standard deviation (new Tab. 10):

MethodAvg. mAP, three seedsMean ±\pm std
Base72.95 / 72.96 / 72.5172.81 ±\pm 0.26
+ TAM74.77 / 74.84 / 74.6374.75 ±\pm 0.11
+ MAR74.75 / 74.29 / 74.3174.45 ±\pm 0.26
+ TAM + MAR (ours)75.53 / 75.61 / 75.4975.54 ±\pm 0.06

Every gain holds in every seed. TAM adds 1.8 to 2.1 over the base detector. MAR adds 1.3 to 1.8 on the base detector and 0.8 to 0.9 on top of TAM. The full model adds 2.6 to 3.0, far above the standard deviation of any row. The comparisons against alternative modules and losses also use three seeds each (new Tabs. 12 and 14).

On the other two datasets, three seeds of the full model give (new Tab. 10):

DatasetAvg. mAP, three seedsMean ±\pm std
ActivityNet-1.342.55 / 42.62 / 42.7042.62 ±\pm 0.08
HACS-Segment46.93 / 47.08 / 47.1247.04 ±\pm 0.10

Every seed exceeds the result of DiGIT in Tabs. 1 and 2 (42.0 and 45.9).

For the filters, we do not read the 0.2 gap as an advantage of the fixed ones. Our statement in Sec. 4.3 is that making the filters learnable does not improve mAP in this ablation, so the two fixed filters suffice.

Reviewer_B3SROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Seam-DETR for temporal action detection, improving localization through modifications to the decoder input and training supervision. Transition-Aware Memory (TAM) constructs two temporal-change views using a Gaussian smoothing residual and the difference between past and future window averages, then injects them into localization cross-attention through lightweight adapters. Match-Aware Ranking (MAR) reduces negative classification supervision for highly overlapping unmatched predictions and adopts existing rank-and-sort losses to better align prediction scores with localization quality. Built on DiGIT with pre-extracted InternVideo2 features, the method achieves average mAP scores of 75.5%, 42.6%, and 47.0% on THUMOS14, ActivityNet-1.3, and HACS-Segment, respectively.

Critical strengths

1.The paper is clearly organized, with the two components addressing temporal-transition information and localization-quality supervision, respectively. The motivations are easy to follow. TAM is relatively straightforward to implement, and MAR introduces no additional inference parameters or computation, giving the approach practical value. 2.The experiments cover three datasets and include component ablations, an adapters-only control, alternative losses, cross-detector validation, and computational-cost analysis. Results on TadTR and TE-TAD suggest that the combination is transferable and that its benefits are not restricted to DiGIT. Critical Weaknesses

Critical weaknesses

1.Limited methodological novelty. TAM's two views are a Gaussian smoothing residual and the difference between past and future window averages: conventional fixed temporal filters connected to an existing detector through zero-initialized residual adapters. MAR explicitly adopts existing rank-and-sort losses and combines them with a strategy that ignores negative supervision for highly overlapping unmatched predictions. Presenting these components as improvements to the input and supervision sides of the decoder provides a clear narrative, but does not establish a sufficiently distinct technical contribution. The experiments demonstrate that the combination works, but do not sufficiently establish a contribution beyond incremental integration and task-specific adaptation of existing components. 2.Important controls are missing to establish the necessity of the selected designs. The adapters-only control in Table 8 rules out an explanation based solely on additional parameters, but does not compare against simple temporal differences or other filters with comparable computational costs. For MAR, the VFL and Align-DETR losses in Table 9 continue to treat unmatched predictions as background, leaving the removal of negative supervision uncontrolled. In Table 5, the ignore band alone achieves 75.0 average mAP, compared with 75.5 for the full method. Comparisons against direct tIoU soft-target supervision under the same ignore band, together with repeated-run variability, are needed to establish the contribution of the ranking formulation itself. 3.The benefit of MAR is not fully separated from additional training. On THUMOS14, MAR adds 20 fine-tuning epochs after TAM training, but the paper does not explicitly report a control that continues from the same TAM checkpoint using only the original detection loss, with the same learning rate and number of epochs. Although the alternative losses also use a fine-tuning schedule, a direct reference using the original objective under that schedule is still missing. Furthermore, DiGIT achieves 73.8 average mAP in Table 1, whereas the reproduced baseline achieves 72.8 in Table 3. This discrepancy should be explained, with a clear distinction between published results and controlled reproductions.

Questions for authors

See above

6 comments

W1. Methodological novelty

Thank you for the comment. Our contribution is to expose temporal changes explicitly to localization attention and extend quality-based score supervision beyond the one-to-one matched proposals. These designs address transition blindness and confidence mismatch, which we observe in TadTR, TE-TAD, and DiGIT (Fig. 1).

TAM supplies local change and longer-range contrasts between past and future only to the cross-attention that localizes boundaries, while preserving the original content memory. This differs from the original designs of FDDet and RTD-Net, which modify the features read by the whole detector. MAR retains one-to-one matching for regression and extends quality-based ranking to eligible same-class proposals, whether matched or not. Its ignore band removes conflicting negative supervision for high-overlap unmatched proposals across decoder layers, allowing the rank-and-sort losses to supervise their existing class scores. Align-DETR and Rank-DETR still treat unmatched proposals of the last layer as background.

With the adapters and insertion point fixed, temporal differences and temporal convolution gain 0.6 and 0.9 mAP over the baseline. Related boundary modules gain 0.5 to 0.8, compared with 1.9 for TAM (new Tab. 12).

MethodAvg. mAP, three seedsMean
DiGIT72.95 / 72.96 / 72.5172.81
+ temporal difference73.39 / 73.13 / 73.7273.41
+ temporal convolution73.78 / 73.53 / 73.7173.67
+ FGAAD (FDDet)72.55 / 73.60 / 73.8573.33
+ boundary module (RTD-Net)74.10 / 72.89 / 73.7473.58
+ TAM (ours)74.77 / 74.84 / 74.6374.75

On the supervision side, the loss and matching of Align-DETR and Rank-DETR bring no gain over the original loss under the same training budget, and MAR is 1.0 higher:

MethodAvg. mAP, three seedsMean
Original loss only74.56 / 74.58 / 74.5674.57
Align-DETR74.59 / 74.59 / 74.6474.61
Rank-DETR74.63 / 74.59 / 74.5374.58
MAR (ours)75.53 / 75.61 / 75.4975.54

The rank-and-sort losses alone bring no gain (Tab. 5). With the ignore band, they improve mAP from 75.21 to 75.54, exceeding both ignore-only supervision and direct tIoU soft targets in every seed (table under W2). These comparisons support ranking high-overlap unmatched proposals after removing their conflicting background supervision. Both components also carry over to TadTR and TE-TAD with the same hyperparameters (+2.2 and +1.1, Appendix A.4).

W2. Controls for the selected designs

We ran the three controls that the reviewer asks for, each with three seeds, and added them to Appendix A.3 (new Tabs. 12 and 14). TAM and MAR remain the highest in every seed.

For TAM, we replace its two views with the first-order temporal difference of the memory and with a learned temporal convolution of the same temporal span. Both keep the adapters and the position of TAM, so their parameters and cost are comparable. They reach 73.41 and 73.67, against 74.75 for TAM (first table under W1).

For MAR, we keep the ignore band and replace the ranking losses with each alternative, under the same training budget:

MethodAvg. mAP, three seedsMean
Original loss, no ignore band74.56 / 74.58 / 74.5674.57
Ignore band only75.29 / 75.18 / 75.1575.21
Ignore band + VFL74.98 / 74.92 / 74.7774.89
Ignore band + Align-DETR loss74.86 / 74.95 / 74.8874.90
Ignore band + tIoU soft targets75.17 / 75.16 / 75.0975.14
Ignore band + ranking losses (MAR)75.53 / 75.61 / 75.4975.54

VFL, the Align-DETR loss, and direct tIoU soft targets all stay below MAR under the same ignore band. The ranking losses improve over the ignore band alone in every seed.

W3. Additional training and the reproduced baseline

We ran this control, and it shows that the gain of MAR does not come from the additional training. Starting from the same TAM model, we continue training with only the original detection loss, at the same learning rate and for the same number of epochs as MAR (new Tab. 14):

MethodAvg. mAP, three seedsMean
TAM74.77 / 74.84 / 74.6374.75
+ further training, original loss74.56 / 74.58 / 74.5674.57
+ further training, MAR (ours)75.53 / 75.61 / 75.4975.54

Further training with the original loss does not improve TAM, and MAR is 0.9 to 1.0 higher than this control in every seed.

On the baseline, 73.8 is the result published in the DiGIT paper, and 72.8 is our own run of the official code with its released configuration, which we did not modify or re-tune. With the settings described in the DiGIT paper (GIoU loss, weight decay of 1e-4), the baseline reaches 72.85 / 72.74 / 72.69 (mean 72.76), against 72.95 / 72.96 / 72.51 (mean 72.81) for the released configuration. The gap is small, so we keep the released configuration. Our model outperforms both the published 73.8 and our reproduced baseline, and all ablations share the released configuration of this baseline (now stated in Sec. 4.3 and Appendix A.1).

Reviewer_B3SROpenReview ↗

Official Comment by Reviewer

The additional experiments address much of my concern about missing controls, particularly the comparisons against temporal difference/convolution alternatives and against tIoU soft targets under the same ignore band. However, they primarily establish that the selected designs outperform these particular alternatives; they do not fully resolve my concern about methodological novelty. TAM still appears to combine conventional temporal filters with residual adapters, while MAR combines existing ranking losses with an ignore-band strategy. The authors clarify where these components are inserted and how they are used, but the remaining question is whether this constitutes a sufficiently distinct methodological contribution beyond a carefully evaluated task-specific integration. The discrepancy between the published DiGIT result (73.8) and the reproduced baseline (72.8) also remains unexplained.

Reviewer_oH1dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Seam-DETR to improve localization quality in query-based temporal action detection. Transition-Aware Memory (TAM) adds explicit temporal change information to the memory used by the decoder for localization. Match-Aware Ranking (MAR) adjusts the supervision of high-quality unmatched proposals and applies localization-quality-based ranking losses. The paper evaluates the method on THUMOS14, ActivityNet-1.3, and HACS-Segment, with component ablations and diagnostic analyses.

Critical strengths
  1. Overall, I consider this a paper of relatively high quality. Its two technical components, TAM and MAR, both address localization quality, giving the paper a unified theme. The figures in the main text help explain the paper, and the tables are few but necessary. State-of-the-art performance among query-based methods is validated on three datasets. The appendix provides necessary details and additional analyses with restraint. I particularly appreciate the authors' effort in Sections 1, 3.2, and A.2 to examine the problems in depth and articulate the motivation for their design.
Critical weaknesses
  1. Figure 1, which supports the motivation, is not sufficiently rigorous.

Figure 1(a). The transition duration in this figure uses the time required for projected video features to move from 10% to 90% of the difference between the pre-boundary and post-boundary means as a proxy. This choice is intuitively reasonable, yet the paper provides no evidence that the proxy corresponds to the actual action transition phase, despite using it as a key support for the motivation.

Figure 1(b). The figure does show that the proposed method gives unmatched duplicate proposals a higher mean confidence score (MCS). However, stronger and more direct evidence is needed to support the motivation for MAR, such as cases where unmatched duplicate proposals have higher tIoU than matched proposals for the same instance but receive lower confidence scores. Figure 1(b) alone does not demonstrate that the lower MCS of unmatched duplicate proposals in existing methods is harmful.

  1. Discussion of and comparisons with related work on boundary changes are limited.

Several methods, such as FDDet [1], RTD-Net [2], and BSP [3], also address boundary-sensitive representations or temporal changes. The paper provides limited discussion of how TAM relates to these approaches and lacks direct comparisons with their comparable components or ideas.

[1] Xinnan Zhu et al. FDDet: Frequency-Decoupling for Boundary Refinement in Temporal Action Detection. arXiv:2504.00647, 2025.

[2] Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu. Relaxed Transformer Decoders for Direct Action Proposal Generation. ICCV, 2021.

[3] Mengmeng Xu et al. Boundary-sensitive Pre-training for Temporal Localization in Videos. ICCV, 2021.

Questions for authors
  1. What evidence supports interpreting the feature-based transition duration as the duration of a gradual action transition? If a rigorous proof is unavailable, human validation on a reasonably sized sample would also be acceptable.

  2. The presentation of Figure 1(b) needs to be improved to support the motivation for MAR.

  3. Could the authors discuss work closely related to TAM and adapt comparable modules or ideas from these methods to the same baseline for comparison?

3 comments

W1a / Q1. Whether the transition proxy reflects the actual transition (Fig. 1(a))

Thank you for the suggestion. We ran the human validation and added it to Appendix A.2. We drew 20 boundaries at random from each of the six proxy groups of Fig. 1(a), 120 in total. An annotator watched a 6s clip around each boundary, in shuffled order and without seeing the proxy. The annotator labeled a change as abrupt if it completes within 1s, such as a take-off, and as gradual if it takes longer, such as a run-up that speeds up. For a gradual change, the annotator also marked where the transition starts and ends. The labels follow the proxy:

Proxy group123456
Median proxy (s)0.670.800.931.201.471.85
Labeled gradual0%0%5%15%25%45%

Among the 56 boundaries with a proxy below 1s, the annotator found one gradual change (2%). Among the 64 boundaries above 1s, the annotator found 17 (27%). At these 18 boundaries, the median marked transition lasts 1.45s and the median proxy is 1.53s. A proxy below 1s thus almost never comes with a gradual transition, and a longer proxy makes one more likely.

W1b / Q2. Direct support for the motivation of MAR (Fig. 1(b))

Thank you for the suggestion. We measured the case that the reviewer describes, quantified its cost, and revised Fig. 1(b) accordingly.

This case is the common one. In the base detector, 79% of the instances have an unmatched proposal that is tighter than the matched one, and this tighter proposal scores lower in every such instance. Fig. 10(b) shows one example: two proposals at IoU 0.54 score 0.62 and 0.56, while two at IoU 0.86 score only 0.37 and 0.22.

These low scores are harmful. We keep the detections of the base detector unchanged and apply an oracle re-ranking, which permutes their original scores inside each instance according to IoU:

Base detector[email protected][email protected]Avg. mAP
Original scores65.452.772.8
Oracle re-ranking74.470.578.2

The boxes stay the same, so the lost AP comes from the scores alone.

The revised Fig. 1(b) now shows this case directly. For each detector, the arrow goes from the matched proposal to the tightest unmatched one, averaged over the instances where the unmatched one is tighter. In existing detectors, the unmatched proposal has the higher IoU (0.82 to 0.87 against 0.69 to 0.74) and a far lower score (0.03 to 0.13 against 0.61 to 0.75). Ours raises this score to 0.25.

W2 / Q3. Related work on boundary changes and direct comparison

Thank you for the suggestion. We added the discussion to Sec. 2 and the comparison to Appendix A.3 (new Tab. 12). On the same baseline and training schedule, TAM gains 1.9 mAP, against 0.5 for the FGAAD module of FDDet and 0.8 for the boundary module of RTD-Net, while our feature-level adaptation of BSP does not improve the baseline:

MethodAvg. mAP, three seedsMean
DiGIT72.95 / 72.96 / 72.5172.81
+ FGAAD (FDDet)72.55 / 73.60 / 73.8573.33
+ boundary module (RTD-Net)74.10 / 72.89 / 73.7473.58
+ feature-level adaptation (BSP)71.77 / 72.15 / 72.2272.05
+ TAM (ours)74.77 / 74.84 / 74.6374.75

TAM is the highest in every seed. We adapted FDDet and RTD-Net from their official code and BSP from its paper, since it has no released code. The original methods strengthen features before decoding.

  • FDDet strengthens the high-frequency part of the input features.
  • RTD-Net weights the features by start and end probabilities from a boundary head trained with frame-level labels.
  • BSP pre-trains a video backbone on synthesized videos; our feature-level adaptation instead trains an encoder on pre-extracted features.

TAM acts inside the decoder. It leaves the memory unchanged, computes its two views with parameter-free filters, without extra supervision or pre-training, and gives them only to the cross-attention that localizes boundaries.

Thank you for the additional analyses. The human validation supports the transition measure as a useful, though noisy, proxy. The revised confidence-mismatch analysis and same-baseline comparisons with related methods address my main concerns. I maintain my weak-accept recommendation.

Thank you for taking the time to review the additional analyses. Your feedback helped us clarify the transition measure and strengthen our evaluation.

Comments on the paper

1 comment
Area_Chair_nbRSOpenReview ↗

Discussion between Authors and Reviewers

Dear Reviewers,

The authors have submitted their rebuttal. Please read it and discuss with the authors via posted messages if necessary.

Note that the author-reviewer discussion phase ends on November 18 at 11:59 pm AoE. It is appreciated if you could share your thoughts as soon as possible, so the authors have enough time to clarify any remaining concerns.

Thank you!

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.