- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper claims that DETR-based joint MR/HD models use saliency only once, so early saliency errors cannot be corrected by later localization. Built on SG-DETR, the proposed ReFocus-DETR introduces BSSI to write saliency-derived span evidence into clip memory, and IMR to iteratively refine saliency and memory with the decoder's current predictions through weight-shared decoder passes. Experiments on four datasets show consistent improvements on both MR and HD.
Critical strengths
The experimental analysis is relatively comprehensive. Beyond the main results on four datasets, the paper includes capacity- and supervision-controlled comparisons, a portability study on Moment-DETR with CLIP+SlowFast features, multi-seed results, and failure-mode diagnostics stratified by initial saliency quality and proposal density.
Critical weaknesses
-
This paper lacks sufficient novelty. Iterative refinement is well established in DETR-style detectors, and reciprocal interaction between saliency and localization has already been explored by TR-DETR and TaskWeave, so the claim that no prior joint framework is designed for such correction is not accurate. The main difference lies in refining the encoder memory instead of the decoder queries, while the four proposed modules together improve over the published SG-DETR by only 1.30 average mAP at a 40.7% latency cost.
-
The design of SRM raises concerns about self-reinforcing errors. The refined saliency is computed as the cosine similarity between clip features and a visual prototype pooled from the predicted spans, without involving the text query, so an incorrect first-pass prediction may raise the saliency of visually similar but irrelevant clips, and MRM may further reinforce this error.
-
The reproduced baseline is substantially weaker than the published SG-DETR, which inflates the reported gains. On QVHighlights val, the +FE baseline reaches 53.03 average mAP and 42.54 HD-mAP, compared with 55.64 and 43.91 for the published SG-DETR, yet this gap is not explained. Because all ablations, controls, and multi-seed results are measured against +FE, the claimed 3.91-point improvement shrinks to 1.30 points relative to the published SG-DETR.
Questions for authors
Since T = 3 executes the decoder three times with deep supervision at every pass, could the authors report a compute-matched control, such as three shared-decoder passes without feedback or a deeper decoder, to clarify how much of the gain comes from the proposed feedback rather than from additional computation and supervision?
Thank you for the detailed and constructive feedback, which helped us further clarify the method and strengthen the evaluation.
(1) Novelty relative to iterative DETR refinement, TR-DETR, and TaskWeave
Response. We agree that iterative decoding and MR/HD reciprocity have precedents, and Section 2 now makes this relationship explicit. Our distinction is not the existence of feedback itself, but where it acts: ReFocus-DETR uses decoder-predicted spans to revise clip saliency and the encoder memory consumed by the next weight-shared decoder pass. SRM and MRM implement this detector-state transition, while BSSI provides a complementary pre-decoder span-conditioned memory initialization. The fixed-state control further isolates this state-update mechanism from repeated decoding. Relative to the published SG-DETR row on QVHighlights validation, ReFocus-DETR improves average MR from 55.64 to 56.94 and HD-mAP from 43.91 to 44.62.
(2) Visual-only SRM prototype and possible self-reinforcing errors
Response. The SRM evidence is already query-conditioned before the final cosine operation. In Eq. 2, the prototype is constructed from query-conditioned predicted spans and their foreground confidences, even though the cosine similarity itself contains no explicit text term. The saliency update is gated from a near-identity initialization, and MRM rebuilds memory from the anchored BSSI state MB rather than recursively accumulating memory residuals. The initial-saliency analysis further shows that the complete model obtains its largest conditional gains in the lowest initial-saliency tier, indicating that the feedback design remains effective when the starting saliency is weak. We use this as evidence for the complete feedback mechanism rather than as an SRM-only robustness test.
(3) The gap between the local +FE control and the published SG-DETR score
Response. The published SG-DETR row and our local +FE row serve different purposes. The published result is the external reference in Table 1: SG-DETR reports 55.64 average MR / 43.91 HD-mAP, while ReFocus-DETR obtains 56.94 / 44.62, giving gains of +1.30 / +0.71. The local +FE row (53.03 / 42.54) is a controlled ablation baseline produced with our own detector-training and feature-extraction pipeline. For all local component studies, InternVideo2-1B features are pre-extracted under the same setup and kept frozen, and every compared variant uses the same data split, optimizer, schedule, losses, and detector training protocol. Consequently, the local baseline is not intended to numerically reproduce the published SG-DETR result; the absolute gap reflects the different controlled training/feature-extraction setting rather than a change applied selectively to our modules. Crucially, every IMR/BSSI ablation is measured against the same local +FE pipeline, while claims against SG-DETR use the published SG-DETR row. Sections 4.2 and 4.5 and Appendix G keep these two reference points explicit and separate.
(4) A three-pass control for decoder repetition and supervision
Response. We added the requested compute-matched three-pass control to separate state feedback from repeated decoding and deep supervision. The fixed-state variant uses the same three shared decoder calls and pass-wise losses as IMR, propagates the decoder query state, and holds saliency and clip memory fixed. It reaches 54.61 average MR / 42.92 HD-mAP, whereas IMR reaches 55.99 / 43.21 at the same decoder depth. The resulting gains of +1.38 average MR and +0.29 HD-mAP isolate the benefit of state feedback beyond repeated supervised decoding. Manuscript Table 11 reports this comparison; Appendix Table 13 provides complementary parameter-matched and training-only controls, and Table 9 reports the associated parameters, FLOPs, and latency.
Three-pass decoder control on QVHighlights validation (BSSI disabled)