acceptodds
Under review as a conference paper at ICLR 2027

ReFocus-DETR: Revisiting Saliency and Memory for Moment Retrieval and Highlight Detection

Abstract

Moment Retrieval (MR) and Highlight Detection (HD) provide complementary views of query-conditioned video relevance, yet most DETR-based systems connect them only once: an early saliency map reweights encoder memory, after which a decoder localizes moments from a fixed representation. Errors in this map therefore persist, while increasingly precise spans cannot repair the features that produced them. We introduce ReFocus-DETR, a closed-loop detector with two complementary feedback paths. Bidirectional Saliency–Span Interaction (BSSI) converts saliency peaks into parameter-free temporal proposals and writes their span semantics back to clip memory through position-Gaussian attention. Iterative Memory Refinement (IMR) then uses the shared decoder's current spans to revise saliency and update memory along position, channel, and intra-moment coherence axes. Near-identity gates and stop-gradient span conditioning keep the recurrent updates stable. Across QVHighlights, TACoS, Charades-STA, and TVSum, ReFocus-DETR consistently improves both localization and saliency metrics. The gains persist across feature regimes and exceed parameter-matched and training-only controls, while adding 2.6% parameters to the detector.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
DETR, Moment Retrieval (MR), Highlight Detection (HD), Iterative Memory Refinement

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VZf5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that DETR-based joint MR/HD models use saliency only once, so early saliency errors cannot be corrected by later localization. Built on SG-DETR, the proposed ReFocus-DETR introduces BSSI to write saliency-derived span evidence into clip memory, and IMR to iteratively refine saliency and memory with the decoder's current predictions through weight-shared decoder passes. Experiments on four datasets show consistent improvements on both MR and HD.

Critical strengths

The experimental analysis is relatively comprehensive. Beyond the main results on four datasets, the paper includes capacity- and supervision-controlled comparisons, a portability study on Moment-DETR with CLIP+SlowFast features, multi-seed results, and failure-mode diagnostics stratified by initial saliency quality and proposal density.

Critical weaknesses
  1. This paper lacks sufficient novelty. Iterative refinement is well established in DETR-style detectors, and reciprocal interaction between saliency and localization has already been explored by TR-DETR and TaskWeave, so the claim that no prior joint framework is designed for such correction is not accurate. The main difference lies in refining the encoder memory instead of the decoder queries, while the four proposed modules together improve over the published SG-DETR by only 1.30 average mAP at a 40.7% latency cost.

  2. The design of SRM raises concerns about self-reinforcing errors. The refined saliency is computed as the cosine similarity between clip features and a visual prototype pooled from the predicted spans, without involving the text query, so an incorrect first-pass prediction may raise the saliency of visually similar but irrelevant clips, and MRM may further reinforce this error.

  3. The reproduced baseline is substantially weaker than the published SG-DETR, which inflates the reported gains. On QVHighlights val, the +FE baseline reaches 53.03 average mAP and 42.54 HD-mAP, compared with 55.64 and 43.91 for the published SG-DETR, yet this gap is not explained. Because all ablations, controls, and multi-seed results are measured against +FE, the claimed 3.91-point improvement shrinks to 1.30 points relative to the published SG-DETR.

Questions for authors

Since T = 3 executes the decoder three times with deep supervision at every pass, could the authors report a compute-matched control, such as three shared-decoder passes without feedback or a deeper decoder, to clarify how much of the gain comes from the proposed feedback rather than from additional computation and supervision?

1 comment

Thank you for the detailed and constructive feedback, which helped us further clarify the method and strengthen the evaluation.

(1) Novelty relative to iterative DETR refinement, TR-DETR, and TaskWeave

Response. We agree that iterative decoding and MR/HD reciprocity have precedents, and Section 2 now makes this relationship explicit. Our distinction is not the existence of feedback itself, but where it acts: ReFocus-DETR uses decoder-predicted spans to revise clip saliency and the encoder memory consumed by the next weight-shared decoder pass. SRM and MRM implement this detector-state transition, while BSSI provides a complementary pre-decoder span-conditioned memory initialization. The fixed-state control further isolates this state-update mechanism from repeated decoding. Relative to the published SG-DETR row on QVHighlights validation, ReFocus-DETR improves average MR from 55.64 to 56.94 and HD-mAP from 43.91 to 44.62.

(2) Visual-only SRM prototype and possible self-reinforcing errors

Response. The SRM evidence is already query-conditioned before the final cosine operation. In Eq. 2, the prototype is constructed from query-conditioned predicted spans and their foreground confidences, even though the cosine similarity itself contains no explicit text term. The saliency update is gated from a near-identity initialization, and MRM rebuilds memory from the anchored BSSI state MB\mathbf{M}_B rather than recursively accumulating memory residuals. The initial-saliency analysis further shows that the complete model obtains its largest conditional gains in the lowest initial-saliency tier, indicating that the feedback design remains effective when the starting saliency is weak. We use this as evidence for the complete feedback mechanism rather than as an SRM-only robustness test.

(3) The gap between the local +FE control and the published SG-DETR score

Response. The published SG-DETR row and our local +FE row serve different purposes. The published result is the external reference in Table 1: SG-DETR reports 55.64 average MR / 43.91 HD-mAP, while ReFocus-DETR obtains 56.94 / 44.62, giving gains of +1.30 / +0.71. The local +FE row (53.03 / 42.54) is a controlled ablation baseline produced with our own detector-training and feature-extraction pipeline. For all local component studies, InternVideo2-1B features are pre-extracted under the same setup and kept frozen, and every compared variant uses the same data split, optimizer, schedule, losses, and detector training protocol. Consequently, the local baseline is not intended to numerically reproduce the published SG-DETR result; the absolute gap reflects the different controlled training/feature-extraction setting rather than a change applied selectively to our modules. Crucially, every IMR/BSSI ablation is measured against the same local +FE pipeline, while claims against SG-DETR use the published SG-DETR row. Sections 4.2 and 4.5 and Appendix G keep these two reference points explicit and separate.

(4) A three-pass control for decoder repetition and supervision

Response. We added the requested compute-matched three-pass control to separate state feedback from repeated decoding and deep supervision. The fixed-state variant uses the same three shared decoder calls and pass-wise losses as IMR, propagates the decoder query state, and holds saliency and clip memory fixed. It reaches 54.61 average MR / 42.92 HD-mAP, whereas IMR reaches 55.99 / 43.21 at the same decoder depth. The resulting gains of +1.38 average MR and +0.29 HD-mAP isolate the benefit of state feedback beyond repeated supervised decoding. Manuscript Table 11 reports this comparison; Appendix Table 13 provides complementary parameter-matched and training-only controls, and Table 9 reports the associated parameters, FLOPs, and latency.

Three-pass decoder control on QVHighlights validation (BSSI disabled)

ModelTAvg. MRHD-mAP
+FE153.0342.54
+3-pass fixed state354.6142.92
+IMR355.9943.21
Reviewer_ThFgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReFocus-DETR addresses the one-way dependence in which saliency affects encoder memory but later localization cannot revise that state. BSSI extracts saliency-derived spans and writes their semantics back to memory; IMR repeatedly updates saliency and memory from a shared decoder's predictions. Results cover four datasets and include multiseed, module, parameter-matched, and training-only supervision controls.

Critical strengths
  • The motivation is clear and reasonable: initial saliency influences localization, while localization results cannot normally revise saliency and memory. Table 18 supports this motivation by showing larger gains on examples with poorer initial saliency.

  • The design aligns with the identified problem. BSSI incorporates semantics from saliency-derived proposals, while IMR uses localization predictions to update saliency and memory. The modules have clear intended roles and receive support from component ablations.

  • Results are competitive on both tasks. Compared with SG-DETR in Table 1, average MR mAP improves from 55.64 to 56.94 and HD-mAP from 43.91 to 44.62 on QVHighlights validation, with improvements also reported on the test split.

Critical weaknesses
  • The iteration-depth analysis is limited to aggregate metrics. Average MR mAP decreases slightly from 55.99 at T=3 to 55.85 at T=4, but the affected cases remain unclear. Case-wise trajectories and breakdowns by event count, duration, and distractors would help explain the change and establish whether it exceeds random variation.

  • Quantitative ablations support the modules, but qualitative evidence of their operation is limited. Existing visualizations show final outputs rather than changes introduced by BSSI or successive IMR passes. Successful and failed trajectories would help verify the intended correction mechanisms.

  • The overall feedback motivation is reasonable, but several detailed choices remain heuristic, including proposal widths, shared prototypes/centroids, and gating. Existing sensitivity tests do not fully establish robustness to incorrect proposals, repeated events, or multiple events. The proposal-IoU drop from 0.68 to 0.43 with increasing peak density motivates stronger stress tests and simpler alternatives.

Questions for authors

see Weaknesses

1 comment

Thank you for the careful analysis of the iterative behavior and for raising important questions about decoder depth, qualitative interpretation, and robustness under harder cases.

(1) Pass-wise behavior, the T=3T=3 to T=4T=4 change, and hard-case strata

Response. At T=3T=3, the model reaches 55.99 average MR / 43.21 HD-mAP; at T=4T=4, these become 55.85 / 43.25. The 0.14-point MR difference is of the same scale as the 0.16 standard deviation measured across five seeds at the selected T=3T=3 setting. Since T=3T=3 gives the higher average MR with one fewer decoder call, we use it as the accuracy-efficiency operating point. We do not attribute the small T=4T=4 change to unmeasured factors such as event count or duration; instead, Appendix F reports initial-saliency and peak-density strata that directly probe the feedback mechanism under harder inputs.

(2) Qualitative trajectories, including correction and failure

Response. Figure 3 and Appendix H present final-output visualizations only and are not used as evidence of pass-wise iterative trajectories. To evaluate the feedback mechanism directly, Table 11 includes a three-pass fixed-state control that retains the same shared decoder calls and pass-wise supervision while holding saliency and clip memory fixed. It reaches 54.61 average MR / 42.92 HD-mAP, compared with 55.99 / 43.21 for IMR. This controlled comparison isolates the benefit of updating the detector state beyond repeated supervised decoding.

(3) Width, shared prototypes/centroids, gates, repeated events, and incorrect proposals

Response. We evaluated the main design choices that control the amount and reliability of span evidence. Appendix Tables 17-18 vary the BSSI width window and the numbers of BSSI proposals and IMR predictions; performance remains stable across the tested ranges, with the default compact evidence set giving the best overall scores. Appendix Table 20 further stratifies videos by the number of valid saliency peaks. Although proposal IoU decreases as peak density increases, the BSSI HD-mAP gain remains positive in every group, including +0.81 for videos with six or more valid peaks. The small initialized gates constrain update magnitude, while the shared SRM prototype and MRM centroid aggregate evidence from the selected spans. Section 4.5 reports these diagnostics together with the corresponding operating scope.

Reviewer_28FaOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReFocus-DETR, a closed-loop refinement framework for joint moment retrieval (MR) and highlight detection (HD). Building on a DETR-style joint architecture, the method addresses the one-way dependency in which an initial saliency estimate modifies the encoder memory, but the decoded moments cannot subsequently correct the state that produced them. The proposed framework contains two main components. BSSI uses peaks from the initial saliency map to construct temporal proposals, pools span-level semantics, and injects this information into the clip memory. IMR repeatedly applies a shared decoder and uses the current predictions to refine saliency and encoder memory through two interacting refinement modules. The resulting state is used by the next decoding iteration. The method is evaluated on QVHighlights, TACoS, Charades-STA, and TVSum, together with component, refinement-depth, feature, portability, and computational-cost analyses. The reported results show that combining closed-loop decoding with memory refinement improves both MR and HD over the corresponding one-pass baselines.

Critical strengths
  1. The paper introduces a coherent closed-loop architecture that updates encoder memory using information from decoded predictions. In contrast to a one-way pipeline, the shared decoder and the two refinement paths allow later predictions to revise the representation used by subsequent decoding iterations. This is the main technical reason supporting acceptance.
  2. The empirical evaluation is extensive and generally supports the effectiveness of combining closed-loop decoding with representation refinement. The paper includes component and iteration-depth ablations, evaluations on multiple MR/HD datasets, an additional feature/backbone setting, repeated-seed results on QVHighlights, and explicit computational-cost measurements. These experiments make the overall empirical improvement credible, even though the contribution of BSSI itself is not yet sufficiently isolated.
Critical weaknesses
  1. The mechanism attributed to BSSI is not directly validated. The paper argues that BSSI uses saliency peaks to incorporate span semantics into clip features, but the reported ablations do not include a BSSI-only configuration. The main comparison adds BSSI on top of IMR, and the refinement-depth experiment at T=1 does not appear to include BSSI. Consequently, the results establish that IMR+BSSI is better than IMR in the tested configuration, but they do not clearly show whether BSSI is independently effective or whether the proposed span-to-clip interaction is the reason for the gain.
  2. The relationship between BSSI and the subsequent iterative refinement mechanism is unclear. BSSI appears to operate on the initial, uncorrected saliency estimate, whereas IMR updates saliency and encoder memory later in the loop. The paper does not provide a mechanism-level explanation or intervention showing that these components interact rather than simply contribute separate effects. This is particularly important when the initial saliency is inaccurate, because BSSI could inject incorrect proposals into the memory and potentially amplify the initial error.
Questions for authors
  1. Could the authors provide a BSSI-only ablation under the same backbone and training protocol, together with a T=1 result that includes BSSI? This would clarify whether BSSI is independently beneficial rather than only useful when added to IMR.
  2. What direct evidence supports the claim that BSSI successfully injects useful span semantics into the clip representation? For example, can the authors compare the current span-conditioned update with a parameter- or compute-matched update using saliency peaks without span pooling, or otherwise measure whether the updated clip features contain more accurate span information?
  3. How does BSSI behave when the initial saliency estimate is inaccurate? Please report performance stratified by initial-saliency quality, or an intervention that degrades or replaces the initial saliency. Would using the saliency refined by IMR, rather than the uncorrected initial saliency, change the result?
  4. Is the improvement from combining BSSI and IMR larger than the sum of their independent effects? Please explain the intended mechanism-level interaction between BSSI, SRM and MRM. This interaction is central to determining whether the method is a unified closed-loop design or two separately useful additions.
2 comments

Thank you for the constructive questions and for highlighting the need to more clearly isolate the contribution of BSSI and its interaction with iterative refinement.

(1) BSSI-only and T=1T=1 with BSSI

Response. We added the requested single-pass BSSI control under the same local +FE setting used for the component study. BSSI reaches 54.12 average MR / 43.07 HD-mAP versus 53.03 / 42.54 for +FE, giving gains of +1.09 / +0.53 points. With IMR, adding BSSI raises 55.99 / 43.21 to 56.94 / 44.62. These results separate BSSI's standalone contribution from its contribution within the complete model. The T=1T=1 row in the depth study remains +FE without BSSI because that ablation isolates decoder depth. The corresponding controlled variants are reported together in Table 11.

(2) Evidence that pooled span semantics, rather than peaks alone, improve clip memory

Response. We directly isolate the descriptor choice with Peak-BSSI. It retains the selected peaks, proposal widths, attention, gate, and decoder schedule, but replaces each interval-pooled span descriptor with the feature at the selected peak. With IMR, Peak-BSSI obtains 56.51 average MR / 43.91 HD-mAP, whereas interval-pooled BSSI obtains 56.94 / 44.62. Because descriptor construction is the changed factor, the gains of +0.43 average MR and +0.71 HD-mAP directly support the contribution of pooled span semantics to the BSSI update. These variants are reported in Table 11 and summarized below.

(3) Inaccurate initial saliency and conditioning BSSI on refined saliency

Response. To directly assess sensitivity to the initial saliency, we stratify QVHighlights validation examples by initial-saliency AP in Appendix Table 19. From the bottom to the top third, adding BSSI to IMR improves [email protected] by 2.96 / 1.99 / 0.93 points and HD-mAP by 1.92 / 1.42 / 0.89 points. The full model follows the same trend, with its largest gains when the initial saliency is weakest. Appendix Table 20 further shows that BSSI retains a positive HD-mAP gain in every evaluated peak-density group. In the proposed architecture, BSSI is deliberately a pre-decoder initialization from s0\mathbf{s}_0; recurrently reapplying BSSI to IMR-refined saliency would define a different computation graph rather than the evaluated BSSI module.

Initial-saliency AP groups on QVHighlights validation

Initial AP tierNFull vs. +FEBSSI vs. +IMR
Δ [email protected]Δ HD-mAPΔ [email protected]Δ HD-mAP
Bottom third517+6.12+3.27+2.96+1.92
Middle third516+3.58+1.91+1.99+1.42
Top third517+1.24+0.72+0.93+0.89

(4) Whether BSSI and IMR interact beyond independent improvements

Response. The controlled comparison shows that BSSI is beneficial both alone and with IMR. BSSI improves +FE by +1.09 average MR / +0.53 HD-mAP; when combined with IMR, it adds +0.95 / +1.41 points and yields the best scores, 56.94 / 44.62. The intended interaction is sequential rather than an arithmetic super-additivity claim. BSSI first supplies span-conditioned memory MB\mathbf{M}_B; SRM then revises saliency from decoded spans, and MRM rebuilds the memory for the next pass from MB\mathbf{M}_B under that revised saliency (Eq. 1). Thus later decoder passes consume the span context introduced by BSSI together with updated localization evidence.

Controlled QVHighlights validation comparisons

ModelTAvg. MRHD-mAP
+FE153.0342.54
+BSSI154.1243.07
+IMR355.9943.21
+IMR+Peak-BSSI356.5143.91
+3-pass fixed state354.6142.92
+IMR+BSSI356.9444.62

Thank you for the detailed explanations and additional experiments, which clarified how BSSI contributes on its own and why pooling features across a span helps. The revised analysis also gave me a clearer understanding of how BSSI works with IMR in the full model. The new evidence addresses my main concerns, and I am raising my rating from 3 to 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.