acceptodds
Under review as a conference paper at ICLR 2027

Structured Posterior Decoding over Contextualized Span Lattices for Video Moment Retrieval

Abstract

A central challenge in video moment retrieval is assigning reliable relevance scores among many plausible temporal spans. Recent work in audio moment retrieval showed that exact segment posteriors, even with simple boundary-based potentials, can provide an effective structured alternative to proposal confidence. We observe, however, that when this formulation is transferred directly to video, its gains diminish as more target moments must be retrieved. We attribute this to the different nature of temporal evidence in video: relevant visual semantics are often distributed throughout a moment, while boundary transitions tend to be less distinct. To address this, we contextualize the two-dimensional span lattice using query-conditioned local 2D convolutions, allowing each candidate to aggregate span-interior semantics and information from neighboring temporal hypotheses before structured inference. We then compute exact segment marginals and use them to form a posterior-weighted moment prototype, whose similarity to each candidate iteratively refines the span potentials. Under matched feature settings, our approach substantially improves multi-moment retrieval over prior methods on QV-M, and with InternVideo2-6B features it establishes a new state of the art on this benchmark. The gains carry over to QVHighlights and Charades-STA, while a cross-modal analysis on audio suggests that the benefit of span-lattice contextualization depends on modality-specific temporal evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video moment retrieval, multi-moment retrieval, segmental conditional random field, temporal grounding

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_N9jEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-moment retrieval in video and proposes a structured posterior decoding framework. It first performs query-conditioned contextualization over a 2D span lattice, and then iteratively refines span potentials using exact segment posteriors to better distinguish and rank multiple candidate moments. The method builds on segmental posterior decoding previously used in audio moment retrieval, and is evaluated on QV-M2, QVHighlights, and Charades-STA.

Critical strengths

One interesting insight of the paper is that MMR should not rely solely on independently scoring candidate spans, but should jointly model their local relationships and global competition, with neighboring spans providing contextual information and the structured posterior further constraining their competition under complete temporal explanations.

Critical weaknesses
  • In the Introduction, the authors mainly divide existing methods into proposal-based approaches and dense span scoring approaches, but the discussion mostly describes how these formulations differ from the proposed method. For proposal-based methods, the paper argues that retrieval coverage is constrained by candidate generation, but it does not explain why better proposal generation, boundary regression, or subsequent verification cannot address this issue. For dense span methods, the discussion mainly focuses on 2D-TAN, while approaches such as SnAG can avoid enumerating all start-end pairs while still maintaining good coverage, yet this line of work is not discussed. Existing MMR-specific methods such as FlashMMR are mentioned, but only briefly, and I do not see a concrete failure mode that naturally motivates structured posterior decoding. Overall, the paper does not clearly establish why existing methods are insufficient.

  • The core structured posterior decoding framework is directly built on prior work in audio moment retrieval, as the authors explicitly state. The 2D span lattice and neighboring-span contextualization are also closely related to the temporal map formulation used in 2D-TAN and MS-2D-TAN. The main new component appears to be the posterior-conditioned refinement, together with the integration of these components for video MMR. Overall, the method feels more like an adaptation and combination of existing structured decoding and 2D span modeling than a new structured formulation, which limits the methodological novelty.

Questions for authors
  • Can the authors clarify what makes structured posterior decoding specifically necessary for MMR, and what limitation of existing proposal-based or dense span methods it addresses that cannot be handled by stronger candidate generation, regression, or post-processing?

  • Can the authors clarify the main methodological novelty of this work and distinguish it more clearly from the prior methods it builds on?

3 comments
AuthorsOpenReview ↗

Response to Reviewer N9jE

We thank the reviewer for the thoughtful review and for highlighting the central insight of our work. Please also refer to the General Response, particularly GR0 for the corrected CLIP features and GR2–GR3 for matched-feature comparisons and statistical uncertainty analyses. Below, we address the reviewer’s two methodological questions using the corrected results.

R1. Why is structured posterior decoding needed beyond better proposals, regression, or post-processing?

Improving proposal quality and post-processing is important for multi-moment retrieval, but the challenge lies not only in generating accurate candidates, but also in assigning relevance scores when multiple temporal hypotheses compete. Existing methods, including FlashMMR, address candidate quality through boundary refinement and post-verification. However, their final relevance scores are assigned to individual candidates rather than jointly normalized over complete temporal explanations.

This distinction is reflected in the empirical results. Despite FlashMMR's dedicated multi-moment refinement and verification, its reported mAP@3+tgt is 22.89, compared with 20.19 for FlashVTG. Under matched CLIP+SlowFast features, our method achieves 27.34 mAP@3+tgt, compared with 22.07 for retrained FlashMMR, with a statistically supported improvement of +5.27 [95% CI: +2.73, +6.92]. This suggests that structured scoring provides benefits beyond improving individual candidates.

Our formulation introduces global competition through a distribution over complete temporal segmentations. Overlapping hypotheses share probability mass, while disjoint moments can coexist. NMS remains useful for removing redundant predictions, and we apply it to our method as well. The distinction is that structured competition is incorporated into the scoring process before NMS, rather than relying on candidate-level scores and subsequent post-processing alone.

To isolate this effect, we additionally compare independent local scoring and structured posterior decoding over the same exhaustive span lattice with identical NMS settings (Table 2). Base (SPD) improves over Base (local-ψ) from 18.62 to 33.87 G-mAP and from 4.62 to 20.25 mAP@3+tgt. Similarly, Full improves over Full (local-ψ) from 28.65 to 37.26 G-mAP and from 17.87 to 27.34 mAP@3+tgt. These controlled comparisons demonstrate the importance of global normalization beyond candidate coverage and local contextualization.

We have also added SnAG to Related Work. Its multi-scale anchor-based regression addresses candidate generation and scalability, which are complementary to our focus on structured relevance assignment.

Revised manuscript: Section 2, Section 5.2, Tables 1–2, Appendices A.2.7 and A.3.

R2. What is methodologically novel relative to the methods we build on?

We acknowledge that segmental posterior decoding and 2D temporal span maps are established components. Our contribution lies in integrating local span contextualization and posterior-conditioned refinement within a globally normalized structured model, supported by controlled experiments that distinguish their roles.

First, we identify a limitation of directly transferring boundary-based segmental decoding to video. Its advantage over DETR confidence decreases from 8.3 AP on single-target queries to 3.4 AP on queries with three or more targets (Section 5.2), motivating richer segment potentials.

Second, unlike 2D-TAN and MS-2D-TAN, which use contextualized span maps to produce independently scored candidates, our 2D contextualizer modifies segment potentials before global normalization. This allows neighboring-span information to influence structured inference while preserving exact segment marginals. The Full (local-ψ) comparison in Table 2 empirically distinguishes these formulations.

Third, we introduce posterior-conditioned refinement, which feeds exact segment marginals back into the potential construction through a posterior-weighted prototype. To evaluate whether the structured posterior contributes beyond iterative refinement itself, we added controlled experiments comparing posterior-marginal weighting with uniform weighting and local softmax weighting, while holding the 2DCon architecture, candidate set, and refinement iterations fixed (Table 4). Posterior-marginal weighting achieves the highest mean G-mAP and improves top-1 localization metrics. We have also clarified that 2DCon accounts for most of the multi-target gains, while ITRef provides a smaller additional benefit when applied on top of 2DCon.

Revised manuscript: Sections 3.2–3.3, Sections 5.2–5.4, Tables 2–4, and Appendix A.7.

We hope these clarifications and additional experiments address the reviewer's concerns about the necessity of structured decoding and the methodological contribution of our framework.

Although the additional experiments demonstrate the effectiveness of the proposed method, my main concern regarding its motivation remains insufficiently addressed. The authors primarily explain the differences between candidate-level relevance scoring and jointly normalized structured posterior scoring, but do not sufficiently clarify why the latter is better suited to MMR.

Specifically, the authors emphasize competition among overlapping candidates, which is also what conventional NMS aims to address. While the authors explain that the two mechanisms operate at the scoring and post-processing stages, respectively, what substantive advantages does incorporating such competition into the scoring process offer? What limitations remain in existing methods after applying NMS? Given that the proposed method itself still employs NMS, the necessity of introducing structured posterior decoding remains unclear to me.

I acknowledge the performance improvements demonstrated by the additional experiments. However, I believe the authors still need to better articulate the limitations of existing methods and why the proposed framework can overcome them, rather than merely emphasizing differences in formulation and improvements in empirical performance.

We thank the reviewer for the thoughtful follow-up and for acknowledging the additional experiments. We agree that our previous response did not sufficiently clarify why structured posterior scoring is advantageous beyond NMS. We address this concern directly below.

The fundamental distinction lies in how relevance scores are obtained, not merely when overlapping predictions are suppressed. Conventional proposal-based methods predict individual moments with independently assigned confidence scores. NMS then suppresses candidates based on temporal IoU and these fixed scores, without reconsidering the visual evidence supporting alternative temporal explanations. This leads to two concrete failure modes in MMR.

  1. Incorrect suppression of distinct moments. Consider two occurrences of a query separated by a short gap, together with an over-extended candidate spanning both. Because the merged candidate contains relevant evidence from both occurrences, it may receive a high confidence score. With a permissive NMS threshold, the merged candidate may survive alongside the correct predictions. With an aggressive threshold, a high-confidence merged candidate may instead suppress correctly localized predictions. NMS cannot determine whether the intervening gap should be treated as background, because its decisions depend on confidence rankings and temporal overlap, without considering how the entire video could be segmented into distinct relevant moments and background regions.

  2. Incorrect ranking of surviving moments. Partially overlapping or over-extended variants of a salient moment may survive NMS when their IoU falls below the threshold. These candidates retain their original confidence scores and may rank above weaker but distinct relevant moments. This is particularly consequential in MMR, where multiple distinct targets must be recovered and redundant predictions can displace correct predictions from higher ranks.

Both problems arise because NMS operates on already independently assigned confidence scores.

In contrast, our method defines a probability distribution over all valid segmentations of the entire video timeline and computes each candidate's exact marginal probability across these globally competing explanations. A merged span and two separate occurrences represent alternative segmentations whose probabilities depend on the learned segment potentials and background evidence, including the intervening gap. Overlapping spans compete for probability mass because they cannot coexist within a segmentation, while disjoint moments can be jointly supported.

This is the substantive advantage for MMR: a candidate's relevance reflects its support across complete, structurally compatible explanations of the timeline, rather than an independently assigned confidence score followed by geometric suppression. Our contextualized span potentials provide the visual evidence needed to distinguish competing hypotheses, while global normalization incorporates their compatibility into the relevance scores.

Additional experimental evidence. Structured marginals do not eliminate all redundant predictions, so NMS remains useful as complementary post-processing. To directly examine the reviewer's concern, we additionally evaluated Full (local-ψ) without NMS, complementing the NMS sensitivity results already reported for Full (SPD) in Table 14 and the local-ψ results in Table 20. Both formulations share the architecture and span lattice, while differing in their training objectives and scoring rules.

QV-M² test results (three-seed mean ± std):

ScoringNo NMS (G-mAP / @3+tgt)NMS 0.7 (G-mAP / @3+tgt)
Full (local-ψ)24.07 ± 0.49 / 11.57 ± 0.9528.65 ± 1.14 / 17.87 ± 0.14
Full (SPD)33.86 ± 0.41 / 22.67 ± 1.3837.26 ± 0.72 / 27.34 ± 0.75

Even without NMS, Full (SPD) outperforms Full (local-ψ), which independently scores each span and applies NMS, by +5.21 G-mAP and +4.80 mAP@3+tgt. Without NMS, the structured formulation also exceeds the local formulation by +9.79 G-mAP and +11.10 mAP@3+tgt. NMS improves both formulations, confirming its complementary role. However, the performance advantage of structured posterior scoring is already substantial before NMS is applied.

These results demonstrate that NMS alone does not recover the performance advantage of structured posterior scoring. Although the comparison also involves differences in training objectives, it directly shows that the improvements cannot be attributed merely to post-processing. Rather than relying solely on suppressing predictions after confidence-based ranking, our framework incorporates competition among complete temporal explanations into relevance scoring itself.

We hope this clarifies why structured posterior decoding is particularly well suited to MMR. We would be happy to provide further clarification or address any remaining concerns the reviewer may have.

Reviewer_smifOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper frames Video Moment Retrieval as a structured ranking problem based on exhaustive temporal segmentation. First, it constructs a 2D "start-end" interval grid using query-conditioned convolutions, incorporating both the semantic information contained within each interval and contextual data from neighboring candidate intervals. Subsequently, it employs forward–backward inference to compute precise segment marginals and iteratively refines span potentials via posterior-weighted representation prototypes. The authors evaluate this approach on the QV-M2, QVHighlights, and Charades-STA datasets, and conduct cross-modal comparative experiments on the CASTELLA audio dataset. The results demonstrate performance improvements in multi-moment retrieval tasks; furthermore, when utilizing InternVideo2-6B features, the method achieves strong performance on the QV-M2 dataset.

Critical strengths
  1. The problem and motivation of this paper are clearly defined. To address the competition among multiple overlapping candidate results in multi-moment retrieval, the authors propose an approach that combines globally normalized segment posterior probabilities with local span-lattice context. Each module is initialized with zeros, which facilitates the analysis of the incremental contribution of individual components.
  2. Ablation studies cover the paper’s primary components. The authors evaluate mechanisms such as DETR confidence scoring, structured posterior decoding, span-lattice context modeling, and iterative refinement, providing experimental results based on three different random seeds. The performance improvements observed under the QV-M2 multi-objective evaluation metric are significant and warrant further investigation.
Critical weaknesses

The basis for the claims regarding SOTA (state-of-the-art) performance and the results from multi-seed experiments is not clearly articulated. For the QV-M2 CLIP+SlowFast configuration, Table 1 presents a G-mAP of 36.89 and an @3+tgt score of 28.40 for the "best-performing model." Table 2 provides the averages from three separate runs with different random seeds (34.95 and 26.86, respectively), with corresponding standard deviations (1.94 and 1.34) included in the appendix. The average G-mAP from these three runs is slightly lower than the 35.14 reported for FlashMMR. The paper should explicitly define what constitutes the "best-performing model," explain how this result was selected, and clarify whether the selection process relied solely on the validation set. Such details are essential for interpreting the comparative results and the reported performance gains. In the InternVideo2-6B/LLaMA configuration, the comparison between the Base (SPD) and Full versions in Table 2—showing an increase in G-mAP from 34.51 to 38.43—demonstrates that the proposed module outperforms the "structured posterior decoding" (SPD) approach when using identical features. However, Table 1 does not specify whether prior methods utilized the same feature configurations. Consequently, while the InternVideo2 results reflect the performance of the overall system—integrating both the proposed method and the specific features—they fail to isolate the distinct advantages of the method itself, independent of the feature representation. The QV-M2 test set is relatively small and exhibits a significant shift in the distribution of target counts; yet, the paper does not report the uncertainty associated with the evaluation samples. According to the appendix, the test set consists of 323 queries and 173 videos, with an average of 3.31 targets per query, whereas the corresponding figure for the training and validation sets is approximately 1.96–1.98. The paper reports the mean and standard deviation based on three random seeds—reflecting variability across different training runs—but fails to provide confidence intervals that would capture uncertainty related to the test samples. Furthermore, the study does not specify the number of queries included in each "number-of-targets" group. Since the multi-target metric assigns equal weight to these different groups, knowing the number of queries in each is crucial for determining their respective contributions to the aggregated results. Without this information—or an uncertainty analysis conducted at the query or video level—it is difficult to assess the robustness of the reported performance gains with respect to the composition of the test set.

Questions for authors
  1. What exactly is meant by the "best-performing model" in Table 1? Does it refer to the optimal configuration, or the result obtained using the best random seed? Is model selection based exclusively on the validation set? Please provide the specific values for the metrics shown in Table 2 across different random seeds, and clarify the distinction between these two tables. Furthermore, would it be possible to consistently present all baseline comparison results using the mean and uncertainty estimates derived from three independent random trials?
  2. For the QV-M2 test set, how many queries and videos are included in each "target-count" group? Could the authors calculate bootstrap confidence intervals—based on either queries or videos—for the G-mAP, @2tgt, and @3+tgt metrics to assess the stability of the reported performance gains?
  3. The datasets, feature encoders, and training configurations used for CASTELLA (audio) and QV-M2 (video) differ. As the paper acknowledges, this comparison cannot isolate the inherent effects of the modalities themselves. Given this, should the claim regarding "modality-specific temporal evidence" be framed as an "observational hypothesis" rather than a conclusion definitively established by the current experiments?
1 comment
AuthorsOpenReview ↗

Response to Reviewer smif

We thank the reviewer for the detailed and constructive review. Please refer to our General Response (GR0–GR3) for the feature correction, model selection protocol, matched-feature comparisons, and statistical uncertainty analyses. Below, we address the reviewer's specific concerns and identify the corresponding manuscript revisions.

R1. “Best-performing model,” per-seed reporting, and baseline comparisons.

We agree that the original reporting could obscure the distinction between model selection and test-set variation. As clarified in GR1, all checkpoints, architectural choices, and decoding settings are selected using validation data only. The “best” row in Table 1 is simply the seed with the highest test G-mAP and is retained only for reference. All claims in the revised manuscript now use three-seed means.

We also retrained QD-DETR and FlashMMR with the same features, evaluator, validation-based checkpoint rule, and three seeds used for our method. Under CLIP+SlowFast, QD-DETR† obtains 27.65 ± 1.55 G-mAP and FlashMMR† obtains 32.02 ± 0.24, compared with 37.26 ± 0.72 for our method. Under InternVideo2+LLaMA, FlashMMR† obtains 35.96 ± 1.26 compared with 38.43 ± 0.73 for ours. This matched-feature comparison separates the contribution of the proposed modeling from that of stronger pretrained features.

Revised manuscript: Section 5.1, Table 1, Appendices A.2.6 and A.5.

R2. Target-count imbalance and test-set uncertainty.

We agree that the QV-M² test split is small and that mAP@3+tgt is sensitive to sparsely populated target-count groups. The revised manuscript therefore reports the number of queries and videos for every target count and adds paired bootstrap confidence intervals using video-level resampling, which keeps queries from the same video together.

With CLIP+SlowFast, Full improves over Base (SPD) by +3.39 G-mAP [95% CI: +2.08, +4.76] and +7.09 mAP@3+tgt [+4.22, +8.92]. Relative to FlashMMR†, the corresponding gains are +5.24 [+3.37, +7.15] and +5.27 [+2.73, +6.92]. The mAP@2tgt improvement over Base (SPD) is not statistically established, which we now state explicitly.

We also tested the sensitivity of mAP@3+tgt to its group weighting. The three groups with only one or two queries account for 30% of the official mAP@3+tgt average. When these groups are removed, or when queries with at least three targets are weighted equally, the CLIP+SlowFast gains remain positive with confidence intervals excluding zero. We therefore retain the official metric for comparability while reporting these robustness analyses alongside it.

Revised manuscript: Appendix A.3, Tables 16–18.

R3. Interpretation of the cross-modal comparison.

We agree that the original claim was too strong. The audio and video experiments differ in datasets, feature encoders, and training configurations, so the comparison cannot isolate the effect of modality itself. We have revised the Abstract and Introduction to present the observed contrast as a hypothesis rather than an established conclusion. Section 5.5 now explicitly acknowledges these confounding factors and limits the interpretation to the evaluated settings. The Conclusion likewise frames modality-dependent temporal evidence as a hypothesis motivating future work on modality-adapted segment potentials.

Revised manuscript: Abstract, Introduction, Section 5.5, and Conclusion.

We thank the reviewer again for these comments, which helped improve the rigor of our experimental reporting and the scope of our claims.

Reviewer_XkYcOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper tries to approach video moment retrieval, and in particular the multi-moment setting of QV-M2, as a structured ranking problem rather than scoring proposals with independent confidences. It starts from a recent audio moment retrieval result, where exact segment marginals from a globally normalized model over complete temporal segmentations replaced DETR confidence scores, and observes that this gain shrinks on video as the number of target moments grows. The authors argue that video evidence lives inside the span rather than at its boundaries, and from there they add two pieces: a query-conditioned 2D convolution over the start-end span lattice so each candidate sees its content and its neighbors before normalization, and an iterative refinement where the exact posterior builds a weighted moment prototype whose similarity is fed back into the span potentials. The method reports clear improvements on QV-M2, QVHighlights and Charades-STA over its baselines.

Critical strengths
  1. The paper is well written, easy to follow, and positioned well against recent work.

  2. I like the idea. It is sound and simple, clearly fits the problem, and achieves good results.

  3. The evaluation is comprehensive, covering a good amount of baselines and benchmarks.

Critical weaknesses

Hyperparameter sensitivity: The method introduces new choices (e.g., NMS threshold and candidate pool) whose sensitivity is not shown, and the baselines do not have these settings. Also, it is not clear whether the kernel size and iteration count were selected from validation- or test-split?

Efficiency: There is no parameter, FLOPs, or latency comparison against the baselines.

Motivation: The 2D module is motivated by span-interior semantics, but the 1x1 rows in Table 3 are exactly that and do not help, which seems to contradict the motivation?

Questions for authors
  1. Can the authors report sensitivity to the NMS threshold, the candidate pool size, the kernel size, and the iteration count, with the choices selected on the validation split?

  2. Can the authors provide a cost comparison (parameters, FLOPs, and latency per query) against QD-DETR and FlashMMR, and show how it scales with the number of clips? That will be more useful to know the trade-offs of the method.

  3. Table 3 shows that span-interior content alone (1x1) does not help and only the 3x3 kernel does, which contradicts the span-interior motivation of the 2D module. What is the justification for the stated mechanism?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XkYc

We thank the reviewer for the positive assessment and for finding the idea sound and simple, the paper easy to follow, and the evaluation comprehensive. Please refer to our General Response for the feature correction (GR0) and model selection protocol (GR1). Below, we address the reviewer's specific questions and summarize the corresponding manuscript revisions.

R1. Sensitivity to NMS, candidate pool, kernel size, and refinement iterations.

We have added sensitivity analyses for all four settings in Tables 10, 14, and 15 and Appendices A.2.4 and A.2.9. All settings were selected exclusively using validation performance, as clarified in GR1. The selected NMS threshold of 0.7, candidate pool of 50, 3×3 kernel, and two refinement iterations achieve the highest validation G-mAP among the evaluated configurations. The results also show that our improvements are robust to alternative NMS thresholds and candidate-pool sizes.

R2. Parameters, FLOPs, and latency.

We added a computational-cost comparison with QD-DETR and FlashMMR in Appendix A.4 (Table 19). Full introduces only 3.5% more parameters and approximately 30% more FLOPs than QD-DETR. The main overhead is inference latency due to sequential forward-backward inference and repeated refinement.

Nevertheless, inference remains sub-second even for a five-minute video (314.9 ms at (L=300), batch size 1, excluding feature extraction and NMS). At batch size 32, the amortized latency is 2.71 ms per query for (L=75). For latency-sensitive applications, 2DCon without refinement substantially reduces latency while preserving most of the multi-target gains.

We now explicitly discuss this computational trade-off and its implications for longer videos in Sec. 6.

R3. Why does span-interior content alone not help?

We agree that our original motivation was imprecise. ITRef already incorporates span-interior content through the mean representation of encoder states within each candidate span (Eq. 9). Therefore, a 1×1 contextualizer largely adds information already available to the model, consistent with its small improvement over ITRef alone.

The key benefit comes from relational context among neighboring span hypotheses, rather than span-interior content alone. Kernels operating along the end-time and duration axes allow overlapping candidates with different boundaries or durations to exchange information before global normalization.

As shown in Table 3, mAP@3+tgt increases from 19.72 with 1×1 to 24.60 with end-time neighbors and 27.34 with 3×3, whereas the capacity-matched 1×1 achieves only 20.69. These results indicate that neighboring-span interactions, rather than additional model capacity, drive the improvement. We have clarified this interpretation in Sec. 5.3.

We thank the reviewer again for the constructive feedback.

Reviewer_XkYcOpenReview ↗

Review

Thanks for the detailed response and additional experiments. The core idea is sound and well-validated, though inference latency remains an interesting, non-trivial challenge. I recommend exploring architectural or engineering improvements to address this in future work. All my concerns are resolved, and I am raising my score to 4.

Thank you! We are glad that our response addressed your concerns. We also appreciate your constructive suggestion regarding computational efficiency, which we agree is an important direction for future work.

Reviewer_rZviOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends segmental posterior decoding to multi-moment video retrieval. Query-conditioned 2D convolutions contextualize a dense span lattice, while a posterior-weighted prototype iteratively refines segment potentials. Exact forward-backward marginals provide retrieval scores, followed by temporal NMS. Experiments on QV-M2, QVHighlights, and Charades-STA support improvements over the structured baseline, especially for multi-target queries. Additional CASTELLA experiments explore differences between audio and video retrieval.

Critical strengths
  1. The framework coherently combines neighboring-span context with structured competition while preserving exact marginal inference under the final potentials.
  2. Multi-target gains are substantial: Full improves QV-M2 mAP@3+tgt from 20.98 to 26.86 over Base (SPD). Capacity-matched kernel ablations support the value of neighboring-span interactions, and improvements persist across feature configurations and additional video benchmarks.
Critical weaknesses
  1. The headline comparison does not establish a robust overall advantage over the strongest prior method under matched features. Table 1 reports best-model G-mAP of 36.89, whereas the three-seed mean is 34.95 ± 1.94, compared with FlashMMR’s 35.14. Across-run model selection is unclear, and the InternVideo2 comparison changes both encoders without equivalently equipped competitors.

  2. Key controls are insufficiently specified to establish the claimed mechanism. The local-ψ baselines’ objectives, background handling, and refinement weights are insufficiently specified. Refinement adds only 0.78 mean G-mAP over 2DCon, and no matched alternative prototype-weighting control establishes the specific benefit of structured posterior feedback.

Questions for authors
  1. Clarification of model selection: How was the best-performing model in Table 1 selected across seeds and configurations? Was every selection decision based exclusively on validation data? Please distinguish this from the within-run DETR validation [email protected] checkpoint rule.
  2. Evidence relevant to the performance claim: Can you provide a comparison with FlashMMR using the same features, evaluation pipeline, and reporting convention, with uncertainty estimates? Please also explain the difference between the QD-DETR result in Table 1 (28.95 G-mAP) and Base (DETR) in Table 2 (25.48), including whether the latter is a separately trained model or a decoding branch of a jointly trained model.
  3. Clarification of the local baselines: What are the exact objectives, ranking scores, background terms, and prototype-weight definitions for Base (local-ψ) and Full (local-ψ)? Which parameters, auxiliary losses, and post-processing settings are shared with their SPD counterparts?
  4. Targeted evidence for refinement: Does ITRef still improve results when compared with the same refinement architecture using uniform weights or normalized local scores instead of structured marginals? Paired per-seed results for 2DCon and Full, particularly G-mAP and mAP@3+tgt, would help determine whether the incremental benefit is reliable. These requested controls would directly affect my assessment of the central mechanism.
1 comment
AuthorsOpenReview ↗

Response to Reviewer rZvi

We thank the reviewer for the careful and constructive review, and for recognizing that our framework combines neighboring-span context with structured competition while retaining exact marginal inference. Please refer to our General Response (GR0–GR3) for the feature extraction correction, model selection protocol, matched-feature comparisons, and statistical uncertainty analyses. We address the reviewer’s specific concerns below.

R1. Model selection and the headline comparison

There is no selection across seeds or test-set configurations. Within each run, we select the checkpoint with the highest validation [email protected] of the shared DETR decoder, using the same rule for all structured models and retrained baselines. Decoding and architectural choices are also selected exclusively on validation data. Table 1 now bases all claims on three-seed means. The "best" row is retained only as a reference to the seed with the highest test G-mAP and is not used for model selection.

With the corrected CLIP+SlowFast features, our three-seed mean is 37.26 ± 0.72 G-mAP, compared with 32.02 ± 0.24 for FlashMMR retrained under the same protocol. The paired video-clustered bootstrap difference is +5.24 G-mAP [95% CI: +3.37, +7.15]. Under matched InternVideo2+LLaMA features, our method also achieves a higher three-seed mean (38.43 vs. 35.96 G-mAP), further supporting the effectiveness of our approach across feature settings.

Revised manuscript: Section 4.2, Section 5.1, Table 1, Appendices A.2.6, A.2.9, A.3, and A.5.

R2. Matched comparison with FlashMMR

We retrained FlashMMR using the same input features, evaluator, validation-based checkpoint rule, and three seeds as our method. Under CLIP+SlowFast, our method achieves 37.26 vs. 32.02 G-mAP and 27.34 vs. 22.07 mAP@3+tgt. Under InternVideo2+LLaMA, it achieves 38.43 vs. 35.96 G-mAP and 26.62 vs. 23.73 mAP@3+tgt. Thus, the advantage is not attributable to stronger pretrained features alone.

Revised manuscript: Section 5.1, Table 1, Appendix A.5.

R3. Why QD-DETR and Base (DETR) differed

Base (DETR) is not a separately trained model. It is the DETR decoder branch of the jointly trained Base (SPD) model, with predictions ranked by decoder confidence. Most of the previously observed discrepancy was caused by the feature extraction issue in GR0. After correction, Base (DETR) reaches 28.48 ± 0.64 G-mAP, while QD-DETR retrained in our pipeline reaches 27.65 ± 1.55, close to the originally reported 28.95. Thus, joint training does not weaken the DETR branch, and Base (DETR) remains an appropriate reference for isolating the effect of structured decoding.

Revised manuscript: Table 2 and its caption, Section 5.2, Appendix A.5.

R4. Specification of the local-ψ controls

We now fully specify these baselines. Each local-ψ model shares the same segment potential network, encoder, 2D contextualizer, refinement module, auxiliary losses, checkpoint rule, candidate pool, and NMS settings as its SPD counterpart. The differences are limited to the scoring objective and normalization. The local model uses the scaled-IoU binary cross-entropy objective of 2D-TAN, ranks spans by σ(ψ(s,e)), removes the partition function and background term, and does not use the marginal-based IoU-risk loss. In Full (local-ψ), prototype weights are obtained from a softmax over ψ on the same candidate set.

Under this matched construction, Base (local-ψ) obtains 18.62 G-mAP compared with 33.87 for Base (SPD), while Full (local-ψ) obtains 28.65 compared with 37.26 for Full. This isolates the contribution of global normalization from that of the shared architecture.

Revised manuscript: Section 5.2, Appendix A.2.7, Table 13.

R5. Targeted evidence for posterior-conditioned refinement

We added a controlled experiment that keeps the 2DCon architecture, candidate set, and two refinement iterations fixed, and changes only the weights used to construct the refinement prototype. Uniform weighting reaches 36.36 G-mAP, and softmax over local ψ reaches 36.40, neither improving over 2DCon at 36.80. Segment-marginal weighting reaches 37.26 and gives the strongest top-1 localization results.

The refinement gains are most consistent in top-1 retrieval and localization. Compared with 2DCon, Full improves [email protected] by +2.89, mIoU@1 by +1.36, mIoU@2 by +1.86, mR@1 by +1.84, and mR@2 by +2.31, with improvements observed across all three seeds. We have revised the manuscript to clarify the distinct contributions of the two modules: 2DCon with global normalization drives the multi-target gains, while ITRef provides additional benefits in top-1 localization when applied on top of 2DCon.

Revised manuscript: Section 5.4, Table 4, Appendix A.7, Conclusion.

We hope these revisions address the reviewer's concerns and clarify both the evaluation protocol and the distinct contributions of structured normalization, span-lattice contextualization, and posterior-conditioned refinement.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all four reviewers for their careful and constructive feedback. We have revised the manuscript to address the concerns raised during the review, including a feature extraction correction, clarification of model selection, matched-feature baseline comparisons, and statistical uncertainty analyses. We summarize the main revisions below.

GR0. Correction and validation of CLIP text features

While investigating the discrepancy between QD-DETR and Base (DETR), as noted by Reviewer rZvi, we identified an inconsistency in CLIP text feature extraction. For the 2,212 queries added by QV-M² to QVHighlights, our script incorrectly applied CLIP's text projection to token-level hidden states, whereas the released QVHighlights features use unprojected states. This affected the entire QV-M² test set and portions of the training and validation sets.

We verified the correction using the official FlashMMR checkpoint, which achieved 4.40 G-mAP with the incorrect features but reproduced its reported 35.14 G-mAP with the corrected features. We subsequently re-extracted the features and retrained all affected CLIP+SlowFast models, including our ablations and the QD-DETR and FlashMMR baselines, using the original configurations and three seeds without hyperparameter retuning.

The corrected results preserve our main conclusions, with one refinement to our interpretation: ITRef alone no longer improves over Base (SPD), but provides additional benefits when applied on top of 2DCon, particularly in top-1 localization. We have revised the manuscript to reflect this distinction. The InternVideo2+LLaMA experiments and other datasets are unaffected. We apologize for this error and thank Reviewer rZvi for raising the question that led to its identification.

Revised manuscript: Tables 1–4, Sections 5.1–5.4, and relevant appendices.

GR1. Model selection and reporting (rZvi, smif, XkYc)

All checkpoints, architectural choices, and decoding settings were selected using validation data only. We have clarified the selection criteria and now base all performance claims on three-seed means. The best-seed results in Table 1 are retained for reference only and were not used for model selection.

Revised manuscript: Section 4.2, Table 1, Appendices A.2.4, A.2.6, and A.2.9.

GR2. Matched-feature comparison (rZvi, smif)

We retrained FlashMMR using identical features, evaluation protocols, checkpoint selection, and seeds under both CLIP+SlowFast and InternVideo2+LLaMA settings. Our method achieves higher three-seed mean G-mAP in both settings (37.26 vs. 32.02 and 38.43 vs. 35.96, respectively), demonstrating that the improvements persist under matched feature conditions.

Revised manuscript: Section 5.1, Table 1, Appendix A.5.

GR3. Statistical uncertainty and robustness (smif, rZvi)

To address the limited QV-M² test size and imbalanced target-count groups, we added video-clustered paired bootstrap confidence intervals and robustness analyses excluding sparsely populated groups or weighting queries equally. The results support improvements over Base (SPD) under both feature settings and over FlashMMR under CLIP+SlowFast, while clarifying the statistical limitations of other comparisons.

Revised manuscript: Appendix A.3, Tables 16–18.

We provide detailed, reviewer-specific responses in separate replies under each review.

Area_Chair_nVHPOpenReview ↗

Invitation to Discuss the Authors’ Response

Dear Reviewers,

The authors have now provided their response. I encourage all reviewers to read it and participate in the discussion with the authors to clarify any remaining questions or concerns.

Thank you to both the authors and reviewers for your time and effort throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.