- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multi-moment retrieval in video and proposes a structured posterior decoding framework. It first performs query-conditioned contextualization over a 2D span lattice, and then iteratively refines span potentials using exact segment posteriors to better distinguish and rank multiple candidate moments. The method builds on segmental posterior decoding previously used in audio moment retrieval, and is evaluated on QV-M2, QVHighlights, and Charades-STA.
Critical strengths
One interesting insight of the paper is that MMR should not rely solely on independently scoring candidate spans, but should jointly model their local relationships and global competition, with neighboring spans providing contextual information and the structured posterior further constraining their competition under complete temporal explanations.
Critical weaknesses
-
In the Introduction, the authors mainly divide existing methods into proposal-based approaches and dense span scoring approaches, but the discussion mostly describes how these formulations differ from the proposed method. For proposal-based methods, the paper argues that retrieval coverage is constrained by candidate generation, but it does not explain why better proposal generation, boundary regression, or subsequent verification cannot address this issue. For dense span methods, the discussion mainly focuses on 2D-TAN, while approaches such as SnAG can avoid enumerating all start-end pairs while still maintaining good coverage, yet this line of work is not discussed. Existing MMR-specific methods such as FlashMMR are mentioned, but only briefly, and I do not see a concrete failure mode that naturally motivates structured posterior decoding. Overall, the paper does not clearly establish why existing methods are insufficient.
-
The core structured posterior decoding framework is directly built on prior work in audio moment retrieval, as the authors explicitly state. The 2D span lattice and neighboring-span contextualization are also closely related to the temporal map formulation used in 2D-TAN and MS-2D-TAN. The main new component appears to be the posterior-conditioned refinement, together with the integration of these components for video MMR. Overall, the method feels more like an adaptation and combination of existing structured decoding and 2D span modeling than a new structured formulation, which limits the methodological novelty.
Questions for authors
-
Can the authors clarify what makes structured posterior decoding specifically necessary for MMR, and what limitation of existing proposal-based or dense span methods it addresses that cannot be handled by stronger candidate generation, regression, or post-processing?
-
Can the authors clarify the main methodological novelty of this work and distinguish it more clearly from the prior methods it builds on?
3 comments
Although the additional experiments demonstrate the effectiveness of the proposed method, my main concern regarding its motivation remains insufficiently addressed. The authors primarily explain the differences between candidate-level relevance scoring and jointly normalized structured posterior scoring, but do not sufficiently clarify why the latter is better suited to MMR.
Specifically, the authors emphasize competition among overlapping candidates, which is also what conventional NMS aims to address. While the authors explain that the two mechanisms operate at the scoring and post-processing stages, respectively, what substantive advantages does incorporating such competition into the scoring process offer? What limitations remain in existing methods after applying NMS? Given that the proposed method itself still employs NMS, the necessity of introducing structured posterior decoding remains unclear to me.
I acknowledge the performance improvements demonstrated by the additional experiments. However, I believe the authors still need to better articulate the limitations of existing methods and why the proposed framework can overcome them, rather than merely emphasizing differences in formulation and improvements in empirical performance.
We thank the reviewer for the thoughtful follow-up and for acknowledging the additional experiments. We agree that our previous response did not sufficiently clarify why structured posterior scoring is advantageous beyond NMS. We address this concern directly below.
The fundamental distinction lies in how relevance scores are obtained, not merely when overlapping predictions are suppressed. Conventional proposal-based methods predict individual moments with independently assigned confidence scores. NMS then suppresses candidates based on temporal IoU and these fixed scores, without reconsidering the visual evidence supporting alternative temporal explanations. This leads to two concrete failure modes in MMR.
-
Incorrect suppression of distinct moments. Consider two occurrences of a query separated by a short gap, together with an over-extended candidate spanning both. Because the merged candidate contains relevant evidence from both occurrences, it may receive a high confidence score. With a permissive NMS threshold, the merged candidate may survive alongside the correct predictions. With an aggressive threshold, a high-confidence merged candidate may instead suppress correctly localized predictions. NMS cannot determine whether the intervening gap should be treated as background, because its decisions depend on confidence rankings and temporal overlap, without considering how the entire video could be segmented into distinct relevant moments and background regions.
-
Incorrect ranking of surviving moments. Partially overlapping or over-extended variants of a salient moment may survive NMS when their IoU falls below the threshold. These candidates retain their original confidence scores and may rank above weaker but distinct relevant moments. This is particularly consequential in MMR, where multiple distinct targets must be recovered and redundant predictions can displace correct predictions from higher ranks.
Both problems arise because NMS operates on already independently assigned confidence scores.
In contrast, our method defines a probability distribution over all valid segmentations of the entire video timeline and computes each candidate's exact marginal probability across these globally competing explanations. A merged span and two separate occurrences represent alternative segmentations whose probabilities depend on the learned segment potentials and background evidence, including the intervening gap. Overlapping spans compete for probability mass because they cannot coexist within a segmentation, while disjoint moments can be jointly supported.
This is the substantive advantage for MMR: a candidate's relevance reflects its support across complete, structurally compatible explanations of the timeline, rather than an independently assigned confidence score followed by geometric suppression. Our contextualized span potentials provide the visual evidence needed to distinguish competing hypotheses, while global normalization incorporates their compatibility into the relevance scores.
Additional experimental evidence. Structured marginals do not eliminate all redundant predictions, so NMS remains useful as complementary post-processing. To directly examine the reviewer's concern, we additionally evaluated Full (local-ψ) without NMS, complementing the NMS sensitivity results already reported for Full (SPD) in Table 14 and the local-ψ results in Table 20. Both formulations share the architecture and span lattice, while differing in their training objectives and scoring rules.
QV-M² test results (three-seed mean ± std):
| Scoring | No NMS (G-mAP / @3+tgt) | NMS 0.7 (G-mAP / @3+tgt) |
|---|---|---|
| Full (local-ψ) | 24.07 ± 0.49 / 11.57 ± 0.95 | 28.65 ± 1.14 / 17.87 ± 0.14 |
| Full (SPD) | 33.86 ± 0.41 / 22.67 ± 1.38 | 37.26 ± 0.72 / 27.34 ± 0.75 |
Even without NMS, Full (SPD) outperforms Full (local-ψ), which independently scores each span and applies NMS, by +5.21 G-mAP and +4.80 mAP@3+tgt. Without NMS, the structured formulation also exceeds the local formulation by +9.79 G-mAP and +11.10 mAP@3+tgt. NMS improves both formulations, confirming its complementary role. However, the performance advantage of structured posterior scoring is already substantial before NMS is applied.
These results demonstrate that NMS alone does not recover the performance advantage of structured posterior scoring. Although the comparison also involves differences in training objectives, it directly shows that the improvements cannot be attributed merely to post-processing. Rather than relying solely on suppressing predictions after confidence-based ranking, our framework incorporates competition among complete temporal explanations into relevance scoring itself.
We hope this clarifies why structured posterior decoding is particularly well suited to MMR. We would be happy to provide further clarification or address any remaining concerns the reviewer may have.
Response to Reviewer N9jE
We thank the reviewer for the thoughtful review and for highlighting the central insight of our work. Please also refer to the General Response, particularly GR0 for the corrected CLIP features and GR2–GR3 for matched-feature comparisons and statistical uncertainty analyses. Below, we address the reviewer’s two methodological questions using the corrected results.
R1. Why is structured posterior decoding needed beyond better proposals, regression, or post-processing?
Improving proposal quality and post-processing is important for multi-moment retrieval, but the challenge lies not only in generating accurate candidates, but also in assigning relevance scores when multiple temporal hypotheses compete. Existing methods, including FlashMMR, address candidate quality through boundary refinement and post-verification. However, their final relevance scores are assigned to individual candidates rather than jointly normalized over complete temporal explanations.
This distinction is reflected in the empirical results. Despite FlashMMR's dedicated multi-moment refinement and verification, its reported mAP@3+tgt is 22.89, compared with 20.19 for FlashVTG. Under matched CLIP+SlowFast features, our method achieves 27.34 mAP@3+tgt, compared with 22.07 for retrained FlashMMR, with a statistically supported improvement of +5.27 [95% CI: +2.73, +6.92]. This suggests that structured scoring provides benefits beyond improving individual candidates.
Our formulation introduces global competition through a distribution over complete temporal segmentations. Overlapping hypotheses share probability mass, while disjoint moments can coexist. NMS remains useful for removing redundant predictions, and we apply it to our method as well. The distinction is that structured competition is incorporated into the scoring process before NMS, rather than relying on candidate-level scores and subsequent post-processing alone.
To isolate this effect, we additionally compare independent local scoring and structured posterior decoding over the same exhaustive span lattice with identical NMS settings (Table 2). Base (SPD) improves over Base (local-ψ) from 18.62 to 33.87 G-mAP and from 4.62 to 20.25 mAP@3+tgt. Similarly, Full improves over Full (local-ψ) from 28.65 to 37.26 G-mAP and from 17.87 to 27.34 mAP@3+tgt. These controlled comparisons demonstrate the importance of global normalization beyond candidate coverage and local contextualization.
We have also added SnAG to Related Work. Its multi-scale anchor-based regression addresses candidate generation and scalability, which are complementary to our focus on structured relevance assignment.
Revised manuscript: Section 2, Section 5.2, Tables 1–2, Appendices A.2.7 and A.3.
R2. What is methodologically novel relative to the methods we build on?
We acknowledge that segmental posterior decoding and 2D temporal span maps are established components. Our contribution lies in integrating local span contextualization and posterior-conditioned refinement within a globally normalized structured model, supported by controlled experiments that distinguish their roles.
First, we identify a limitation of directly transferring boundary-based segmental decoding to video. Its advantage over DETR confidence decreases from 8.3 AP on single-target queries to 3.4 AP on queries with three or more targets (Section 5.2), motivating richer segment potentials.
Second, unlike 2D-TAN and MS-2D-TAN, which use contextualized span maps to produce independently scored candidates, our 2D contextualizer modifies segment potentials before global normalization. This allows neighboring-span information to influence structured inference while preserving exact segment marginals. The Full (local-ψ) comparison in Table 2 empirically distinguishes these formulations.
Third, we introduce posterior-conditioned refinement, which feeds exact segment marginals back into the potential construction through a posterior-weighted prototype. To evaluate whether the structured posterior contributes beyond iterative refinement itself, we added controlled experiments comparing posterior-marginal weighting with uniform weighting and local softmax weighting, while holding the 2DCon architecture, candidate set, and refinement iterations fixed (Table 4). Posterior-marginal weighting achieves the highest mean G-mAP and improves top-1 localization metrics. We have also clarified that 2DCon accounts for most of the multi-target gains, while ITRef provides a smaller additional benefit when applied on top of 2DCon.
Revised manuscript: Sections 3.2–3.3, Sections 5.2–5.4, Tables 2–4, and Appendix A.7.
We hope these clarifications and additional experiments address the reviewer's concerns about the necessity of structured decoding and the methodological contribution of our framework.