acceptodds
Under review as a conference paper at ICLR 2027

Grounding Confusable Movie Moments

Abstract

Characters, locations, and actions recur throughout a movie, so many of its moments differ only in fine-grained details. Grounding such a moment in a full-length movie requires resolving that detail rather than matching the attributes it shares with other moments, yet existing temporal movie grounding benchmarks rarely test this. We introduce CoM-Bench, a benchmark of confusable query pairs, where the two queries in each pair share most attributes but differ in a single distinguishing detail within a full movie, posing a challenge for recent temporal grounding approaches. To locate where grounding breaks down, we analyze a retrieve-then-localize pipeline, in which a retriever first selects a short candidate chunk and a localizer then performs temporal grounding within it. Separating the two stages lets us attribute each failure to either retrieval or localization. We find that retrieval is the major bottleneck: state-of-the-art retrievers often select a moment's confusable counterpart, because their choices follow the content the two moments share rather than the detail that tells them apart. To address these failures, we propose a simple, training-free remedy, Agentic Window Search (AWS): an MLLM agent the query into attribute-level sub-queries, then each candidate window against them and the window when evidence is cut off at its edge. AWS works with existing retrievers and localizers without additional training, raising mIoU from 18.5 to 26.8.

Primary area:
datasets and benchmarks
Keywords:
Video temporal grounding, Long-form video understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QdSUOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  1. This paper introduces CoM2-Bench, a benchmark for grounding confusable moments in full-length movies. Each query pair describes two different moments that share most attributes but differ in one distinguishing detail. The benchmark contains 3,014 queries from 46 movies.

2.The authors analyze a retrieve-then-localize pipeline and find that retrieving the correct video chunk is a major bottleneck. They then propose Agentic Window Search (AWS), which decomposes queries into individual conditions, checks retrieved video windows against these conditions, and expands windows when necessary. Without additional training, AWS improves mIoU from 18.5 to 26.8 on the proposed benchmark.

Critical strengths
  1. A meaningful benchmark with a clear evaluation objective.

I find the benchmark to be the strongest part of the paper. Existing temporal grounding evaluations do not necessarily reveal whether a model distinguishes two similar events within the same video. Constructing paired queries that differ in a specific attribute provides a useful way to examine this limitation. The PairSuccess metric also makes the paired evaluation explicit.

  1. The retrieval analysis is convincing and leads to a useful solution.

The oracle experiments clearly show that incorrect chunk retrieval is a major source of error. The comparison between random and confusable alternatives provides further evidence that retrievers often rely on shared attributes while overlooking distinguishing details.

AWS is relatively simple, but its improvement over Movie RAG is substantial. The additional results with different retrievers and on MomentSeeker also help support the general usefulness of evidence verification.

Critical weaknesses
  1. The benchmark annotation quality and coverage need stronger validation.

Although the dataset includes human verification, only 18 of the 46 movies receive a second independent review. The reported Cohen's kappa of 0.771 measures agreement on pair acceptance rather than agreement on temporal boundaries. Since the ground-truth intervals are initially generated by an MLLM and subsequently accepted or rejected by annotators, it is difficult to assess their boundary accuracy independently.

The distribution of distinguishing attributes is also uneven. According to Table B, 1,310 of the 3,014 queries use setting as the contrasting attribute, whereas only 12 involve temporal composition. This raises questions about how well the benchmark measures different types of fine-grained discrimination.

  1. The proposed verification procedure and its contribution require further analysis.

The Inspect prompt considers a statement satisfied if it holds somewhere within the candidate window, and explicitly allows different statements to hold at different times. This may be reasonable for an event spanning several actions, but it can also accept a window in which the conditions are satisfied by unrelated events. It is unclear how the method ensures that the verified attributes correspond to the same coherent event.

I also find the component analysis incomplete. In Table 4, a single-pass 27B verifier already achieves 25.1 mIoU, compared with 26.8 for the full AWS method. The paper does not separately quantify the effects of query decomposition, cascaded inspection, and window expansion. Moreover, AWS can make multiple 27B model calls per query, but the average inference cost is not reported. This makes it difficult to assess the benefit of the complete procedure relative to its computational overhead.

Questions for authors
  1. Could the authors provide additional evidence of temporal boundary accuracy and query uniqueness, particularly for the movies without a second independent review? It would also be useful to see AWS results broken down by distinguishing attribute.

  2. How does the verifier prevent different sub-queries from being satisfied by unrelated events within the same window? Have the authors considered checking whether these conditions form a temporally coherent event rather than merely appearing somewhere in the window?

  3. Could the authors report separate ablations for query decomposition and window expansion, together with average model calls and inference time? A comparison with a simpler verifier under a similar computational budget would help clarify the practical value of the full AWS procedure.

7 comments
AuthorsOpenReview ↗

Response to Reviewer QdSU (1/2)

Thank you for your thoughtful feedback and for recognizing the clear evaluation objective of our benchmark, the PairSuccess metric, the retrieval analysis comparing random and confusable alternatives, and the substantial gain of AWS over Movie RAG. We also appreciate your positive note on our experiments across different retrievers and on MomentSeeker. We address each of your concerns and questions below.

[W1-1] "Although the dataset includes human verification, only 18 of the 46 movies receive a second independent review. The reported Cohen's kappa of 0.771 measures agreement on pair acceptance rather than agreement on temporal boundaries. … it is difficult to assess their boundary accuracy independently." [Q1-1] "Could the authors provide additional evidence of temporal boundary accuracy and query uniqueness, particularly for the movies without a second independent review?"

We checked the 28 movies without a second review on both points. Annotators who had not seen our labels marked boundaries that match ours with a median IoU of 0.77, and of 100 sampled windows that AWS accepted outside both paired labeled moments, only 6 fit the full query. These checks measure boundaries and uniqueness directly, complementing the pair-acceptance κ.

For boundary tightness, the annotators marked 100 sampled queries inside clips that hide where the moment is (the moment plus 120 s of extra video), and 89% match our labels above 0.5 IoU. For moment uniqueness, we checked those windows because another fitting moment is most likely to be found there. For performance consistency, Movie RAG (18.5 vs. 18.5 mIoU) and AWS (27.1 vs. 26.2) perform almost the same on these 28 movies as on the 18 cross-reviewed ones. The low baseline scores therefore come from retrieval failures rather than label ambiguity, as the same localizer reaches 53.8 mIoU given the oracle chunk compared to 18.5 for Movie RAG (Table 2).

[W1-2] "The distribution of distinguishing attributes is also uneven. According to Table B, 1,310 of the 3,014 queries use setting as the contrasting attribute, whereas only 12 involve temporal composition. This raises questions about how well the benchmark measures different types of fine-grained discrimination." [Q1-2] "It would also be useful to see AWS results broken down by distinguishing attribute."

Each of the six main attribute types has at least 196 queries, and AWS improves mIoU significantly on every type. Temporal composition was already rare before human verification, with only 12 of the 4,080 candidate queries, and verification kept all 12. Averaging the six types with equal weight gives a gain of +7.5, close to the overall gain of +8.3, so the gain does not depend on any single attribute. Temporal composition needs two moments in the same movie with the same subject, action, and setting that differ only in the order of events, which rarely happens. We therefore report its 12 queries in a separate row and leave them out of the average.

Distinguishing attributeNmIoU Movie RAG → AWS
setting131021.3 → 30.3
core action56615.9 → 26.0
subject38216.1 → 22.5
visual anchor33618.2 → 25.4
motion21214.3 → 20.2
interaction19616.6 → 23.1
temporal composition1225.3 → 27.6 (small N)
all301418.5 → 26.8 (average of six main types 17.1 → 24.6)

We agree that the distribution is uneven, so we add these per-type results to the Appendix.

[W2-1] "The Inspect prompt considers a statement satisfied if it holds somewhere within the candidate window, and explicitly allows different statements to hold at different times. … it can also accept a window in which the conditions are satisfied by unrelated events. It is unclear how the method ensures that the verified attributes correspond to the same coherent event." [Q2] "How does the verifier prevent different sub-queries from being satisfied by unrelated events within the same window? Have the authors considered checking whether these conditions form a temporally coherent event rather than merely appearing somewhere in the window?"

Accepted windows rarely combine statements from separate times: this happens in only 0.3% of them, measured with the cooccurrence note that the Inspect prompt (Appendix G) already asks Inspect to write for every window. In 97.9%, the span where all statements hold overlaps the ground-truth moment, and the remaining 1.8% point to a similar moment elsewhere in the window. We counted over the 1,289 accepted windows that contain the ground-truth moment and have such a note. In addition, the localizer grounds the full query as one contiguous span inside the accepted window, so the final answer is one contiguous span rather than separate pieces. We do not require all statements in a single frame because one event often spans several shots, such as an action followed by a reaction.

AuthorsOpenReview ↗

Response to Reviewer QdSU (2/2)

[W2-2] "I also find the component analysis incomplete. In Table 4, a single-pass 27B verifier already achieves 25.1 mIoU, compared with 26.8 for the full AWS method. The paper does not separately quantify the effects of query decomposition, cascaded inspection, and window expansion. Moreover, AWS can make multiple 27B model calls per query, but the average inference cost is not reported. …" [Q3] "Could the authors report separate ablations for query decomposition and window expansion, together with average model calls and inference time? A comparison with a simpler verifier under a similar computational budget would help clarify the practical value of the full AWS procedure."

We ablated each part, and query decomposition is the main source of the gain (+1.2 mIoU), while AWS averages 3.1 MLLM calls per query (median 2). Against a simpler baseline with the same Qwen3.6-27B model, AWS is 2.9 mIoU higher at a matched token budget, and this baseline only matches AWS at 1.4× the tokens. The effect of each part:

  • Query decomposition: +1.2 mIoU both with and without expansion, the main source of the gain.
  • Cascaded inspection: with the full query alone it reaches 25.5 mIoU, close to the single-pass verifier (AWS w/o inspection, which gives the 27B model all top-5 candidate windows in one call) while using 98k instead of 172k input tokens.
  • Window expansion: triggers on 2.5% of queries, raising their mIoU from 17.2 to 22.5.
  • Model calls and inference time: 3.1 calls per query on average (median 2), taking 72 s of MLLM time per query, close to the single-pass verifier (63 s).
ConfigDecomposeCascaded inspectionExpandmIoUMLLM calls / queryInput tokens / queryMLLM time / query (s, median)
Single pass (Table 4)–––25.11172k63
w/o decomposition & expansion✗✓✗25.52.098k36
w/o expansion✓✓✗26.73.1111k70
w/o decomposition✗✓✓25.62.0100k53
Full AWS✓✓✓26.83.1112k72

Simpler baseline under a matched budget. We introduce a new baseline that uses the same 27B model as a reranker (the scoring baseline), with the same frames, candidates, and character memory as AWS. It scores each candidate independently via P(Yes | full query, window) and selects the highest-scoring window, without query decomposition or cascaded inspection. Scoring the top 2 candidates (K = 2) uses a token budget similar to AWS, and scoring the top 3 (K = 3) uses 1.4× the tokens of AWS.

MethodInput tokens / querymIoUΔ vs AWS [95% CI]
scoring baseline, K = 2 (matched budget)101k23.9−2.9 [−3.7, −2.0]
scoring baseline, K = 3 (1.4× budget)152k27.3+0.5 [−0.3, +1.3]
AWS112k26.8–

95% CI from a paired bootstrap over the 1,507 pairs (10,000 resamples). An interval that includes 0 means no significant difference.

In the revision, we will add the independent boundary and uniqueness checks of the 28 movies next to Table A, add the per-attribute breakdown alongside an explanation of the attribute distribution, describe how Inspect and the localizer divide this check, with the cooccurrence statistics above, and include both the full component ablation and a new Inference Cost section.


We hope the independent boundary and uniqueness checks, the co-occurrence check of accepted windows, and the full component ablation under a matched budget address the concerns raised in your review. If any point remains unresolved, we would be happy to discuss it further during the discussion period.

AuthorsOpenReview ↗

Follow-up to Reviewer QdSU

Dear Reviewer QdSU,

Thank you again for your thoughtful and detailed review. We have posted a detailed response and uploaded a revised manuscript with new experiments, clarifications, and corrections.

In particular, we added an independent boundary check on the 28 movies that did not receive a second review (median IoU 0.77 against annotators who did not see our labels), and a co-occurrence analysis showing that accepted windows combine statements from unrelated events in only 0.3% of cases. These directly address your concerns on boundary accuracy and on whether the verified attributes form a temporally coherent event.

We would appreciate it if you could take another look at our response and let us know whether your concerns have been resolved. If so, we would respectfully appreciate your support for our work and your kind consideration of increasing your score accordingly.

Thank you for your time and effort.

Reviewer_nuvdOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The submission studies temporal grounding of confusable moments in full-length movies. These are moments that share most query attributes and differ in a single detail. It introduces CoM2-Bench, comprising 1,507 verified query pairs (3,014 queries) over 46 movies (90.4 h), each pair sharing three attributes and differing in one, evaluated with a pair-level metric (PairSuccess@θ) that requires both queries to ground correctly. It then diagnoses a retrieve-then-localize pipeline (Movie RAG, a Qwen3-VL-Embedding retriever plus TimeLens2 localizer), reporting that chunk retrieval is the dominant bottleneck and that the retriever follows shared content rather than the distinguishing detail (§3, Table 2, Fig. 4). Finally it proposes Agentic Window Search (AWS), a training-free MLLM agent that decomposes the query into sub-queries, inspects and expands candidate windows, and uses a character memory. With the retriever and localizer fixed, AWS raises mIoU from 18.5 to 26.8 (27B verifier) (§4-5).

Critical strengths
  1. The benchmark is novel and well motivated, and the diagnosis is controlled. CoM2-Bench operationalizes an under-measured failure mode (paired look-alike moments) with a pair-level metric that cannot be gamed by predicting one interval for both queries. Table 2's oracle substitution isolates retrieval cleanly. An oracle chunk lifts mIoU from 18.5 to 53.8, while an oracle localizer on the retrieved chunk reaches only 31.0. The result turns "models are bad at long video" into a specific and actionable attribution to retrieval, and the delta versus CoMET-Bench, MAD and MomentSeeker (Table 1) holds up against public abstracts.

  2. The evidence reporting is honest at the point of failure. §5.1 openly states that AWS "closes 24% of the gap" (26.8 vs 53.8) and that [email protected] is only 8.8%. The paper does not hide its bounds, which raises my confidence in the numbers it does report.

Critical weaknesses
  1. I think the headline gain conflates "adding a 27B verifier" with the novel agentic machinery, and the incremental deltas are not quantified against noise. A single 27B pass over the top-5 windows already reaches 25.1 mIoU (from 18.5). Cascaded inspection adds +1.7 and character memory adds +0.7. With stochastic verifiers (Inspect temperature 0.4, no seeds) and no confidence intervals, these two increments are difficult to distinguish from run-to-run variation, so the specific contribution of DECOMPOSE, INSPECT and EXPAND is not yet established. The abstract's "raising mIoU from 18.5 to 26.8" therefore seems to attribute to AWS what is largely the effect of inserting a strong MLLM reader. It would help to report at least 3 seeds or bootstrapped CIs over the 1,507 pairs for each ablation row.

2.The benchmark labels that carry most of the argument are verified by the submission's own authors, with blind cross-review on only 18 of 46 movies. The uniqueness criterion (Fig. C, item 4) is exactly the property that makes a pair confusable. If the criterion is lenient, the near-zero baseline scores may partly reflect label ambiguity rather than model failure. Separately, the "shared query" counterfactual in §3.3 (Figure 4b, the 47.6% figure) is LLM-generated and has not been human-validated. It would help to have independent annotation of the 28 non-cross-reviewed movies, and a human spot-check that each generated shared query removes the distinguishing detail without perturbing the other three attributes.

  1. AWS's attainable mIoU is bounded by top-5 retrieval recall, and the paper does not report that number. AWS inspects only K=5 candidates and stops at the first ACCEPT, so if recall@5 is low, the reported 26.8 is close to the method's ceiling and the agentic gains cannot be separated from what retrieval already makes available. It would help to report recall@5 of an oracle-overlapping chunk and to show how much headroom remains above 26.8.

  2. The method runs a 27B verifier and up to K(M+1)=20 Inspect calls per query over 3,014 queries, with windows of up to 32 frames per expansion, yet the paper gives no GPU-hours, API cost or wall-clock time. For a training-free method whose appeal is practicality, that budget matters. It would help to report the realized cost and latency per query and per benchmark, and to compare them with the retrieve-then-localize baseline.

Questions for authors
  1. What is the retrieval recall@K (K=5) of an oracle-overlapping chunk on CoM2-Bench? AWS inspects only K=5 candidates and stops at the first ACCEPT, so this number bounds the method's attainable mIoU and separates "search never finds it" from "verifier rejects the right one." If recall@5 is low, 26.8 is near the method's ceiling.
  2. Do the Table 4 component deltas (+1.7 for cascaded inspection, +0.7 for character memory) exceed run-to-run variance? Please report seeds or bootstrapped CIs over the 1,507 pairs.
  3. For the 28 movies without blind cross-review (Table A), what independent check confirms uniqueness (Fig. C) and boundary tightness? 4.] How was the LLM-written "shared query" in §3.3 validated to remove exactly the distinguishing detail without changing the other three attributes?
  4. A compute-matched control would help. Use the same 27B MLLM, the same sampled frames and the same top-5 candidates, and ask it to ground directly without decomposition, to isolate the agentic contribution from the stronger reader.
5 comments
AuthorsOpenReview ↗

Response to Reviewer nuvd (1/2)

Thank you for your thoughtful feedback and for recognizing the benchmark as novel and well-motivated, alongside our controlled diagnosis and transparent reporting of the method's upper bounds. We address each of your concerns and questions below. As suggested, all confidence intervals below are paired bootstrap 95% CIs (B = 10,000) computed over the 1,507 pairs. Resampling at the movie level (46 movies) yields the same conclusions for the gains from the 27B verifier, the agentic procedure, and decomposition.

[W1] "I think the headline gain conflates "adding a 27B verifier" with the novel agentic machinery, and the incremental deltas are not quantified against noise. … It would help to report at least 3 seeds or bootstrapped CIs over the 1,507 pairs for each ablation row." [Q2] "Do the Table 4 component deltas (+1.7 for cascaded inspection, +0.7 for character memory) exceed run-to-run variance? Please report seeds or bootstrapped CIs over the 1,507 pairs."

We bootstrapped 95% CIs over the 1,507 pairs, and both increments you asked about stay above zero: +1.7 mIoU for the agentic procedure over the single-pass verifier [+0.7, +2.7] and +0.7 for character memory [+0.2, +1.0]. Of the +8.3 headline gain, +6.6 comes from adding Qwen3.6-27B as the single-pass verifier (AWS w/o inspection), which reads all top-5 candidate windows in one call with the same candidates and character memory as AWS. The agentic procedure adds the remaining +1.7, most of which comes from query decomposition (+1.2).

ComparisonmIoUΔ95% CI
Adding the 27B verifier (Movie RAG → single pass)18.5 → 25.1+6.6[+5.5, +7.6]
Agentic procedure (single pass → AWS)25.1 → 26.8+1.7[+0.7, +2.7]
Character memory (w/o memory → AWS)26.1 → 26.8+0.7[+0.2, +1.0]
Query decomposition (w/o decomposition → AWS)25.6 → 26.8+1.2[+0.5, +1.9]
Window expansion (w/o expansion → AWS)26.7 → 26.8+0.1[0.0, +0.3]
Headline (Movie RAG → AWS)18.5 → 26.8+8.3[+7.4, +9.2]

Window expansion adds only +0.1 overall because it triggers on only 2.5% of queries, but on those queries it raises mIoU from 17.2 to 22.5, which suggests that learning when to expand can extend this gain to more queries.

[W2-1] "The benchmark labels that carry most of the argument are verified by the submission's own authors, with blind cross-review on only 18 of 46 movies. … If the criterion is lenient, the near-zero baseline scores may partly reflect label ambiguity rather than model failure." [Q3] "For the 28 movies without blind cross-review (Table A), what independent check confirms uniqueness (Fig. C) and boundary tightness?"

We checked the 28 movies independently on both points. Annotators who had not seen our labels marked boundaries that match ours with a median IoU of 0.77, and of 100 sampled windows that AWS accepted outside both paired labeled moments, only 6 fit the full query.

  • Boundary tightness. We sampled 100 queries from the 28 movies, evenly across distinguishing attributes. For each query, the annotators watched a clip that contains the moment plus 120 s of extra video, split randomly before and after so that the clip does not reveal where the moment is. They marked where the full query holds, without seeing our boundaries. Compared with our labels, the median IoU is 0.77, 89% of queries exceed 0.5, and the median start and end differences are 3.5 s and 4.8 s.
  • Uniqueness. We checked the windows that AWS accepted outside both paired labeled moments, since another fitting moment is most likely to be found there. In the 28 movies, AWS accepts such a window for 576 queries (31%). We sampled 100 of these windows, and the annotators watched each one and judged whether it fits the full query. Only 6 do, and most of the rest differ in the distinguishing attribute, such as a similar action in a different setting. To check the judgments, we mixed in 20 accepted windows that do contain the labeled moment without telling the annotators, and 18 of them were judged to fit the query.

As further support, Movie RAG (18.5 vs. 18.5 mIoU) and AWS (27.1 vs. 26.2) perform almost the same on these 28 movies as on the 18 cross-reviewed ones (κ = 0.771), so the labels of the 28 movies show no sign of being looser. The low baseline scores therefore come from retrieval failures rather than label ambiguity, as the same localizer reaches 53.8 mIoU given the oracle chunk compared to 18.5 for Movie RAG (Table 2).

AuthorsOpenReview ↗

Response to Reviewer nuvd (2/2)

[W2-2] "Separately, the "shared query" counterfactual in §3.3 (Figure 4b, the 47.6% figure) is LLM-generated and has not been human-validated. … a human spot-check that each generated shared query removes the distinguishing detail without perturbing the other three attributes." [Q4] "How was the LLM-written "shared query" in §3.3 validated to remove exactly the distinguishing detail without changing the other three attributes?"

We validated the shared queries of 100 randomly sampled pairs by hand, and 89 of them drop the distinguishing attribute and keep the rest unchanged. For each pair, we compared the shared query with its two original queries. In §3.3, when a shared query favors the wrong moment, adding the distinguishing attribute back moves only 47.6% of queries to the correct one. On the 89 valid pairs, this rate is 49.4%, almost the same, so the conclusion does not depend on generation errors.

[W3] "AWS's attainable mIoU is bounded by top-5 retrieval recall, and the paper does not report that number. … It would help to report recall@5 of an oracle-overlapping chunk and to show how much headroom remains above 26.8." [Q1] "What is the retrieval recall@K (K=5) of an oracle-overlapping chunk on CoM2-Bench? … this number bounds the method's attainable mIoU and separates "search never finds it" from "verifier rejects the right one." If recall@5 is low, 26.8 is near the method's ceiling."

For 70.9% of queries, a chunk that overlaps the ground truth is among the top-5 candidates (R@5). If AWS always picked the best of these five windows, the same localizer would reach at least 32.7 mIoU, 5.9 points above AWS, so 26.8 is well below this ceiling.

K13510
Chunk overlapping the ground truth, in top-K37.260.470.984.8
  • "Search never finds it" (no overlapping chunk in the top-5): 29.1% of queries.
  • "Verifier rejects the right one" (an overlapping chunk is in the top-5, but AWS selects another window): 18.3% of queries.
  • Headroom. Measured by how much of the ground truth the selected window covers, top-1 gives 31.0, the best of the top-5 gives 61.9, and AWS reaches 44.8, closing 45% of this gap.

[W4] "The method runs a 27B verifier and up to K(M+1)=20 Inspect calls per query over 3,014 queries … yet the paper gives no GPU-hours, API cost or wall-clock time. … It would help to report the realized cost and latency per query and per benchmark, and to compare them with the retrieve-then-localize baseline."

We measured the actual cost against retrieve-then-localize: AWS averages 3.1 MLLM calls per query (median 2, well below the worst-case bound) and adds 72 s of MLLM time per query, close to the 63 s of the single-pass verifier with the same 27B model. There is no API cost because all models run locally with open weights.

Retrieve-then-localizeSingle passAWS
MLLM calls / query (mean / median)013.1 / 2
MLLM time / query (s, median)06372

Per-benchmark cost scales linearly with the 3,014 queries.

[Q5] "A compute-matched control would help. Use the same 27B MLLM, the same sampled frames and the same top-5 candidates, and ask it to ground directly without decomposition, to isolate the agentic contribution from the stronger reader."

This control is the single-pass verifier in Table 4 (25.1 mIoU), and AWS outperforms it by +1.7 mIoU [+0.7, +2.7] while reading fewer frames (373 vs. 900 per query) and using 1.5× fewer input tokens (112k vs. 172k). The single-pass verifier uses the same 27B MLLM, sampled frames and top-5 candidates, and grounds the query in a single call without decomposition.

MethodFrames / queryInput tokens / queryMLLM calls / querymIoU
Single pass900172k125.1
AWS373112k3.126.8

Furthermore, the variant that keeps cascaded inspection but removes query decomposition reaches 25.6 mIoU, 1.2 below AWS, which isolates the contribution of decomposition.

In the revision, we will report the gains from the 27B verifier and from the agentic procedure separately wherever we state the headline gain, add bootstrap CIs for each ablation row and input token counts to Table 4, add the independent boundary and uniqueness checks of the 28 movies next to Table A, and include both the recall@K analysis and a new Inference Cost section.


We hope the bootstrap CIs on every component, the recall@K ceiling analysis, the independent re-annotation of the 28 movies, and the full inference-cost accounting address the concerns raised in your review. If any point remains unresolved, we would be happy to discuss it further during the discussion period.

AuthorsOpenReview ↗

Follow-up to Reviewer nuvd

Dear Reviewer nuvd,

Thank you again for your thoughtful and detailed review. We have posted a detailed response and uploaded a revised manuscript with new experiments, clarifications, and corrections.

As you requested, we now report paired bootstrap 95% CIs over the 1,507 pairs for every ablation row, and separate the headline gain into the 27B verifier (+6.6) and the agentic procedure (+1.7, CI [+0.7, +2.7]). We also added an independent re-annotation of the 28 movies that did not receive a second review (boundary IoU 0.77), the recall@5 ceiling analysis (70.9%), and a full inference-cost accounting.

We would appreciate it if you could take another look at our response and let us know whether your concerns have been resolved. We are also happy to run any further analysis during the discussion period.

Thank you for your time and effort.

Reviewer_FYFAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal grounding of confusable moments in full-length movies. The authors introduce CoM2-Bench, which contains 3,014 queries from 46 movies. Each query pair shares most attributes but differs in one detail.

The authors identify retrieval as the main bottleneck in existing grounding pipelines. They then propose Agentic Window Search (AWS), which decomposes queries, verifies candidate windows, and expands them when needed. AWS improves mIoU from 18.5 to 26.8 without additional training.

Critical strengths
  1. I think the benchmark is the strongest part of this paper. The confusable query pairs provide a meaningful way to evaluate fine-grained temporal grounding. The dataset construction is also reasonably well designed, with human verification and a pair-level evaluation metric.

  2. I appreciate the analysis of retrieval and localization errors. The oracle experiments in Table 2 clearly show that retrieval is a major bottleneck. The experiments also cover several baselines and different pipeline configurations, which makes the findings more convincing.

Critical weaknesses
  1. Limited novelty and unclear computational efficiency. AWS mainly combines existing techniques, including query decomposition, MLLM verification, and window expansion. The single-pass verifier already achieves 25.1 mIoU, compared to 26.8 for full AWS. The additional methodological contribution seems limited. Also, the lack of compute-matched comparisons makes it hard to judge whether the gains justify the extra cost.

  2. Concerns about benchmark evaluation validity. It is unclear whether CoM2-Bench truly requires understanding the full query, as the distinguishing attribute alone might be sufficient for grounding. Also, the improvement in PairSuccess may simply reflect better overall grounding accuracy. The imbalanced attribute distribution further raises concerns about the benchmark's coverage of different types of confusable moments.

Questions for authors

Q1: Could the authors provide separate ablations for query decomposition, cascaded inspection, and window expansion? Q2: What are the average inference time and number of MLLM calls per query? Q3: How does AWS compare with simpler verification or reranking methods under the same computational budget? Q4: How well would models perform using only the distinguishing attribute instead of the full query? Q5: Could the authors report pairwise confusion rates for AWS, ideally broken down by contrast attribute, to show whether it actually improves discrimination between similar moments

3 comments
AuthorsOpenReview ↗

Response to Reviewer FYFA (1/2)

Thank you for your thoughtful feedback and for finding the benchmark to be the strongest part of the paper, particularly its confusable query pairs, human verification, and pair-level metric. We also appreciate your recognition of the oracle experiments in Table 2 and our range of baselines and pipeline configurations. We address each of your concerns and questions below.

[W1-1] "Limited novelty and unclear computational efficiency. AWS mainly combines existing techniques, including query decomposition, MLLM verification, and window expansion. The single-pass verifier already achieves 25.1 mIoU, compared to 26.8 for full AWS. The additional methodological contribution seems limited."

Our main contributions are the benchmark and the diagnosis of where grounding fails, and AWS is a simple, training-free method that turns this diagnosis into a fix: its gain comes from checking the distinguishing attribute on its own, not from more MLLM calls. With the same Qwen3.6-27B as verifier, it raises mIoU from 25.1 to 26.8 (+1.7, significant over the 1,507 pairs) while reading fewer frames (373 vs. 900 per query) and using 1.5× fewer input tokens (112k vs. 172k).

Our diagnosis (§3) shows that the retriever ranks candidates by the content that confusable moments share, so the distinguishing attribute has to be checked on its own. AWS does exactly this, which links the benchmark, the diagnosis, and the method into one line of work. Inspecting the candidates one by one with the full query gives 25.5 mIoU, close to the single-pass verifier (AWS w/o inspection, which gives the 27B model all top-5 candidate windows in one call) with 43% fewer input tokens, and adding decomposition raises mIoU by 1.2 both with and without expansion (see [Q1]). The gain therefore comes from checking the distinguishing attribute on its own, not from more MLLM calls. AWS also adds to existing retrieval fixes rather than replacing them, reaching 28.2 mIoU on top of a reranker and 30.6 on top of a fine-tuned retriever (Table 4). We compare AWS with a new baseline under a matched token budget right below ([W1-2]).

[W1-2] "Also, the lack of compute-matched comparisons makes it hard to judge whether the gains justify the extra cost." [Q3] "How does AWS compare with simpler verification or reranking methods under the same computational budget?"

At a matched token budget, AWS outperforms a simpler scoring baseline that uses the same 27B model by 2.9 mIoU, significant over the 1,507 pairs. Even with 1.4× the input tokens, the baseline only matches AWS, as the 95% CI of the difference includes zero.

Following your request, we introduce a new baseline that uses the same 27B model as a reranker, which we call the scoring baseline. With the same frames and character memory as AWS, it scores each candidate via P(Yes | full query, window) and returns the highest-scoring window. Scoring the top 2 candidates (K = 2) uses a token budget similar to AWS, and scoring the top 3 (K = 3) uses 1.4× the tokens of AWS.

MethodInput tokens / querymIoUΔ vs AWS [95% CI]
scoring baseline, K = 2 (matched budget)101k23.9−2.9 [−3.7, −2.0]
scoring baseline, K = 3 (1.4× budget)152k27.3+0.5 [−0.3, +1.3]
AWS112k26.8–

95% CI from a paired bootstrap over the 1,507 pairs (10,000 resamples). An interval that includes 0 means no significant difference.

[W2-1] "Concerns about benchmark evaluation validity. It is unclear whether CoM2-Bench truly requires understanding the full query, as the distinguishing attribute alone might be sufficient for grounding." [Q4] "How well would models perform using only the distinguishing attribute instead of the full query?"

We reran Movie RAG with only the distinguishing attribute, and mIoU drops from 18.5 to 10.0 and [email protected] from 4.1 to 1.0. Grounding on CoM²-Bench therefore requires understanding all attributes of the query, both those the pair shares and the one that tells them apart.

Following the shared-query construction in §3.3, we prompted an LLM to rewrite each query into a distinguishing-attribute query that keeps only the distinguishing attribute in its original wording and drops the three shared attributes (e.g., "A scene in the ship bar." vs. "A scene in the terminal."). We then ran the Movie RAG pipeline from Table 3 on these queries.

QueryOracle chunk at top-1mIoU[email protected]
Full query (Table 3, Movie RAG)32.018.54.1
Distinguishing attribute only20.710.01.0
AuthorsOpenReview ↗

Response to Reviewer FYFA (2/2)

[W2-2] "Also, the improvement in PairSuccess may simply reflect better overall grounding accuracy."

The PairSuccess gain does not simply come from higher per-query accuracy: AWS also sends both confusable queries of a pair to the same window less often, reducing such pairs from 27.1% to 18.0%. Per-query accuracy alone can hide this failure. For example, because the two moments of every pair never overlap, a model that grounds both queries of each pair exactly on one of the two moments would reach 50% R@[email protected], above every method in Table 4, yet 0% PairSuccess. Among windows that land on either moment of a pair, AWS also lowers the share that lands on the wrong one from 19.7% to 14.0%, so AWS tells the two moments apart better, not just finds the scene more often.

[W2-3] "The imbalanced attribute distribution further raises concerns about the benchmark's coverage of different types of confusable moments." [Q5] "Could the authors report pairwise confusion rates for AWS, ideally broken down by contrast attribute, to show whether it actually improves discrimination between similar moments"

Each of the six main attribute types has at least 196 queries, enough to evaluate on its own, and on every type AWS improves grounding and reduces confusion with the partner moment (overall 19.7% → 14.0%). For every type, the retriever ranks the target above its paired moment in only 56.6–77.3% of cases, where chance is 50% (Table B), so every type remains hard to tell apart. The mIoU gain is significant on every type, and averaging the six types with equal weight gives a gain of +7.5, close to the overall gain of +8.3, so the gain does not depend on any single attribute.

Distinguishing attributeNmIoU Movie RAG → AWSConfusion rate† Movie RAG → AWS
setting131021.3 → 30.314.3 → 8.3
core action56615.9 → 26.026.3 → 22.1
subject38216.1 → 22.520.7 → 15.7
visual anchor33618.2 → 25.419.6 → 13.6
motion21214.3 → 20.229.9 → 23.1
interaction19616.6 → 23.128.5 → 16.8
all301418.5 → 26.8 (average of six main types 17.1 → 24.6)19.7 → 14.0

† Window-level: among selected windows that overlap either moment of a pair, the share that overlaps the partner moment rather than the target moment.

We agree that the distribution is uneven, so we add these per-type results to the Appendix.

[Q1] "Could the authors provide separate ablations for query decomposition, cascaded inspection, and window expansion?"

We ablated each component separately, and query decomposition is the main source of the gain: it adds +1.2 mIoU both with and without expansion. Window expansion triggers on only 2.5% of queries, raising mIoU on those queries from 17.2 to 22.5. All other settings remain fixed across rows.

ConfigDecomposeCascaded inspectionExpandmIoUMLLM calls / queryInput tokens / query
Single pass (Table 4)–––25.11172k
w/o decomposition & expansion✗✓✗25.52.098k
w/o expansion✓✓✗26.73.1111k
w/o decomposition✗✓✓25.62.0100k
Full AWS✓✓✓26.83.1112k

[Q2] "What are the average inference time and number of MLLM calls per query?"

AWS makes 3.1 MLLM calls per query on average (median 2) and spends 72 s of MLLM time per query, close to the 63 s of the single-pass verifier. For 63% of queries, inspection stops at the first candidate, so AWS uses about 112k input tokens per query, compared to 172k for the single-pass verifier.

Single passAWS
MLLM calls / query (mean / median)13.1 / 2
Input tokens / query172k112k
MLLM time / query (s, median)6372

In the revision, we will add the component ablation, a new Inference Cost section featuring the cost–accuracy comparison, the per-attribute breakdown, and the confusion and collapse statistics alongside PairSuccess.


We hope the compute-matched comparison, the per-component ablation, and the distinguishing-attribute control together address your concerns about the novelty and efficiency of AWS and the validity of the benchmark. If any point remains unresolved, we would be happy to discuss it further during the discussion period.

AuthorsOpenReview ↗

Follow-up to Reviewer FYFA

Dear Reviewer FYFA,

Thank you again for your thoughtful and detailed review. We have posted a detailed response and uploaded a revised manuscript with new experiments, clarifications, and corrections.

In particular, we added a compute-matched comparison against a scoring baseline that uses the same 27B model (AWS +2.9 mIoU at a matched token budget), and a control that keeps only the distinguishing attribute, where grounding drops from 18.5 to 10.0 mIoU once the shared attributes are removed. These directly address your questions on efficiency and on whether the benchmark truly requires the full query.

We would appreciate it if you could take another look at our response and let us know whether your concerns have been resolved. If so, we would respectfully appreciate your support for our work and your kind consideration of increasing your score accordingly.

Thank you for your time and effort.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank all reviewers for their constructive feedback. We are encouraged that reviewers found the confusable-pair benchmark and the PairSuccess metric well-motivated (FYFA, nuvd, QdSU), and valued both our controlled diagnosis of where retrieve-then-localize grounding fails (FYFA, nuvd, QdSU) and the substantial gains of AWS over Movie RAG (QdSU). Below, we summarize the new evidence addressing the shared concerns across reviews. Detailed responses are provided in the replies to each reviewer.


  • Agentic gain under the same model and matched compute (FYFA, nuvd, QdSU)

AWS outperforms two controls that use the same Qwen3.6-27B model, top-5 candidates, and character memory. The first is the single-pass verifier (AWS w/o inspection), which gives the 27B model all top-5 candidate windows in one call. Of the +8.3 total gain over Movie RAG, the single-pass verifier accounts for +6.6, and the agentic procedure adds the remaining +1.7, a statistically significant gain across the 1,507 pairs (paired bootstrap) that comes mainly from query decomposition. AWS reaches this with 1.5× fewer input tokens per query (112k vs. 172k) and similar MLLM time (72 vs. 63 s), as 63% of queries are accepted at the first candidate (3.1 MLLM calls per query on average, median 2). The second is a scoring baseline that asks the same model whether each candidate matches the full query and picks the highest-scoring one, and AWS outperforms it by 2.9 mIoU at a similar token budget. This gain follows from our diagnosis, as the retriever ranks candidates by what confusable moments share, so AWS checks the distinguishing attribute on its own.

In the revision, Section 5.2 adds Table 5 and an Inference cost paragraph that compare AWS with the single-pass verifier and the scoring baseline at matched input tokens, and the new Inference Cost appendix reports calls, tokens, and MLLM time.

  • Component-wise ablation study on AWS (FYFA, nuvd, QdSU)

We ablate query decomposition, cascaded inspection, and window expansion individually while holding the retriever, localizer, verifier, candidates, and character memory fixed. Compared to 26.8 mIoU for the full AWS and 25.1 for the single-pass verifier, removing query decomposition drops mIoU to 25.6, removing window expansion drops it to 26.7, and removing both lowers it to 25.5. The gain therefore comes mainly from decomposition, which checks the distinguishing attribute on its own. Window expansion is triggered in only 2.5% of queries, so it adds almost no cost, and on those queries it raises mIoU from 17.2 to 22.5, which suggests that learning when to expand can extend this gain to more queries.

In the revision, Table 4 lists each ablation with its MLLM calls and input tokens, and the new Statistical Significance appendix gives a bootstrap CI for each row.

  • Reliability of the benchmark labels (nuvd, QdSU, FYFA)

We checked the labels of the 28 movies outside the cross-review in three ways. For boundary tightness, we sampled 100 queries and had annotators mark their boundaries without seeing our labels, and their intervals match ours with a median IoU of 0.77. For moment uniqueness, annotators judged 100 sampled windows that AWS accepted outside both paired labeled moments, where another fitting moment is most likely to appear, and only 6 fit the full query. For performance consistency, Movie RAG (18.5 vs. 18.5 mIoU) and AWS (27.1 vs. 26.2) perform almost the same on these 28 movies as on the 18 cross-reviewed ones. The low baseline scores therefore come from retrieval failures rather than label ambiguity, as the same localizer reaches 53.8 mIoU given the oracle chunk compared to 18.5 for Movie RAG.

In the revision, Section 2 adds a Label Reliability paragraph with the three checks, detailed in the Human Verification appendix.

  • Imbalanced distinguishing attributes (FYFA, QdSU)

The distinguishing attributes are uneven (Table B), but each of the six main types still contains at least 196 queries. AWS improves mIoU over Movie RAG significantly on every type, and averaging the six types with equal weight gives a gain of +7.5, close to the overall gain of +8.3, so the gain does not depend on any single attribute.

In the revision, Section 5.1 states that the gain holds on every attribute, and the new Results by Contrast Attribute appendix reports per-attribute mIoU and confusion rates.


Additional analyses requested by individual reviewers, including the distinguishing-attribute-only results, the co-occurrence statistics of Inspect, the Recall@K headroom analysis, and pair confusion/collapse statistics, are detailed in the respective reviewer responses. In the revision, Section 2 states that grounding needs all attributes of a query, Section 5.1 adds the pair collapse statistic, and new appendices report pair confusion and retrieval recall with headroom. We hope these results address the reviewers' concerns and welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.