- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper introduces CoM2-Bench, a benchmark for grounding confusable moments in full-length movies. Each query pair describes two different moments that share most attributes but differ in one distinguishing detail. The benchmark contains 3,014 queries from 46 movies.
2.The authors analyze a retrieve-then-localize pipeline and find that retrieving the correct video chunk is a major bottleneck. They then propose Agentic Window Search (AWS), which decomposes queries into individual conditions, checks retrieved video windows against these conditions, and expands windows when necessary. Without additional training, AWS improves mIoU from 18.5 to 26.8 on the proposed benchmark.
Critical strengths
- A meaningful benchmark with a clear evaluation objective.
I find the benchmark to be the strongest part of the paper. Existing temporal grounding evaluations do not necessarily reveal whether a model distinguishes two similar events within the same video. Constructing paired queries that differ in a specific attribute provides a useful way to examine this limitation. The PairSuccess metric also makes the paired evaluation explicit.
- The retrieval analysis is convincing and leads to a useful solution.
The oracle experiments clearly show that incorrect chunk retrieval is a major source of error. The comparison between random and confusable alternatives provides further evidence that retrievers often rely on shared attributes while overlooking distinguishing details.
AWS is relatively simple, but its improvement over Movie RAG is substantial. The additional results with different retrievers and on MomentSeeker also help support the general usefulness of evidence verification.
Critical weaknesses
- The benchmark annotation quality and coverage need stronger validation.
Although the dataset includes human verification, only 18 of the 46 movies receive a second independent review. The reported Cohen's kappa of 0.771 measures agreement on pair acceptance rather than agreement on temporal boundaries. Since the ground-truth intervals are initially generated by an MLLM and subsequently accepted or rejected by annotators, it is difficult to assess their boundary accuracy independently.
The distribution of distinguishing attributes is also uneven. According to Table B, 1,310 of the 3,014 queries use setting as the contrasting attribute, whereas only 12 involve temporal composition. This raises questions about how well the benchmark measures different types of fine-grained discrimination.
- The proposed verification procedure and its contribution require further analysis.
The Inspect prompt considers a statement satisfied if it holds somewhere within the candidate window, and explicitly allows different statements to hold at different times. This may be reasonable for an event spanning several actions, but it can also accept a window in which the conditions are satisfied by unrelated events. It is unclear how the method ensures that the verified attributes correspond to the same coherent event.
I also find the component analysis incomplete. In Table 4, a single-pass 27B verifier already achieves 25.1 mIoU, compared with 26.8 for the full AWS method. The paper does not separately quantify the effects of query decomposition, cascaded inspection, and window expansion. Moreover, AWS can make multiple 27B model calls per query, but the average inference cost is not reported. This makes it difficult to assess the benefit of the complete procedure relative to its computational overhead.
Questions for authors
-
Could the authors provide additional evidence of temporal boundary accuracy and query uniqueness, particularly for the movies without a second independent review? It would also be useful to see AWS results broken down by distinguishing attribute.
-
How does the verifier prevent different sub-queries from being satisfied by unrelated events within the same window? Have the authors considered checking whether these conditions form a temporally coherent event rather than merely appearing somewhere in the window?
-
Could the authors report separate ablations for query decomposition and window expansion, together with average model calls and inference time? A comparison with a simpler verifier under a similar computational budget would help clarify the practical value of the full AWS procedure.
7 comments
Response to Reviewer QdSU (2/2)
[W2-2] "I also find the component analysis incomplete. In Table 4, a single-pass 27B verifier already achieves 25.1 mIoU, compared with 26.8 for the full AWS method. The paper does not separately quantify the effects of query decomposition, cascaded inspection, and window expansion. Moreover, AWS can make multiple 27B model calls per query, but the average inference cost is not reported. …" [Q3] "Could the authors report separate ablations for query decomposition and window expansion, together with average model calls and inference time? A comparison with a simpler verifier under a similar computational budget would help clarify the practical value of the full AWS procedure."
We ablated each part, and query decomposition is the main source of the gain (+1.2 mIoU), while AWS averages 3.1 MLLM calls per query (median 2). Against a simpler baseline with the same Qwen3.6-27B model, AWS is 2.9 mIoU higher at a matched token budget, and this baseline only matches AWS at 1.4× the tokens. The effect of each part:
- Query decomposition: +1.2 mIoU both with and without expansion, the main source of the gain.
- Cascaded inspection: with the full query alone it reaches 25.5 mIoU, close to the single-pass verifier (AWS w/o inspection, which gives the 27B model all top-5 candidate windows in one call) while using 98k instead of 172k input tokens.
- Window expansion: triggers on 2.5% of queries, raising their mIoU from 17.2 to 22.5.
- Model calls and inference time: 3.1 calls per query on average (median 2), taking 72 s of MLLM time per query, close to the single-pass verifier (63 s).
| Config | Decompose | Cascaded inspection | Expand | mIoU | MLLM calls / query | Input tokens / query | MLLM time / query (s, median) |
|---|---|---|---|---|---|---|---|
| Single pass (Table 4) | – | – | – | 25.1 | 1 | 172k | 63 |
| w/o decomposition & expansion | ✗ | ✓ | ✗ | 25.5 | 2.0 | 98k | 36 |
| w/o expansion | ✓ | ✓ | ✗ | 26.7 | 3.1 | 111k | 70 |
| w/o decomposition | ✗ | ✓ | ✓ | 25.6 | 2.0 | 100k | 53 |
| Full AWS | ✓ | ✓ | ✓ | 26.8 | 3.1 | 112k | 72 |
Simpler baseline under a matched budget. We introduce a new baseline that uses the same 27B model as a reranker (the scoring baseline), with the same frames, candidates, and character memory as AWS. It scores each candidate independently via P(Yes | full query, window) and selects the highest-scoring window, without query decomposition or cascaded inspection. Scoring the top 2 candidates (K = 2) uses a token budget similar to AWS, and scoring the top 3 (K = 3) uses 1.4× the tokens of AWS.
| Method | Input tokens / query | mIoU | Δ vs AWS [95% CI] |
|---|---|---|---|
| scoring baseline, K = 2 (matched budget) | 101k | 23.9 | −2.9 [−3.7, −2.0] |
| scoring baseline, K = 3 (1.4× budget) | 152k | 27.3 | +0.5 [−0.3, +1.3] |
| AWS | 112k | 26.8 | – |
95% CI from a paired bootstrap over the 1,507 pairs (10,000 resamples). An interval that includes 0 means no significant difference.
In the revision, we will add the independent boundary and uniqueness checks of the 28 movies next to Table A, add the per-attribute breakdown alongside an explanation of the attribute distribution, describe how Inspect and the localizer divide this check, with the cooccurrence statistics above, and include both the full component ablation and a new Inference Cost section.
We hope the independent boundary and uniqueness checks, the co-occurrence check of accepted windows, and the full component ablation under a matched budget address the concerns raised in your review. If any point remains unresolved, we would be happy to discuss it further during the discussion period.
Follow-up to Reviewer QdSU
Dear Reviewer QdSU,
Thank you again for your thoughtful and detailed review. We have posted a detailed response and uploaded a revised manuscript with new experiments, clarifications, and corrections.
In particular, we added an independent boundary check on the 28 movies that did not receive a second review (median IoU 0.77 against annotators who did not see our labels), and a co-occurrence analysis showing that accepted windows combine statements from unrelated events in only 0.3% of cases. These directly address your concerns on boundary accuracy and on whether the verified attributes form a temporally coherent event.
We would appreciate it if you could take another look at our response and let us know whether your concerns have been resolved. If so, we would respectfully appreciate your support for our work and your kind consideration of increasing your score accordingly.
Thank you for your time and effort.
Response to Reviewer QdSU (1/2)
Thank you for your thoughtful feedback and for recognizing the clear evaluation objective of our benchmark, the PairSuccess metric, the retrieval analysis comparing random and confusable alternatives, and the substantial gain of AWS over Movie RAG. We also appreciate your positive note on our experiments across different retrievers and on MomentSeeker. We address each of your concerns and questions below.
We checked the 28 movies without a second review on both points. Annotators who had not seen our labels marked boundaries that match ours with a median IoU of 0.77, and of 100 sampled windows that AWS accepted outside both paired labeled moments, only 6 fit the full query. These checks measure boundaries and uniqueness directly, complementing the pair-acceptance κ.
For boundary tightness, the annotators marked 100 sampled queries inside clips that hide where the moment is (the moment plus 120 s of extra video), and 89% match our labels above 0.5 IoU. For moment uniqueness, we checked those windows because another fitting moment is most likely to be found there. For performance consistency, Movie RAG (18.5 vs. 18.5 mIoU) and AWS (27.1 vs. 26.2) perform almost the same on these 28 movies as on the 18 cross-reviewed ones. The low baseline scores therefore come from retrieval failures rather than label ambiguity, as the same localizer reaches 53.8 mIoU given the oracle chunk compared to 18.5 for Movie RAG (Table 2).
Each of the six main attribute types has at least 196 queries, and AWS improves mIoU significantly on every type. Temporal composition was already rare before human verification, with only 12 of the 4,080 candidate queries, and verification kept all 12. Averaging the six types with equal weight gives a gain of +7.5, close to the overall gain of +8.3, so the gain does not depend on any single attribute. Temporal composition needs two moments in the same movie with the same subject, action, and setting that differ only in the order of events, which rarely happens. We therefore report its 12 queries in a separate row and leave them out of the average.
We agree that the distribution is uneven, so we add these per-type results to the Appendix.
Accepted windows rarely combine statements from separate times: this happens in only 0.3% of them, measured with the
cooccurrencenote that the Inspect prompt (Appendix G) already asks Inspect to write for every window. In 97.9%, the span where all statements hold overlaps the ground-truth moment, and the remaining 1.8% point to a similar moment elsewhere in the window. We counted over the 1,289 accepted windows that contain the ground-truth moment and have such a note. In addition, the localizer grounds the full query as one contiguous span inside the accepted window, so the final answer is one contiguous span rather than separate pieces. We do not require all statements in a single frame because one event often spans several shots, such as an action followed by a reaction.