acceptodds
Under review as a conference paper at ICLR 2027

GOOSE: Which Video Does Not Belong? Multi-Video Reasoning for Out-of-Context Detection in Event Streams

Abstract

Recent multimodal large language models (MLLMs) show strong multi-video reasoning, yet social-media event streams pose a distinct challenge: authentic footage can still be *contextually false* when it originates from a different event. Existing benchmarks typically ask predefined questions about known targets, rather than requiring models to reconstruct an event and discover conflicting posts. To address this gap, we introduce **GOOSE** (**G**roup reasoning **O**ver **O**ut-of-context videos in **S**ocial-media **E**vents), a benchmark for discovering genuine footage falsely presented as evidence of another event. GOOSE contains 281 real-world cases, 351 OOC videos, and 4,085 videos across 6 domains, covering six evidence mechanisms involving scene, entity, temporal–causal, physical, multi-view, and provenance conflicts. It emphasizes **group-level evidence aggregation**, **event-level reasoning**, and **open-ended investigation**: models must discover unspecified OOC posts, while truthfully captioned off-event posts prevent relevance-based shortcuts. We evaluate two settings: Direct-IO jointly processes uniformly sampled frames, whereas a VideoAgent-style ReAct framework iteratively selects and inspects videos. Discovery is measured by case-macro OOC@K, and a rubric-guided native-video MLLM Judge evaluates grounded cross-video explanations; the Evidence-Grounded Score (EGS) jointly captures both aspects. Across 14 leading MLLMs, Gemini 3.8 Flash retrieves **86.73%** of gold OOCs but achieves only **58.56% EGS**, showing that correct discovery does not guarantee a valid explanation. VideoAgent also often trails Direct-IO because fragmented inspection weakens global event reconstruction and candidate–anchor comparison. These findings identify grounded cross-video verification, rather than visual plausibility or provenance recall alone, as the central bottleneck in real-world OOC detection.

Primary area:
datasets and benchmarks
Keywords:
out-of-context misinformation, multi-video reasoning, event-stream understanding, multimodal large language models, video misinformation detection, evidence-grounded reasoning, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oG3vOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces GOOSE, a benchmark for discovering out of context videos within event streams. It contains 281 event streams with 4,085 videos, including 351 misleadingly attributed videos. Models must identify suspicious posts, select authentic reference videos, explain observable conflicts. Truthfully captioned unrelated posts serve as distractors. The evaluation separates detection accuracy from evidence grounded explanation quality. Experiments cover 15 multimodal models, direct inference, agent based exploration, external verification tools. Validation includes annotation agreement, human performance, human assessment of explanation scores.

Critical strengths
  1. The task formulation addresses a meaningful verification problem. Models must discover suspicious posts within mixed event streams. Truthfully captioned unrelated posts test whether models confuse event irrelevance with false attribution. Explicit evidence requirements make unsupported correct predictions distinguishable from grounded verification. This combination provides a useful benchmark contribution.

  2. The dataset construction includes substantive quality controls. Accepted cases require observable conflicts supported by at least two independently sourced authentic videos. Independent reviewers assess case validity. The reported admission agreement reaches a Cohen’s kappa of 0.91. Explanation evaluation includes a human assessment of 50 examples. The primary judge achieves an ICC of 0.713 against human ratings. These checks provide meaningful support for the dataset quality and evaluation reliability.

Critical weaknesses
  1. The stated evaluation protocol conflicts with the provided prompts. Section 4.1 states that models do not know the number of OOC posts. Appendix G.1 explicitly reveals that single OOC streams contain exactly one target. These streams account for 227 of 281 cases. The prompts request only the top prediction. The task definition describes a complete ranking. This inconsistency limits confidence in the claimed evaluation under an unknown OOC count.

  2. The experimental comparisons do not fully isolate the sources of performance changes. The isolated task permits an insufficient information response. The event stream task requires candidate selection or ranking. The reported improvement changes the available evidence, response format, task priors. The Agent comparison also changes evidence visibility through sequential browsing and a limit of 48 active images per round. These differences weaken the attribution of gains to cross video reasoning and losses to evidence fragmentation.

Questions for authors
  1. Protocol clarification: Which prompts produced the main results? Did models receive the exact OOC count for single OOC streams? Can the authors reconcile these prompts with the unknown count statement in Section 4.1?

2/ Experimental clarification: Is there a comparison that holds the prompt, response format, OOC count information fixed while removing or replacing authentic reference videos? Such a comparison would clarify how much improvement comes from relevant cross video evidence.

  1. Experimental clarification: Does the Agent performance gap persist when previously inspected evidence remains jointly accessible? A controlled comparison would help distinguish evidence fragmentation from limitations imposed by the controller.
7 comments
AuthorsOpenReview ↗

Response to Reviewer oG3v: Protocol Consistency, Cross-Video Evidence, and Agent Memory

Thank you for recognizing the task formulation, evidence requirement, and dataset controls. We organize the two weaknesses and their three clarification questions into three responses: the exact OOC-count protocol, controlled attribution of the cross-video gain, and whether persistent global evidence changes Agent performance.

ConcernWhat we didWhat it shows
C1. OOC-count protocol\textcolor[rgb]{0.12,0.40,0.68}{\text{Prompt, output, and scorer audit for both subsets.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{227 disclose one target; only 54 multi-OOC cases hide K.}}
C2. Cross-video attribution\textcolor[rgb]{0.12,0.40,0.68}{\text{Forced independent reranking + anchor clarification.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{Joint access adds 29.81 points beyond independent scoring.}}
C3. Agent evidence fragmentation\textcolor[rgb]{0.12,0.40,0.68}{\text{Full-set global-to-local Agent test.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{It gains 5.00 Acc. and 4.56 EGS points.}}
AuthorsOpenReview ↗

C1. Regarding OOC-Count Disclosure

\textcolor[rgb]{0.82,0.32,0.10}{\text{C1. Regarding OOC-Count Disclosure}}

Thank you for identifying an overgeneralized sentence in Sec. 4.1 that obscured the actual subset-specific protocol.

Concern C1. Protocol clarification: Which prompts produced the main results? Did models receive the exact OOC count for single OOC streams? Can the authors reconcile these prompts with the unknown count statement in Section 4.1?

\textcolor[rgb]{0.82,0.32,0.10}{\text{Response C1: Only multi-OOC prompts hide K.}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Protocol audit.}} The executed prompts and scorers are internally consistent; Sec. 4.1 incorrectly generalized the multi-OOC rule to every case.

Case typeCasesOOC count shown to modelRequested outputScoring
Single-OOC227Exactly oneTop-1 with grounded evidenceOOC@1
Multi-OOC54UndisclosedComplete ranking with candidate-specific evidenceRecall@K using gold K

\textcolor[rgb]{0.12,0.40,0.68}{\text{Conclusion.}} For 227 single-OOC cases, the prompt states exactly one target and OOC@1 scores the requested top prediction. For 54 multi-OOC cases, the prompt requests a complete ranking without revealing KK; hidden gold KK is applied only during Recall@KK scoring. The executed evaluation therefore matches its prompts and requires no score change. We will correct Sec. 4.1 and report both subsets explicitly.

AuthorsOpenReview ↗

C2. Regarding Reference Videos

\textcolor[rgb]{0.82,0.32,0.10}{\text{C2. Regarding Cross-Video Attribution}}

Thank you for asking for a controlled test of whether authentic cross-video evidence, rather than isolated recognition, drives the gain.

Concern C2 (Weakness 2 and Question 2). The isolated task permits an insufficient-information response, whereas the stream task requires selection or ranking; evidence, response format, and task priors therefore change. Is there a comparison that holds these factors fixed while removing or replacing authentic reference videos?

\textcolor[rgb]{0.82,0.32,0.10}{\text{Response C2: We add a forced independent-ranking control.}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Direct clarification.}} We agree that the original isolated task is a diagnostic, not a causal baseline: it receives a supplied gold OOC, permits “insufficient information,” and does not perform candidate discovery. We therefore do not use its 32.07% rejection rate to estimate the causal gain from group reasoning.

The at-least-two Real videos are a construction-time admission guarantee: annotators retain a group only when its Real evidence makes the OOC contradiction resolvable. They are not annotated or released as gold anchor IDs, and the model is neither told nor required to use a particular Real video. The retained Real posts reconstruct the target event from complementary views, so valid reasoning may use any subset of them.

Consequently, removing a post hoc “anchor” would first require us to designate the model's correct reasoning path after annotation, which would inject a new gold label into the experiment. Replacing a Real video would alter the factual event stream, while removing all Real videos would remove the target-event context altogether and reduce the task to isolated mismatch detection. Neither operation both preserves the task and changes only “reference evidence.” Thus, a prompt-identical “remove the references” ablation is not a well-defined benchmark intervention unless we first add oracle anchor labels that the task deliberately does not provide.

\textcolor[rgb]{0.12,0.40,0.68}{\text{Controlled experiment.}} We instead introduce forced independent reranking on all 281 streams. Independent input/output: after favorable public-claim filtering removes Unrelated posts, Gemini receives the target claim and one candidate's 16 frames with no other posts or anchors; it must return an ooc_score (0--100), reason, and cited frames for every candidate, with no insufficient-information option. Scores are aggregated into a Top-1 or complete ranking. Joint input/output: the same model receives the complete stream and directly returns the Top-1 or ranking with evidence. Both use the original OOC-count protocol and OOC@1/Recall@KK scoring. This removes the insufficient-information confound and aligns the final decision and metric. The model, underlying streams, labels, and count protocol are fixed; the per-call JSON necessarily differs between candidate-wise scoring and list-wise inference, so we present this as a matched decision-level control rather than a literally prompt-identical ablation. The independent side also receives favorable distractor filtering.

SettingCross-video accessOverall OOC@KKΔ\Delta vs. independent Gemini
RankVideo, independentNo44.09%-12.83
Gemini 3.8 Flash, independentNo56.92%Ref.
Gemini 3.8 Flash, joint streamYes\color[rgb]{0.62,0.20,0.18}{\mathbf{86.73\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{+29.81}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Result and conclusion.}} Joint access improves OOC@KK from 56.92% to 86.73% (Δ=+29.81\Delta=+29.81), with 80 corrections versus 5 regressions on the 227 single-OOC streams (p=1.81×10−18p=1.81\times10^{-18}). Thus, authentic event-level cross-video context contributes substantial information beyond isolated recognition. This experiment does not assign the gain to one designated Real video, because no such inference-time designation exists; our claim is about the measured benefit of event-group access over forced independent scoring, not the causal contribution of one unique anchor.

Ref.

[1] Skow T, Martin A, Van Durme B, et al. Rankvideo: Reasoning reranking for text-to-video retrieval[J]. arXiv preprint arXiv:2602.02444, 2026.(https://arxiv.org/abs/2602.02444)

Reviewer_fo74OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce GOOSE, a benchmark of 281 real-world social-media event streams in which models must find out-of-context videos among real and unrelated posts, and explain each detection using evidence from the other videos. They evaluate 15 MLLMs with direct input and a ReAct-style agent, with analyses across six evidence mechanisms, tool use, isolated versus stream-level detection, and a human reference.

Critical strengths
  1. The resource produced in the paper is a great resource for misinformation research.
Critical weaknesses

W1. The claimed gaps are overstated, and the paper doesn’t test whether existing pipelines already solve the task.

Overstated Gaps:

  1. The intro argues that existing work starts from a pre-selected target, but reranking an event-query candidate list already requires removing off-event videos from a pool. Tasks like MultiVENT2.0 [1], the MAGMaR shared task [2], and rerankers like Qwen3-VL-Reranker [3] and RankVideo [4] already test/model these capabilities.
  2. The intro argues that event-level reasoning over many videos doesn't exist. But WikiVideo [6] and the MAGMaR generation track [2] require building an account of a real-world event from multiple videos and grounding claims in specific ones. GOOSE’s Task 2 also provides a target-event description, so it is conditioned on an event query much like these tasks.

[1] https://arxiv.org/abs/2410.11619 [2] https://aclanthology.org/2026.magmar-main.16/ [3] https://arxiv.org/abs/2601.04720 [4] https://arxiv.org/abs/2602.02444 [6] https://arxiv.org/abs/2504.00939

Pipeline

The task introduced in this paper may be already solvable with a simple pipeline.

  • Unrelated posts are truthfully captioned as other events, so a text-only check of whether the caption claims target an event should remove them. Thus, the task could be solved by [Video] -> [Caption] -> [LLM/Reranker/Decision Model]. If this works group reasoning isn't needed. But another text baseline (a list-wise reranker or LLM inferencing over all the captions) could also do this if the cross-source reasoning is required.
  • For example, this system [1] that operates over text-only for video is able to differentiate between videos very well.

[1] https://arxiv.org/abs/2606.07924

W2. Flawed evaluation.

  1. The headline discovery-explanation gap depends on the judge model.
  2. Cross-judge agreement is low.
  3. Every core OOC is supposed to be plausible on its own, yet Gemini 3.1 Pro rejects 48% of OOCs in isolation, and isolated detection is about 32% overall, meaning a sizable fraction of cases can be solved without the group.
Questions for authors

m/a

7 comments
AuthorsOpenReview ↗

Response to Reviewer fo74: Task Novelty, Independent Baselines, and Judge Robustness

Thank you for identifying the closest retrieval and multi-video generation literature. We understand the review as questioning the task boundary, whether a simple independent pipeline is sufficient, whether Judge choice drives the conclusion, and whether isolated cues invalidate group reasoning. Our responses separate these four issues:

ConcernWhat we didWhat it shows
C1. Novelty boundary\textcolor[rgb]{0.12,0.40,0.68}{\text{Task-objective comparison with cited prior work.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{GOOSE tests false attribution, not relevance.}}
C2. Simple independent pipeline\textcolor[rgb]{0.12,0.40,0.68}{\text{Independent Gemini and RankVideo baselines; 281 streams.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{Independent scoring falls 29.81 points short.}}
C3. Judge dependence\textcolor[rgb]{0.12,0.40,0.68}{\text{Four-Judge and four-human re-scoring.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{5/6 pairs preserved; human ICCs 0.856 and 0.713.}}
C4. Isolated detectability\textcolor[rgb]{0.12,0.40,0.68}{\text{Same-model independent vs. joint control.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{Isolated cues do not explain the 29.81-point joint gain.}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Clarification.}} The benchmark supplies normalized event-claim text, not generated video descriptions. A text filter can favorably remove Unrelated posts, but it cannot separate Real from OOC because both assert the target event.

AuthorsOpenReview ↗

C1. Regarding the Novelty Boundary

\textcolor[rgb]{0.82,0.32,0.10}{\text{C1. Regarding the Novelty Boundary}}

Thank you for identifying the need to distinguish GOOSE from adjacent retrieval and multi-video generation tasks.

Concern C1. Event-query reranking already removes off-event videos from a candidate pool, while WikiVideo and MAGMaR already reconstruct and ground real-world events from multiple videos. Because Task 2 also provides a target event description, the novelty boundary is unclear.

\textcolor[rgb]{0.82,0.32,0.10}{\text{Response C1: GOOSE tests false attribution, not relevance.}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Clarification.}} We agree that retrieval, reranking, and multi-video synthesis already exist. The distinction is the required decision and what counts as an error:

Work / task familyGiven input and objectiveBoundary relative to GOOSE
MultiVENT 2.0, Qwen3-VL-Reranker, RankVideoRetrieve or rerank videos for a supplied event/query by relevanceTests query--video relevance, not whether on-topic footage is falsely attributed to that event
WikiVideo, MAGMaRRetrieve videos and/or generate an evidence-grounded event article or reportTests grounded synthesis from relevant videos, not discovery of a misleading post among mixed social-media roles
GOOSEGiven a target event and a mixed Real/OOC/Unrelated stream, discover the unspecified falsely attributed post and ground the contradiction in other posts.Separates contextual falsity from both relevance and synthesis quality

\textcolor[rgb]{0.12,0.40,0.68}{\text{Conclusion.}} An honestly captioned Unrelated video should be rejected by retrieval but must not be labeled OOC; conversely, an OOC may be highly relevant while false in time, place, entity, or event phase. We will add these citations, narrow the novelty claim, and replace “target-free” with suspect-unspecified: the event is given, but the misleading post is not.

Ref.

[1] Kriz R, Sanders K, Etter D, et al. Multivent 2.0: A massive multilingual benchmark for event-centric video retrieval[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 24149-24158.(https://arxiv.org/abs/2410.11619);

[2] Li M, Zhang Y, Long D, et al. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking[J]. arXiv preprint arXiv:2601.04720, 2026.(https://arxiv.org/abs/2601.04720);

[3] Skow T, Martin A, Van Durme B, et al. Rankvideo: Reasoning reranking for text-to-video retrieval[J]. arXiv preprint arXiv:2602.02444, 2026.(https://arxiv.org/abs/2602.02444);

[4] Martin A, Kriz R, Walden W G, et al. Wikivideo: Article generation from multiple videos[C]//Findings of the Association for Computational Linguistics: ACL 2026. 2026: 45-70.(https://arxiv.org/abs/2504.00939);

[5] Alexander Martin, Dengjia Zhang, Joel Brogan, Francis Ferraro, Jeremy Gwinnup, Reno Kriz, Teng Long, Kenton Murray, Andrew Yates, and Xiang Xiang. 2026. Findings of the MAGMaR 2026 Shared Task. In Proceedings of the 2nd Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2026), pages 144–150, San Diego, USA. Association for Computational Linguistics.(https://aclanthology.org/2026.magmar-main.16/).

AuthorsOpenReview ↗

C2. Regarding Independent Pipelines

\textcolor[rgb]{0.82,0.32,0.10}{\text{C2. Regarding Independent Pipelines}}

Thank you for proposing a concrete alternative pipeline that can be tested directly rather than discussed hypothetically.

Concern C2. Unrelated posts may be removed by checking whether their text claims the target event, after which a video-captioning system and an LLM or reranker might solve the task without group reasoning.

\textcolor[rgb]{0.82,0.32,0.10}{\text{Response C2: Independent scoring falls 29.81 points short.}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Clarification.}} The benchmark supplies normalized event-claim text, not generated video descriptions. A text filter can favorably remove Unrelated posts, but it cannot separate Real from OOC because both assert the target event.

\textcolor[rgb]{0.12,0.40,0.68}{\text{Controlled experiment.}} Purpose. Test whether isolated evidence alone can reproduce the full-stream discovery result. Independent input/output. Gemini receives the target-event claim and one candidate's 16 uniformly sampled frames, with no other posts or anchors, and returns a JSON ooc_score (0--100), a short reason, and cited frame IDs; these scores are aggregated into a Top-1 or complete ranking. Joint input/output. The same model receives the target claim and all sampled posts together and directly returns the Top-1 prediction or ranking with evidence. Both conditions cover all 281 streams and use the same OOC@1/Recall@KK metrics; RankVideo provides an additional independent-reranker check.

SettingInformation per decisionSingle OOC@1Multi Recall@KKOverall OOC@KKΔ\Delta vs. independent Gemini
RankVideo, independentClaim + one candidate42.73%49.78%44.09%-12.83
Gemini 3.8 Flash, independentClaim + one candidate52.42%75.83%56.92%Ref.
Gemini 3.8 Flash, joint streamClaim + all sampled posts\color[rgb]{0.62,0.20,0.18}{\mathbf{85.46\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{92.07\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{86.73\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{+29.81}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Result and conclusion.}} The same-model comparison controls model capability, while RankVideo checks a cited specialist reranker. Independent cues solve a real subset, but joint access corrects 80 errors while causing only 5 regressions. Caption-mediated systems and C2F-RAG remain relevant modular architectures, but both reintroduce cross-post aggregation; they do not show that group information is unnecessary.

Ref.

[1] Skow T, Martin A, Van Durme B, et al. Rankvideo: Reasoning reranking for text-to-video retrieval[J]. arXiv preprint arXiv:2602.02444, 2026.(https://arxiv.org/abs/2602.02444)

[2] Martin A, Zhang D, Brogan J, et al. Findings of the MAGMaR 2026 Shared Task[C]//Proceedings of the 2nd Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2026). 2026: 144-150.( https://aclanthology.org/2026.magmar-main.16/、https://aclanthology.org/2026.magmar-main.12.pdf).

Reviewer_uqNPOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces GOOSE (Group Reasoning Over Out-of-context videos in Social-media Events), a benchmark designed for discovering genuine footage falsely presented as evidence of another event. GOOSE comprises 281 real-world cases, 351 OOC videos, and 4,085 videos across 6 domains, covering six evidence mechanisms involving scene, entity, temporal–causal, physical, multi-view, and provenance conflicts. The benchmark emphasizes group-level evidence aggregation, event-level reasoning, and open-ended investigation, requiring models to discover unspecified OOC posts while using truthfully captioned off-event posts to prevent relevance-based shortcuts. The authors evaluate two settings: Direct-IO, which jointly processes uniformly sampled frames, and a VideoAgent-style ReAct framework that iteratively selects and inspects videos. Discovery is measured by case-macro OOC@K, and a rubric-guided native-video MLLM Judge evaluates grounded cross-video explanations, with the Evidence-Grounded Score (EGS) jointly capturing both aspects. Extensive experiments across 14 leading MLLMs reveal that while the top-performing model (Gemini 3.8 Flash) retrieves 86.73% of gold OOCs, it achieves only 58.56% EGS, demonstrating that correct discovery does not guarantee a valid explanation.

Critical strengths
  1. The motivation behind the dataset construction is very clear and well-justified. Moving beyond traditional sample-level verification to evaluate target-free, cross-video group reasoning in noisy social-media event streams addresses a realistic and critical gap in existing benchmarks.
  2. The evaluation of models is very comprehensive, covering a wide range of the latest and strongest open-weight and proprietary MLLMs. By benchmarking leading models across both direct inference and agentic settings, the paper provides a thorough assessment of the current frontier capabilities.
Critical weaknesses
  1. The paper’s claims appear stronger than the actual setup supports. The benchmark is repeatedly described as requiring group-level reasoning because each retained OOC video is “plausible in isolation” and contains “no isolated shortcut” (Sections 2.1, 3.1, A.2). However, the paper’s own isolated evaluation shows non-trivial single-video detectability: the seven-model average isolated rejection rate is 32.07%, and Gemini 3.1 Pro reaches 48.43% (Sections 4.2, 5.3). This suggests that the benchmark does not fully eliminate isolated cues available to current MLLMs, so the “only resolvable through cross-video comparison” narrative appears overstated.

  2. The 16-frame evaluation interface may be too sparse to represent the original videos faithfully. Benchmark construction and human auditing are described over full, playable source videos, whose average duration is 203 seconds and whose maximum duration is 21,485 seconds (Tables 1 and 8). In contrast, the main Direct-IO evaluation uses only 16 uniformly sampled frames per post, with even lower per-video sampling when a stream exceeds 15 posts due to the 250-image cap (Sections 4.1, E.2). This creates a mismatch between construction-time evidence availability and test-time evidence availability

  3. The human verification pipeline is not described with enough operational detail to assess its rigor. The paper states that annotators and reviewers check playability, auditability, independent anchors, and provenance through multi-stage human review (Section 3.1, Appendix A.4). However, given the dataset scale (4,085 videos, average duration 203 seconds, with a long tail up to nearly 6 hours), the practical execution of this process is difficult to assess from the current description alone. The paper does not report annotation time, viewing policy for long videos, or whether videos were fully watched or partially inspected. As a result, the rigor and consistency of the full-video human verification process remain doubtful.

Questions for authors

See weakness.

11 comments
AuthorsOpenReview ↗

Response to Reviewer uqNP: Group Reasoning, Frame Coverage, and Verification Rigor

Thank you for recognizing the motivation and broad model evaluation. We understand the review as raising three concerns: whether group context is truly useful, whether sampled frames are sufficient, and whether human verification was operationally rigorous. We address each concern with a scoped correction or new evidence:

ConcernWhat we didWhat it shows
C1. Group reasoning vs. isolated cues\textcolor[rgb]{0.12,0.40,0.68}{\text{Same-model independent vs. joint test; 281 streams.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{Joint access adds 29.81 points; 80 corrections vs. 5 regressions.}}
C2. Sampled-frame sufficiency\textcolor[rgb]{0.12,0.40,0.68}{\text{8/16/32-frame test; two models, 50 cases.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{More frames help Qwen, but not Gemini.}}
C3. Human verification rigor\textcolor[rgb]{0.12,0.40,0.68}{\text{Viewing policy, five-stage review, and adjudication audit.}}\textcolor[rgb]{0.82,0.32,0.10}{\text{Admission agreement is 94.7 percent with kappa 0.91.}}
AuthorsOpenReview ↗

C1. Regarding Group Reasoning

\textcolor[rgb]{0.82,0.32,0.10}{\text{C1. Regarding Group Reasoning}}

Thank you for identifying that our original wording overstated the necessity of group reasoning.

Concern C1. The benchmark is described as requiring group-level reasoning because each OOC is “plausible in isolation” and contains “no isolated shortcut,” yet the isolated rejection rate is 32.07% on average and reaches 48.43% for Gemini 3.1 Pro.

\textcolor[rgb]{0.82,0.32,0.10}{\text{Response C1: Joint context adds 29.81 points beyond isolated cues.}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Clarification.}} The original isolated diagnostic asks whether one supplied gold OOC conflicts with the event claim; it does not require finding that OOC among Real and Unrelated posts. Its 32.07% rejection rate therefore establishes that some cases expose isolated cues or memorized knowledge, not that group context is unnecessary. We will remove the universal phrases “no isolated shortcut” and “only resolvable.”

\textcolor[rgb]{0.12,0.40,0.68}{\text{Controlled experiment.}} Purpose. Test whether isolated evidence alone can reproduce the full-stream discovery result. Independent input/output. Gemini receives the target-event claim and one candidate's 16 uniformly sampled frames, with no other posts or anchors, and returns a JSON ooc_score (0--100), a short reason, and cited frame IDs; these scores are aggregated into a Top-1 or complete ranking. Joint input/output. The same model receives the target claim and all sampled posts together and directly returns the Top-1 prediction or ranking with evidence. Both conditions cover all 281 streams and use the same OOC@1/Recall@KK metrics; RankVideo provides an additional independent-reranker check.

SettingInformation per decisionSingle OOC@1Multi Recall@KKOverall OOC@KKΔ\Delta vs. independent Gemini
RankVideo, independentClaim + one candidate42.73%49.78%44.09%-12.83
Gemini 3.8 Flash, independentClaim + one candidate52.42%75.83%56.92%Ref.
Gemini 3.8 Flash, joint streamClaim + all sampled posts\color[rgb]{0.62,0.20,0.18}{\mathbf{85.46\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{92.07\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{86.73\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{+29.81}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Result and conclusion.}} Joint access improves the same model from 56.92% to 86.73%; on 227 single-OOC cases, it produces 80 corrections versus 5 regressions, and the paired asymmetry is significant (p=1.81×10−18p=1.81\times10^{-18}). Thus, isolated evidence solves a meaningful subset, but it cannot explain the main joint-stream result. Our corrected claim is measured rather than universal: cross-video context provides substantial additional evidence.

Ref.

[1] Skow T, Martin A, Van Durme B, et al. Rankvideo: Reasoning reranking for text-to-video retrieval[J]. arXiv preprint arXiv:2602.02444, 2026.(https://arxiv.org/abs/2602.02444)

AuthorsOpenReview ↗

C2. Regarding Frame Sampling

\textcolor[rgb]{0.82,0.32,0.10}{\text{C2. Regarding Frame Sampling}}

Thank you for raising whether sparse frame sampling, rather than reasoning ability, is the dominant bottleneck.

Concern C2. The main Direct-IO evaluation uses only 16 uniformly sampled frames per post, and fewer for large streams, whereas construction and human auditing use the full videos. This may make the evaluation interface too sparse to represent the original videos faithfully.

\textcolor[rgb]{0.82,0.32,0.10}{\text{Response C2: More sampled frames do not consistently improve performance.}}

\textcolor[rgb]{0.12,0.40,0.68}{\text{Controlled experiment.}} We rerun Direct IO on the same fixed 200-stream subset with 8, 16, and 32 uniformly sampled frames per post for Gemini 3.8 Flash, Qwen3.8 Flash, and the Qwen3-VL-8B-Instruct. The subset preserves the benchmark's single-/multi-OOC and domain proportions (162 single; 38 multi) through stratified sampling. The same cases, prompts, decoding, labels, and scoring are used in all three conditions. Thus, only the number of sampled frames changes. If 16-frame sampling were the principal bottleneck, moving to 32 frames should improve both discovery and grounded explanation across models.

ModelFramesAcc.EGSΔ\Delta Acc. / EGS vs. 16 frames
Gemini 3.8 Flash879.88%53.72%-1.28 / -3.24
16\color[rgb]{0.62,0.20,0.18}{\mathbf{81.15\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{56.96\\%}}Ref.
3279.58%56.05%-1.57 / -0.91
Qwen3.8 Flash856.40%26.37%+0.29 / +0.58
1656.11%25.79%Ref.
32\color[rgb]{0.62,0.20,0.18}{\mathbf{61.75\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{26.95\\%}}+5.64 / +1.16
Qwen3-VL-8B-Instruct89.35%4.35%-2.07 / -0.62
1611.42%4.97%Ref.
32\color[rgb]{0.62,0.20,0.18}{\mathbf{11.50\\%}}\color[rgb]{0.62,0.20,0.18}{\mathbf{5.10\\%}}+0.08 / +0.13

\textcolor[rgb]{0.12,0.40,0.68}{\text{Result and conclusion.}} The effect is not monotonic across models and does not support 16-frame sampling as the principal bottleneck. From 16 to 32 frames, Qwen3.8 Flash improves from 56.11% to 61.75% Acc. and from 25.79% to 26.95% EGS. In contrast, Gemini decreases from 81.15% to 79.58% Acc. and from 56.96% to 56.05% EGS, while Qwen3-VL-8B-Instruct changes only from 11.42% to 11.50% Acc. and from 4.97% to 5.10% EGS. More frames can help an individual model, but they do not produce a consistent gain across models. Moreover, the discovery--grounding gap remains at every frame budget, so sparse sampling alone cannot explain the benchmark-wide performance gap.

Comments on the paper

1 comment
AuthorsOpenReview ↗

REBUTTAL SUMMARY FOR THE PROGRAM CHAIRS AND AREA CHAIR

\textcolor[rgb]{0.82,0.32,0.10}{\text{REBUTTAL SUMMARY FOR THE PROGRAM CHAIRS AND AREA CHAIR}}

Dear Program Chairs and Area Chair,

We sincerely thank you for the time devoted to this review process, and we also appreciate the Area Chair's efforts to encourage reviewer participation. The constructive guidance helped us narrow claims, clarify the protocol, and add controlled experiments. Our current scores are 4, 3, and 2.

\textcolor[rgb]{0.12,0.40,0.68}{\text{Discussion and response by reviewer.}}

ReviewerReviewer's feedbackOur responseDiscussion outcome
oG3v (4)Clarify OOC-count disclosure, cross-video evidence, and Agent evidence fragmentation.We audited the protocol, added an independent-versus-joint control, and evaluated a global-to-local Agent with persistent evidence.The review strongly supports the task formulation, evidence requirement, and dataset controls. We sincerely appreciate this positive assessment and guidance.
uqNP (3)Reconsider universal group-reasoning claims, frame coverage, and human verification.We narrowed the claim, added same-model independent/joint and 8/16/32-frame experiments, and documented human review and adjudication.The reviewer acknowledged our rebuttal effort and suggested higher-rate sampling and difficulty stratification. We will include the requested 0.5/1 FPS evaluation, complete results, and difficulty analysis in the final repository and camera-ready version.
fo74 (2)Clarify task novelty, independent pipelines, Judge dependence, and isolated cues.We added a revised task comparison, independent baselines, four alternative Judges, human calibration, and a same-model control.While no follow-up was posted, the overlapping questions about isolated cues and cross-video evidence were also discussed with the other reviewers, and the corresponding responses were positively received. Every remaining point was also answered in detail.

\textcolor[rgb]{0.12,0.40,0.68}{\text{Main strengths recognized in the reviews.}}

  • GOOSE connects multi-video understanding with multimodal social-media misinformation and provides a valuable benchmark resource. (fo74, oG3v, uqNP)
  • Its suspect-unspecified mixed-stream task evaluates both discovery and evidence-grounded explanation, rather than relevance alone. (oG3v)
  • The evaluation covers proprietary and open MLLMs under Direct-IO and Agent settings. (uqNP)
  • The dataset uses explicit Real/OOC/Unrelated roles, six falsifiable mechanisms, and controlled admission criteria. (oG3v)

The benchmark required two months of work by five senior undergraduate annotators, independent review and adjudication, and approximately USD 5,000 in model evaluation, reflecting the care invested in this public research resource.

\textcolor[rgb]{0.12,0.40,0.68}{\text{What the rebuttal established and changed.}}

  • Cross-video access provides a measured benefit, not a universal necessity. On all 281 streams, the same Gemini model improves from 56.92% under independent scoring to 86.73% with joint access: +29.81 points, with 80 corrections versus 5 regressions. We removed “only resolvable” and “no isolated shortcut.”
  • The protocol is internally consistent. Single-OOC cases disclose one target and request Top-1; only multi-OOC cases hide KK and request a complete ranking. We corrected the overgeneralized paper description.
  • The frame effect is not monotonic across models. On the same stratified 200 streams, 32 frames help Qwen3.8 Flash, slightly reduce Gemini performance, and leave Qwen3-VL-8B-Instruct nearly unchanged. We will extend this analysis following uqNP's recommendation.
  • The conclusions do not depend on one Judge. Four alternative Judges preserve five of six pairwise model orderings; Human-Human ICC is 0.856 and Primary-Human ICC is 0.713.
  • Evidence fragmentation is a demonstrated Agent bottleneck. Persistent global evidence improves the original Agent by 5.00 accuracy and 4.56 EGS points; we no longer present it as the only cause.

We uploaded a revised manuscript with corrected claims, protocol clarification, expanded related work, annotation details, and new experiments. We again thank the Program Chairs and Area Chair for your time and service. We sincerely hope that GOOSE will receive your support, and would be deeply grateful if the complete discussion and responses could be considered in the final assessment.

We wish you all the best with the remainder of the review process.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.