- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces GOOSE, a benchmark for discovering out of context videos within event streams. It contains 281 event streams with 4,085 videos, including 351 misleadingly attributed videos. Models must identify suspicious posts, select authentic reference videos, explain observable conflicts. Truthfully captioned unrelated posts serve as distractors. The evaluation separates detection accuracy from evidence grounded explanation quality. Experiments cover 15 multimodal models, direct inference, agent based exploration, external verification tools. Validation includes annotation agreement, human performance, human assessment of explanation scores.
Critical strengths
-
The task formulation addresses a meaningful verification problem. Models must discover suspicious posts within mixed event streams. Truthfully captioned unrelated posts test whether models confuse event irrelevance with false attribution. Explicit evidence requirements make unsupported correct predictions distinguishable from grounded verification. This combination provides a useful benchmark contribution.
-
The dataset construction includes substantive quality controls. Accepted cases require observable conflicts supported by at least two independently sourced authentic videos. Independent reviewers assess case validity. The reported admission agreement reaches a Cohen’s kappa of 0.91. Explanation evaluation includes a human assessment of 50 examples. The primary judge achieves an ICC of 0.713 against human ratings. These checks provide meaningful support for the dataset quality and evaluation reliability.
Critical weaknesses
-
The stated evaluation protocol conflicts with the provided prompts. Section 4.1 states that models do not know the number of OOC posts. Appendix G.1 explicitly reveals that single OOC streams contain exactly one target. These streams account for 227 of 281 cases. The prompts request only the top prediction. The task definition describes a complete ranking. This inconsistency limits confidence in the claimed evaluation under an unknown OOC count.
-
The experimental comparisons do not fully isolate the sources of performance changes. The isolated task permits an insufficient information response. The event stream task requires candidate selection or ranking. The reported improvement changes the available evidence, response format, task priors. The Agent comparison also changes evidence visibility through sequential browsing and a limit of 48 active images per round. These differences weaken the attribution of gains to cross video reasoning and losses to evidence fragmentation.
Questions for authors
- Protocol clarification: Which prompts produced the main results? Did models receive the exact OOC count for single OOC streams? Can the authors reconcile these prompts with the unknown count statement in Section 4.1?
2/ Experimental clarification: Is there a comparison that holds the prompt, response format, OOC count information fixed while removing or replacing authentic reference videos? Such a comparison would clarify how much improvement comes from relevant cross video evidence.
- Experimental clarification: Does the Agent performance gap persist when previously inspected evidence remains jointly accessible? A controlled comparison would help distinguish evidence fragmentation from limitations imposed by the controller.
7 comments
C1. Regarding OOC-Count Disclosure
\textcolor[rgb]{0.82,0.32,0.10}{\text{C1. Regarding OOC-Count Disclosure}}
Thank you for identifying an overgeneralized sentence in Sec. 4.1 that obscured the actual subset-specific protocol.
Concern C1. Protocol clarification: Which prompts produced the main results? Did models receive the exact OOC count for single OOC streams? Can the authors reconcile these prompts with the unknown count statement in Section 4.1?
\textcolor[rgb]{0.82,0.32,0.10}{\text{Response C1: Only multi-OOC prompts hide K.}}
\textcolor[rgb]{0.12,0.40,0.68}{\text{Protocol audit.}} The executed prompts and scorers are internally consistent; Sec. 4.1 incorrectly generalized the multi-OOC rule to every case.
| Case type | Cases | OOC count shown to model | Requested output | Scoring |
|---|---|---|---|---|
| Single-OOC | 227 | Exactly one | Top-1 with grounded evidence | OOC@1 |
| Multi-OOC | 54 | Undisclosed | Complete ranking with candidate-specific evidence | Recall@K using gold K |
\textcolor[rgb]{0.12,0.40,0.68}{\text{Conclusion.}} For 227 single-OOC cases, the prompt states exactly one target and OOC@1 scores the requested top prediction. For 54 multi-OOC cases, the prompt requests a complete ranking without revealing ; hidden gold is applied only during Recall@ scoring. The executed evaluation therefore matches its prompts and requires no score change. We will correct Sec. 4.1 and report both subsets explicitly.
C2. Regarding Reference Videos
\textcolor[rgb]{0.82,0.32,0.10}{\text{C2. Regarding Cross-Video Attribution}}
Thank you for asking for a controlled test of whether authentic cross-video evidence, rather than isolated recognition, drives the gain.
Concern C2 (Weakness 2 and Question 2). The isolated task permits an insufficient-information response, whereas the stream task requires selection or ranking; evidence, response format, and task priors therefore change. Is there a comparison that holds these factors fixed while removing or replacing authentic reference videos?
\textcolor[rgb]{0.82,0.32,0.10}{\text{Response C2: We add a forced independent-ranking control.}}
\textcolor[rgb]{0.12,0.40,0.68}{\text{Direct clarification.}} We agree that the original isolated task is a diagnostic, not a causal baseline: it receives a supplied gold OOC, permits “insufficient information,” and does not perform candidate discovery. We therefore do not use its 32.07% rejection rate to estimate the causal gain from group reasoning.
The at-least-two Real videos are a construction-time admission guarantee: annotators retain a group only when its Real evidence makes the OOC contradiction resolvable. They are not annotated or released as gold anchor IDs, and the model is neither told nor required to use a particular Real video. The retained Real posts reconstruct the target event from complementary views, so valid reasoning may use any subset of them.
Consequently, removing a post hoc “anchor” would first require us to designate the model's correct reasoning path after annotation, which would inject a new gold label into the experiment. Replacing a Real video would alter the factual event stream, while removing all Real videos would remove the target-event context altogether and reduce the task to isolated mismatch detection. Neither operation both preserves the task and changes only “reference evidence.” Thus, a prompt-identical “remove the references” ablation is not a well-defined benchmark intervention unless we first add oracle anchor labels that the task deliberately does not provide.
\textcolor[rgb]{0.12,0.40,0.68}{\text{Controlled experiment.}}
We instead introduce forced independent reranking on all 281 streams.
Independent input/output: after favorable public-claim filtering removes
Unrelated posts, Gemini receives the target claim and one candidate's 16 frames
with no other posts or anchors; it must return an ooc_score (0--100), reason, and cited
frames for every candidate, with no insufficient-information option. Scores are
aggregated into a Top-1 or complete ranking. Joint input/output: the same
model receives the complete stream and directly returns the Top-1 or ranking
with evidence. Both use the original OOC-count protocol and OOC@1/Recall@
scoring. This removes the insufficient-information confound and aligns the
final decision and metric. The model, underlying streams, labels, and count
protocol are fixed; the per-call JSON necessarily differs between candidate-wise
scoring and list-wise inference, so we present this as a matched decision-level
control rather than a literally prompt-identical ablation. The independent side
also receives favorable distractor filtering.
| Setting | Cross-video access | Overall OOC@ | vs. independent Gemini |
|---|---|---|---|
| RankVideo, independent | No | 44.09% | -12.83 |
| Gemini 3.8 Flash, independent | No | 56.92% | Ref. |
| Gemini 3.8 Flash, joint stream | Yes | \color[rgb]{0.62,0.20,0.18}{\mathbf{86.73\\%}} | \color[rgb]{0.62,0.20,0.18}{\mathbf{+29.81}} |
\textcolor[rgb]{0.12,0.40,0.68}{\text{Result and conclusion.}} Joint access improves OOC@ from 56.92% to 86.73% (), with 80 corrections versus 5 regressions on the 227 single-OOC streams (). Thus, authentic event-level cross-video context contributes substantial information beyond isolated recognition. This experiment does not assign the gain to one designated Real video, because no such inference-time designation exists; our claim is about the measured benefit of event-group access over forced independent scoring, not the causal contribution of one unique anchor.
Ref.
[1] Skow T, Martin A, Van Durme B, et al. Rankvideo: Reasoning reranking for text-to-video retrieval[J]. arXiv preprint arXiv:2602.02444, 2026.(https://arxiv.org/abs/2602.02444)
Response to Reviewer oG3v: Protocol Consistency, Cross-Video Evidence, and Agent Memory
Thank you for recognizing the task formulation, evidence requirement, and dataset controls. We organize the two weaknesses and their three clarification questions into three responses: the exact OOC-count protocol, controlled attribution of the cross-video gain, and whether persistent global evidence changes Agent performance.