acceptodds
Under review as a conference paper at ICLR 2027

LongCoref-AV: Benchmarking Audio-Visual Identity Coreference in Long-Form Video

Abstract

An unresolved challenge for multimodal models processing long-form video is determining which faces, voices, and actions across distant moments belong to the same person. To measure this capability, we introduce LongCoref-AV, a benchmark of 1,126 human-verified multi-choice questions across 678 long-form videos. Questions identify a target individual only through visible or audible cues, such as clothing, gestures, or laughter, and are designed to rely on evidence from moments minutes apart. Each question is labeled with the category of evidence it requires to be answered correctly: voice, visual, or joint. Across 16 systems, closed frontier models far exceed a text-only baseline, open models lag far behind, and voice-dependent questions prove to be the most challenging across all models. When given only the annotated video segments containing the relevant evidence rather than the full video, open models improve far more than frontier models, suggesting that existing open models mainly fail to find evidence, while frontier models are more likely to locate it but often attribute it to the wrong person. In a separate identity-matching test, the two strongest open omni models both incorrectly label more than half of pairs depicting different individuals as the same person, despite their strong performance on multi-choice questions. This discrepancy shows that end-to-end question accuracy does not certify the intermediate identity links: models can select correct options from partial contextual cues without reliably distinguishing individuals across moments. Our findings expose persistent identity attribution as a critical weakness of multimodal LLMs, positioning LongCoref-AV to support more reliable attribution of actions to individuals in long-form video. Code and data will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
audio-visual benchmark, long video understanding, identity coreference, speaker attribution, MLLM, audio-visual reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_toJqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The manuscript introduces the concept of identity linking and introduces a benchmark for evaluating audio and visual coreference at different moments in videos. The benchmark contains 678 videos with human reviewed questions and were tested by 16 LLM models for a set of experiments in evidence retrieval, identity attribution, etc.

Critical strengths

The paper argue for an under-addressed problem in multimodal understanding of long videos, which is that it is now difficult to consistently associate person identities across audio-visual modalities and time. The set of experiments is extensive. These analyses provide insights and potential explanations to the current multimodal language models in completing this task.

Critical weaknesses

I'm not entirely convinced by the problem definition. It seems like a combination of speaker diarization and person-re-id, audio-visual association mapped onto a visual question-answering task. What is distinguishing identity co-reference from this combination, especially when the experiments are organized around disentangling these tasks? Does identity coreference introduce a distinct computational challenge that is not captured by the individual tasks?

Questions for authors

What if perpetual cues can be attributed to multiple people (i.e., there are multiple correct answers)? It could be the case with complex human interaction scene. how would the difficulty of the task change when there are way more people in the scene or varying number of people in the time segment of interest.

In terms of the generation of questions by GPT sol, how was it validated that the incorrrect answers are not obviously wrong, to avoid shortcut guessing? The options-only result is quite high. So even answering the question correctly may not imply that the models reflect this identitiy coreference capability but had done so through a process of answer elimination.

I think the discussion about open vs. closed models is a moot point as the models being open or close is not scientifically interesting and attributing what the model does well or not well towards openness is questionable. what are the factors underlying some cases and examples that could explain why certian models struggle with evidence retrieval and others with identity attribution?

5 comments
AuthorsOpenReview ↗

Response to Reviewer toJq (1/2)

We thank the reviewer for the review and the questions. We rebuilt the question set so that it cannot be answered without the video, reran every evaluation on it, and ran new experiments for the reviewer's questions. The revised set has 572 questions from 450 videos, and the revised paper reports its results on it. Please see the updated PDF, where all changes from the submission are highlighted.

W1. Is identity coreference more than diarization + re-identification + audio-visual association?

Our goal is to measure whether multimodal LLMs can perform identity coreference as one unified task, end to end, from raw video and audio. A question names a person only by a description and asks what that person said or did elsewhere in the video. Answering requires three abilities:

  1. Grounding the description. The target is specified only by an open-vocabulary perceptual description, such as "the person who holds up and reads the large old book" or "the person who lets out the lone celebratory 'Woo!'". The model must decide which person this description picks out. Diarization and re-identification produce anonymous voice clusters and face tracks with no connection to such descriptions.
  2. Linking that person across time and modalities. In 71% of questions, the evidence lies at two or more moments, separated by a median of 2.6 minutes, and it can include speech heard while the speaker is off screen, where audio-visual association does not apply because it needs the face and the voice at the same time. The model must carry the person's identity from the cue to the evidence, possibly through intermediate moments.
  3. Attributing the content to that person, rather than to the other people in the video, whose statements and actions appear as the distractors. All three must be done without cast lists, reference faces or voice samples, or speaker segmentation.

Diarization, re-identification, and audio-visual association each contribute to these abilities, but none of them is the task, and none of them connects its output to the person the question describes. We tested this directly: giving GPT-5.5 what such a pipeline produces (a speaker-attributed transcript and a face-track timeline, as text, in addition to its frames) raises accuracy only from 80.4% to 85.0%, and no model given this text clearly exceeds its own video-and-audio score. The tools supply anonymous clusters and tracks; the link from the described person to a face and from that face to a voice is still left to the model.

The experiments split the task into steps to diagnose where models fail, not to define it. The task definition, with the three abilities above, is now stated at the start of the introduction.

Q1. Cues that fit several people, and the effect of the number of people

Multiple correct answers. Uniqueness is enforced at two stages. When the attribute inventory is built, each visual cue carries a flag for whether it fits exactly one person in the video; the generator writes cues from this inventory, and cues include disambiguating detail where needed, as in "the man who had been seated on the viewer's left". Human reviewers then checked correctness and uniqueness for every submitted question, and a question whose cue fit more than one person, or whose answer was not unique, was repaired or rejected. The revised set keeps these human-verified target people and evidence moments. Rewritten questions were checked again against the video.

Number of people. Accuracy by the number of audible speakers in the video (93, 157, 100, and 222 questions):

2 speakers345 or more
Gemini-3.1-Pro94.6%77.7%87.0%83.8%
Gemini-4-Argon91.4%87.9%83.0%82.9%
GPT-5.584.9%79.0%83.0%78.4%
Qwen3-Omni-Instruct73.1%70.7%71.0%65.8%
Nemotron-3-Nano-Omni68.8%68.2%71.0%59.5%

The task gets harder with more people: from two speakers to five or more, every model loses 6–11 points. The decline is steady for Gemini-4-Argon and not monotonic for Gemini-3.1-Pro, which is lowest at three speakers.

AuthorsOpenReview ↗

Response to Reviewer toJq (2/2)

Q2. Validating the distractors

Where the wrong options come from. We traced every wrong option to its source. 44% of the wrong options are something another person in the same video actually says or does, with a pointer checked against the transcript or a frame. Ruling such an option out requires knowing who said or did it, which is the identity resolution the benchmark tests.

Can they be eliminated without the video? No. Each question was tested without the video by readers from three model families, and options that could be guessed were replaced or the question was dropped. Given the options alone, GPT-5.5 scores 18.2% on the revised set (40.9% on the submitted set) and Gemini-3.1-Pro 23.4%; given the question and options, Gemini-3.1-Pro scores 27.8% (was 59.6%) and GPT-5.5 23.3% (was 58.9%); chance is 25%. With the video, the same two models score 84.4% and 80.4%. One surface regularity remains: a word-overlap rule that picks the option worded least like the other three is right in 38% of questions. It does not explain the models' scores, as the options-only results show; we report it in the paper and will reword the affected answers for the camera-ready. In the target-swapped question pairs (reply to Reviewer KaaD), the options are identical within a pair, so eliminating options cannot help at all.

How they are validated. Every changed option was checked against the video and the annotations, so that exactly one option is correct for the person asked about.

Q3. Open vs. closed models

We agree that openness itself is not a scientific variable, and the paper does not use it to explain results. We refer to the strongest systems (Gemini-4-Argon, Gemini-3.1-Pro, GPT-5.5) as frontier models, a label for the best-performing systems rather than for how they are licensed; the results table groups systems by access and input type only for readability. The differences between models are explained by measurable factors, which the results on the revised set show directly:

  • Comparing voices is hard for every model; comparing faces is not. Given the two moments directly, every model tells two faces apart almost perfectly (97–99%) but reaches only 69–75% on two voices.
  • Finding the moments is what limits some models in the same-person test. Qwen3-Omni-Instruct compares two faces at 96.8% when given the moments but at chance (50.2%) when it has to find them in the whole video. For the multiple-choice questions, telling a model where to look in the unchanged video changes accuracy by −3.0 to +4.3 points, never significantly.
  • How much of the input reaches the model. MiniCPM-o's official inference path keeps at most 64 one-second frame-and-audio blocks, so on a whole video it hears only 8–17% of the track; it scores 9.7% on voice questions against 56.1% on face questions. GPT-Audio, which hears the audio but sees nothing, scores 33.7%.
  • How speech is received. Gemma-4-31B, which reads a transcript instead of hearing audio, reaches 72.0%, on par with Qwen3-Omni-Instruct with native audio (69.2%).
  • What the mistakes are. For Gemini-3.1-Pro, Gemini-4-Argon, GPT-5.5, and Qwen3-Omni-Instruct, 43–46% of the mistakes pick something that another person in the video really said or did, that is, the right content attributed to the wrong person; 10–15% pick the right person at the wrong moment.

Dear Reviewer toJq,

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We would greatly appreciate any feedback, and are happy to clarify anything further before the deadline. If our response resolves your concerns, we would be grateful if you would consider updating your score.

Thank you for your time.

Best,

LongCoref-AV Authors

Reviewer_KaaDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LongCoref-AV, a benchmark of 1,126 human-verified multiple-choice questions across 678 videos, targeting identity association across temporally separated visual and auditory observations. It evaluates 16 systems and provides diagnostics involving evidence-restricted inputs, identity matching, modality ablations, and post-training. The main finding is that strong question-answering performance does not necessarily imply reliable identity matching.

Critical strengths
  • The benchmark addresses a practically important issue: reliably attributing speech and actions to individuals across a video. Human verification and annotated evidence spans make it a useful resource for studying this problem.

  • The comparison between question answering and direct identity matching is informative. Together with text-only baselines and modality ablations, it highlights limitations that aggregate QA accuracy can obscure.

Critical weaknesses
  • The validity of QA accuracy as a measure of identity coreference is insufficiently established. Text-only accuracy approaches 60%, options-only accuracy reaches 40.9%, and selecting the longest option achieves 35.3%. These results indicate substantial shortcuts. Human verification establishes answer correctness but does not demonstrate that answering requires the intended identity links. The discrepancy between QA and identity-matching scores therefore also raises concerns about the benchmark’s primary metric. Its incremental contribution over audiovisual diarization and speaker-centric benchmarks such as AV-SpeakerBench and DramaSR needs sharper justification.

  • The diagnostics do not adequately isolate the claimed failure mechanisms. Evidence-restricted evaluation changes context length and sampling density, while omitting an identity-link moment for 519 questions, limiting the attribution of gains to retrieval. Likewise, limited improvements from post-training on one model family do not establish an architectural bottleneck. The text-side voiceprint experiment is not a substitute for a complete audiovisual identity-tracking baseline. These limitations weaken the broader conclusions about why models fail and which remedies are ineffective.

Questions for authors
  • What evidence shows that successful QA depends on correctly resolving the intended identity links? Do you have matched, per-question results relating QA correctness to the correctness of the necessary links, or target-swapped question pairs whose answers must change?

  • Does the retrieval-versus-attribution conclusion hold when all necessary identity-link moments are retained and frame sampling is controlled between full-video and evidence-restricted conditions?

  • Can you clarify what capability is newly evaluated beyond audiovisual diarization and existing speaker-centric QA, and whether a pipeline combining face tracking, active-speaker detection, and speaker verification has been evaluated?

4 comments
AuthorsOpenReview ↗

Response to Reviewer KaaD (1/2)

We thank the reviewer for the detailed review and questions. We rebuilt the question set to remove the text shortcuts, reran every evaluation on it, and ran the experiments the reviewer asked for: target-swapped question pairs, per-question identity links, the clip controls, and a pipeline-information condition. The revised set has 572 questions from 450 videos; please see the updated PDF (changes highlighted).

W1 / Q1. Does QA accuracy measure identity coreference?

Text-only and option-only solvability. The revised set is built on the human-verified question pool of the submission: every question keeps its video, target person, and annotated evidence moments. Each question was tested without the video by readers from three model families (GPT, Mistral, Qwen), in four option orders, with and without the question text. Wherever a reader beat chance, we (1) replaced the wrong options with things that other people in the same video actually say or do, each checked against the transcript or a frame; (2) reworded the correct answer if that was not enough; (3) otherwise rewrote the person description or the question; and (4) dropped the question if it could still be guessed.

Without the video (chance is 25%):

No videoSubmitted (1,126)Revised (572)
Gemini-3.1-Pro, question + options59.6%27.8%
GPT-5.5, question + options58.9%23.3%
GPT-5.5, options only40.9%18.2%
Always pick the longest option35.3%16.8%

Models from families that played no part in building the revised set (Gemini-3.1-Pro, Gemma-4, GLM-4.5V, Nemotron, phi-4) score 18–28%. Gemini-4-Argon, released after the set was built, scores 33.7%. With the video, Gemini-4-Argon reaches 85.7%, Gemini-3.1-Pro 84.4%, and GPT-5.5 80.4%.

One surface regularity remains. The correct answer is often the option whose wording differs most from the other three, and a word-overlap rule that picks that option is right in 38% of questions. It does not explain the models' scores: given the options alone, GPT-5.5 scores 18.2% and Gemini-3.1-Pro 23.4%. We report this in the paper and will reword the affected answers for the camera-ready.

Target-swapped question pairs. We built the pairs the reviewer describes. The two questions of a pair use the same video and exactly the same four options; they differ only in which person they ask about, so the correct answer differs. A model that does not resolve who is who gets both right only by chance (6.25%). Speech pairs ask about two people identified by something they say, with spoken lines as options; visual pairs ask about people identified by appearance, with visible details as options. Share of pairs with both questions right, with the video / without it:

Speech pairs (151)Visual pairs (100)
Gemini-4-Argon43.7% / 13.9%72.0% / 8.0%
Gemini-3.1-Pro29.1% / 11.9%75.0% / 14.0%
GPT-5.533.1% / 13.2%89.0% / 16.0%
Qwen3-Omni-Instruct18.5% / 6.6%45.0% / 4.0%

Every model gets far more pairs right with the video than without it (p < 0.001 for each model and pair type). This is controlled evidence that the video is what lets a model follow the person. The pairs also show how much ordinary accuracy overstates this ability: for "who said what", the best model gets both questions right in under half of the pairs. Visual identity is much easier (72–89% of pairs for Gemini-4-Argon, Gemini-3.1-Pro, and GPT-5.5).

Per-question identity links. For each question we also gave the model five short clips, the described moment and the four option moments, with no option text, and asked which clip shows the same person (chance is 25%). On all 233 questions whose four options each have a recorded moment, Gemini-4-Argon finds the right person in 56.2%, answers the question correctly in 84.5%, and does both in 48.9%. For Gemini-3.1-Pro the numbers are 49.8%, 82.4%, and 42.9%; for Qwen3-Omni-Instruct, on the 228 questions it answered, 41.2%, 67.1%, and 29.4%. A correct answer therefore does not guarantee that the model can pick out the person in isolation. The revised paper states that accuracy alone does not certify the identity links and reports the target-swapped pair accuracy and this joint score alongside it.

W1 / Q3. What is new beyond diarization and speaker-centric QA

Diarization yields anonymous voice clusters, re-identification yields anonymous face tracks, and active-speaker detection links a voice to a face only when the two co-occur. Identity coreference additionally requires (i) grounding an open-vocabulary perceptual description ("the person who holds up and reads the large old book") to one person, which none of these subtasks performs; (ii) linking that person across minutes and across modalities, including speech heard while the speaker is off screen, where active-speaker detection does not apply; and (iii) doing both without cast lists, reference samples, or pre-segmented turns.

AuthorsOpenReview ↗

Response to Reviewer KaaD (2/2)

This is also what separates LongCoref-AV from the two benchmarks the reviewer names. AV-SpeakerBench uses 5–30 s single-scene clips, so no person has to be followed across time. DramaSR covers full episodes but supplies a cast list and pre-segmented utterances, which reduces attribution to closed-set recognition. We state this contrast in §1 and §2.

Pipeline information. We gave each model what a diarization and face-tracking pipeline produces, as text: a speaker-attributed transcript and a face-track timeline, together with 32 frames.

32 frames + plain transcript32 frames only+ pipeline textNative video + audio
GPT-5.580.4%–85.0%–
Gemini-3.1-Pro–64.2%83.0%84.4%
Gemini-4-Argon–73.1%86.7%85.7%

Over a plain transcript, the pipeline's speaker labels and face tracks add 4.5 points for GPT-5.5, and for the Gemini models the pipeline text supplies roughly what the audio track gives them; no model clearly exceeds its native video-and-audio result. The tools supply anonymous voice clusters and face tracks; the link from the described person to a face and from that face to a voice still has to be made by the model, and that link is what the questions test.

W2 / Q2. Diagnostics and failure mechanisms

  • Evidence-restricted inputs. We ran two controls on the revised questions; details are in our reply to Reviewer Q4Dq (W2). (1) Giving the model the times of the relevant moments while keeping the full video and its sampling unchanged (233 questions): Gemini-3.1-Pro moves from 82.4% to 86.7%, Gemini-4-Argon from 84.5% to 87.1%, and Qwen3-Omni-Instruct from 67.4% to 64.4%, none significantly. (2) Clips that keep all four options' moments give Qwen3-Omni-Instruct only 69.5% (68.7% at the denser sampling rate), against 84.5% for the relevant clips. Its clip gain therefore comes from the clips hiding the other options' evidence, not from search or sampling density. The abstract's statement about open and frontier models is replaced by these model-specific results.
  • Identity-link test with the moments supplied directly. With the two moments cut out and given to the model, face pairs are solved (Gemini-3.1-Pro 98.4%, Gemini-4-Argon 98.6%, Qwen3-Omni-Instruct 96.8%; Qwen3-Omni-Instruct was at chance on face pairs in the whole video), while voice pairs stay at 69–75% for every model. Comparing two voices is the step that remains hard, independent of retrieval.
  • Post-training. We make no architectural-bottleneck claim and state only what we tested. We reran the training experiments on the revised set (Qwen3-Omni-Thinking under the per-turn protocol, 572 questions, two seeds; the frozen model scores 64.1%):
    • Training improves evidence-grounded reasoning. The rate at which the model's most-cited turns include the annotated evidence rises from 48.1% (frozen) to 55.3% with full-parameter tuning (+7.3 points), and by 6.9 and 7.2 points with GRPO. Answers that are correct and cite the evidence rise from 36.5% to 41.2%, and correct answers without evidence fall from 27.5% to 25.0%.
    • Accuracy and identity linking do not change reliably. Full-parameter tuning changes accuracy by +2.3 and +2.1 points after one and two epochs, GRPO on top of it by +1.4 and +1.3, and LoRA by −0.2, all within run-to-run variation. Balanced accuracy on voice pairs stays at 59–63% (frozen: 60.7%).
    • Frame placement and audio length have no reliable effect. No placement of dense frames changes accuracy by more than 1.7 points for any checkpoint, and training and testing with whole-turn audio gives 66.5%, against 66.2% with 3 s clips. We withdraw the submitted claims that turn selection and whole-turn audio help.
    • The 4B replication agrees. Removing the voiceprint token changes accuracy by −0.9 points; only dropping the base frame grid hurts (−7.0). Training thus teaches the model to cite the right evidence, but not yet to link identities. The 17,893-question training split is released with the benchmark so that further recipes can be compared on the same footing.
  • Text-side voiceprint study. This study is not intended as an identity-tracking baseline. It answers a specific question that matters for agentic systems: does letting a model choose which voiceprint checks to run beat fixed access to the same verdicts? On the revised set it does not. The agent trails fixed access for GPT-5.5 (51.9% vs. 61.9% with all verdicts) and Qwen3-Omni-Instruct (21.2% vs. 43.5%), and is level with it for the Gemini models (Gemini-3.1-Pro 62.9% vs. 61.8%, Gemini-4-Argon 68.5% vs. 69.4%). We state this scope in the Related Work section (Section 2).
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer KaaD,

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We would greatly appreciate any feedback, and are happy to clarify anything further before the deadline. If our response resolves your concerns, we would be grateful if you would consider updating your score.

Thank you for your time.

Best,

LongCoref-AV Authors

AuthorsOpenReview ↗

Final follow-up before the discussion deadline

Dear Reviewer KaaD,

As the author-reviewer discussion closes today, we wanted to follow up once more. We would be grateful for any feedback on our response and the revised paper, and are happy to clarify anything before the deadline. If our response resolves your concerns, we would appreciate it if you would consider raising your rating.

Thank you for your time.

Best,

LongCoref-AV Authors

Reviewer_Q4DqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LongCoref-AV, a benchmark of 1,126 human-verified questions over 678 long-form videos. Questions identify individuals through perceptual cues and require associating their appearances, voices, speech, or actions across time. Beyond evaluating 16 systems, the paper studies modality contributions, evidence-restricted inputs, identity matching, post-training, and identity-grounded captioning.

Critical strengths
  • The task addresses an important limitation of long-video understanding: correct content recognition does not ensure correct attribution to individuals.
  • The evaluation goes beyond aggregate question accuracy. Text-only baselines, option rotation, modality ablations, identity-pair tests, and captioning diagnostics provide complementary perspectives.
  • The appendix documents substantial implementation detail and openly discusses several confounds and negative results. The accompanying identity timelines and evidence annotations could support useful follow-up research.
Critical weaknesses
  • Residual shortcuts materially limit interpretation of the main score. Question-and-option baselines approach 60%, options alone reach 40.9%, and a longest-option rule reaches 35.3%. Reporting these controls is valuable, but the benchmark still lacks an independently validated subset that more clearly requires the intended identity chain. Human reviewers saw the answer key; independent human task performance and question-level evidence-necessity agreement are not reported.
  • The retrieval-versus-attribution diagnosis is not sufficiently controlled. Evidence-restricted inputs change context length and sampling density, while omitting an identity-link moment for 519 questions. Consequently, their gains cannot cleanly isolate retrieval, and remaining errors cannot be assigned specifically to identity attribution. The direct identity-pair protocol also requires locating moments within the full video, leaving retrieval as a possible contributor.
Questions for authors
  • Can a representative subset be independently answered and annotated by humans without access to the answer key, including full-input and modality-removed conditions?
  • Can evidence-restricted evaluation preserve the referring cue, intermediate identity-link moments, and answer evidence while controlling sampling density? How do identity-pair results change when the two target clips are supplied directly?
  • When will the benchmark be open-sourced?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Q4Dq (1/2)

We thank the reviewer for the careful reading, including the appendix, and for the constructive questions. We rebuilt the question set to remove the text shortcuts, reran every evaluation on it, and ran the retrieval controls the reviewer suggested. The revised set has 572 questions from 450 videos, and the revised paper reports its results on it. Please see the updated PDF, where all changes from the submission are highlighted.

W1 / Q1. Residual shortcuts and validation

Text-only and option-only solvability. The revised set is built on the human-verified question pool of the submission: every question keeps its video, target person, and annotated evidence moments. Each question was tested without the video by readers from three model families (GPT, Mistral, Qwen), in four option orders, with and without the question text. Wherever a reader beat chance, we (1) replaced the wrong options with things that other people in the same video actually say or do, each checked against the transcript or a frame; (2) reworded the correct answer if that was not enough; (3) otherwise rewrote the person description or the question; and (4) dropped the question if it could still be guessed.

Without the video (chance is 25%):

No videoSubmitted (1,126)Revised (572)
Gemini-3.1-Pro, question + options59.6%27.8%
GPT-5.5, question + options58.9%23.3%
GPT-5.5, options only40.9%18.2%
Always pick the longest option35.3%16.8%

Models from families that played no part in building the revised set (Gemini-3.1-Pro, Gemma-4, GLM-4.5V, Nemotron, phi-4) score 18–28%. Gemini-4-Argon, released after the set was built, scores 33.7%. The main table of the paper gives each system's text-only score as the mean over four option orders (Gemini-3.1-Pro 28.1%, GPT-5.5 24.2%, Gemini-4-Argon 33.7%).

With the video, Gemini-4-Argon reaches 85.7%, Gemini-3.1-Pro 84.4%, and GPT-5.5 80.4%, so Gemini-3.1-Pro is now 57 points above its no-video score (27 points on the submitted set).

One surface regularity remains. The correct answer is often the option whose wording differs most from the other three, and a word-overlap rule that picks that option is right in 38% of questions. It does not explain the models' scores: given the options alone, GPT-5.5 scores 18.2% and Gemini-3.1-Pro 23.4%. We report this in the paper and will reword the affected answers for the camera-ready.

Human verification. Every question in the pool was verified by human reviewers against the video for target necessity (replacing the target with another plausible person in the video must change the answer), correctness and uniqueness within the annotated spans, and transcript insufficiency (dialogue alone must not determine the answer). The revised set keeps these human-verified videos, target people, and evidence moments. The rewritten wording and options were checked against the video and the annotations. Whether each question needs the evidence it is labeled with is measured directly below.

Do questions need the evidence they are labeled with? Each question is now labeled by the identity link it tests, read from its annotated evidence: two voices (31 questions), two faces (330), or a face and a voice (211); the paper's tables call these voice, visual, and joint. Replacing the audio with noise gives:

Full input → audio replaced by noiseTwo voicesTwo facesFace and voice
Gemini-3.1-Pro71.0% → 25.8%80.9% → 82.7%91.9% → 54.0%
Gemini-4-Argon77.4% → 48.4%83.6% → 84.2%90.0% → 77.3%
Qwen3-Omni-Instruct45.2% → 25.8%70.6% → 71.8%70.6% → 33.2%

Face questions are unaffected, voice questions fall to or toward chance, and face-and-voice questions lose 13–38 points. The questions need the evidence their label says they need.

AuthorsOpenReview ↗

Response to Reviewer Q4Dq (2/2)

W2 / Q2. Retrieval vs. attribution

Evidence-restricted inputs. We ran two controls on the revised questions and rewrote the claim.

(1) Telling the model where to look without changing its input. The model receives the same full video with the same sampling; the question additionally lists the times of the described moment and of each option's moment, sorted and unlabeled. Both controls use all 233 questions whose four options each have a recorded moment. Gemini-3.1-Pro moves from 82.4% to 86.7%, Gemini-4-Argon from 84.5% to 87.1%, and Qwen3-Omni-Instruct from 67.4% to 64.4%; none of these changes is significant.

(2) Clips that keep every option's moments. We cut the described moment and all four options' moments (±15 s) into one clip, about 30% of the video. Qwen3-Omni-Instruct scores 69.5% on it at the full video's frame rate and 68.7% at the denser rate of the relevant clips, against 67.4% on the full video. The relevant clips, which keep only the correct answer's moments, give 84.5%. For the Gemini models, the relevant clips add 3.9 points over these matched clips (Gemini-3.1-Pro 83.7% → 87.6%, Gemini-4-Argon 85.0% → 88.8%).

The Gemini models gain at most a few points from either control. For Qwen3-Omni, the clip gain is neither search nor sampling density: it comes from the clips hiding the other options' evidence, so the model no longer has to decide whose moment is the right one. The abstract and results now report these model-specific results in place of the statement about open and frontier models.

Identity-link test with the two moments supplied directly. We cut the two moments out of the video (two audio clips for a voice pair, two face crops for a face pair) and asked the same SAME/DIFFERENT question on pairs drawn from the identity-link test (balanced accuracy; chance is 50%):

Voice pairsFace pairs
Gemini-3.1-Pro74.8%98.4%
Gemini-4-Argon68.6%98.6%
Qwen3-Omni-Instruct70.2%96.8%

Once the moments are given, comparing faces is solved for every model. In the whole-video protocol, Qwen3-Omni-Instruct was at chance on face pairs (50.2%), so its earlier failure was a failure to locate the moments. Comparing voices stays at 69–75% for every model, including the newest Gemini: deciding whether two voices belong to the same person is the step that remains hard, independent of retrieval.

Q3. Release

The full benchmark will be released publicly with the camera-ready version, including the videos, questions, annotations, training split, evaluation code, and model predictions.

Thanks for the additional experiments. The matched-clip controls and direct identity-matching tests help clarify the role of retrieval and address much of my concern.

My remaining concern is the human validation of the final question set. Since many questions and options were rewritten, could you report independent human evaluation on a representative subset, with annotators answering without seeing the answer key? Including both full-input and modality-removed conditions would help confirm that the revised questions have unambiguous answers and require the intended evidence.

AuthorsOpenReview ↗

Response to Reviewer Q4Dq: independent blind human evaluation

We thank the reviewer for this suggestion. We conducted the requested study: independent annotators answered the revised questions without access to the answer key, both with the full input and with the required modality removed. The results show that the questions require the evidence their labels state and cannot be answered from the text alone.

Design. We sampled 36 questions from the revised set, stratified by required-evidence label (7 voice, 16 joint, and 13 visual) and covering all six scene families. The median video length is 7.4 minutes. Voice questions are over-represented relative to the full set so that every label is measured, and we report all results by label.

We recruited 23 participants on Prolific with no connection to the authors. Each used a desktop computer with audio and was paid for their time. Each participant answered 18 questions as text only (question and options), 9 with the full video, and 9 with the modality required by the label removed: the audio is muted for voice and joint questions, and the picture is removed for visual questions.

The design is counterbalanced. Within every group of four participants, each question appears once with the full video, once with the modality removed, and twice as text only, and no participant sees any question more than once. The study collected 828 answers: 207 with the full video, 207 with the modality removed, and 414 as text only.

The format matches the benchmark (four options, forced choice), and the order of the options is rotated across participants. Unlabeled buttons below the player jump to the moment of the person description and to the moment of every option, so they help navigation without indicating the answer.

Results. Accuracy (%). Chance is 25%.

Full videoModality removedText onlyFull − removed
All questions (36)58.925.628.0+33.3
Voice (7)57.515.029.6+42.5
Joint (16)50.030.424.5+19.6
Visual (13)70.725.331.5+45.3
  • The questions require the labeled evidence.
    • Removing the labeled modality reduces accuracy from 58.9% to 25.6%, which is chance.
    • Full video is more accurate on 31 of the 36 questions.
    • The result replicates across recruitment waves: the first 8 participants score 63.9 / 30.6 / 25.0 (full / removed / text), and the remaining 15 score 56.3 / 23.0 / 29.6.
  • The text alone does not reveal the answer. Text-only accuracy is at chance overall and for every label.
  • Participants recognize when the evidence is missing. They rated their answer as a guess on 59% of modality-removed and 79% of text-only questions, against 9% with the full video.

Joint questions. The gap is smaller for joint questions, and two properties of individual questions contribute to this:

  • Some joint questions use abstract or compound framings (for example, "which pair of claims…"), which lower accuracy even with the full video. Across all labels, full-video accuracy is 65.1% on the 26 concretely worded questions and 43.1% on the 10 abstractly framed ones, while modality-removed accuracy stays near chance for both (28.7% and 17.5%).
  • A few videos contain burned-in subtitles, which convey the speech even when the audio is muted.

Both are properties of particular questions rather than of the joint label, and we address both in the final version.

Human accuracy. Human accuracy with the full video is lower than that of the strongest models, and answers rated as certain were correct 73% of the time. Each question concerns a video of about seven minutes, and participants spent a median of two minutes per question.

Scope of this study. Given the length of the discussion period, this study was designed and run within a brief period and should be read as an initial, focused validation rather than a complete one. The overall result is clear and replicates across recruitment waves, but it rests on 36 questions and 23 participants. The per-label estimates, particularly for the joint label, are therefore less precise and may be treated as preliminary. For the final version, we will run the same protocol on a larger sample whose label proportions match the benchmark, with more participants per question. In the final version, we will also detect burned-in subtitles across the full question set, rewrite abstract framings into concrete wording, and report all model results on the revised set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.