- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The manuscript introduces the concept of identity linking and introduces a benchmark for evaluating audio and visual coreference at different moments in videos. The benchmark contains 678 videos with human reviewed questions and were tested by 16 LLM models for a set of experiments in evidence retrieval, identity attribution, etc.
Critical strengths
The paper argue for an under-addressed problem in multimodal understanding of long videos, which is that it is now difficult to consistently associate person identities across audio-visual modalities and time. The set of experiments is extensive. These analyses provide insights and potential explanations to the current multimodal language models in completing this task.
Critical weaknesses
I'm not entirely convinced by the problem definition. It seems like a combination of speaker diarization and person-re-id, audio-visual association mapped onto a visual question-answering task. What is distinguishing identity co-reference from this combination, especially when the experiments are organized around disentangling these tasks? Does identity coreference introduce a distinct computational challenge that is not captured by the individual tasks?
Questions for authors
What if perpetual cues can be attributed to multiple people (i.e., there are multiple correct answers)? It could be the case with complex human interaction scene. how would the difficulty of the task change when there are way more people in the scene or varying number of people in the time segment of interest.
In terms of the generation of questions by GPT sol, how was it validated that the incorrrect answers are not obviously wrong, to avoid shortcut guessing? The options-only result is quite high. So even answering the question correctly may not imply that the models reflect this identitiy coreference capability but had done so through a process of answer elimination.
I think the discussion about open vs. closed models is a moot point as the models being open or close is not scientifically interesting and attributing what the model does well or not well towards openness is questionable. what are the factors underlying some cases and examples that could explain why certian models struggle with evidence retrieval and others with identity attribution?
5 comments
Response to Reviewer toJq (2/2)
Q2. Validating the distractors
Where the wrong options come from. We traced every wrong option to its source. 44% of the wrong options are something another person in the same video actually says or does, with a pointer checked against the transcript or a frame. Ruling such an option out requires knowing who said or did it, which is the identity resolution the benchmark tests.
Can they be eliminated without the video? No. Each question was tested without the video by readers from three model families, and options that could be guessed were replaced or the question was dropped. Given the options alone, GPT-5.5 scores 18.2% on the revised set (40.9% on the submitted set) and Gemini-3.1-Pro 23.4%; given the question and options, Gemini-3.1-Pro scores 27.8% (was 59.6%) and GPT-5.5 23.3% (was 58.9%); chance is 25%. With the video, the same two models score 84.4% and 80.4%. One surface regularity remains: a word-overlap rule that picks the option worded least like the other three is right in 38% of questions. It does not explain the models' scores, as the options-only results show; we report it in the paper and will reword the affected answers for the camera-ready. In the target-swapped question pairs (reply to Reviewer KaaD), the options are identical within a pair, so eliminating options cannot help at all.
How they are validated. Every changed option was checked against the video and the annotations, so that exactly one option is correct for the person asked about.
Q3. Open vs. closed models
We agree that openness itself is not a scientific variable, and the paper does not use it to explain results. We refer to the strongest systems (Gemini-4-Argon, Gemini-3.1-Pro, GPT-5.5) as frontier models, a label for the best-performing systems rather than for how they are licensed; the results table groups systems by access and input type only for readability. The differences between models are explained by measurable factors, which the results on the revised set show directly:
- Comparing voices is hard for every model; comparing faces is not. Given the two moments directly, every model tells two faces apart almost perfectly (97–99%) but reaches only 69–75% on two voices.
- Finding the moments is what limits some models in the same-person test. Qwen3-Omni-Instruct compares two faces at 96.8% when given the moments but at chance (50.2%) when it has to find them in the whole video. For the multiple-choice questions, telling a model where to look in the unchanged video changes accuracy by −3.0 to +4.3 points, never significantly.
- How much of the input reaches the model. MiniCPM-o's official inference path keeps at most 64 one-second frame-and-audio blocks, so on a whole video it hears only 8–17% of the track; it scores 9.7% on voice questions against 56.1% on face questions. GPT-Audio, which hears the audio but sees nothing, scores 33.7%.
- How speech is received. Gemma-4-31B, which reads a transcript instead of hearing audio, reaches 72.0%, on par with Qwen3-Omni-Instruct with native audio (69.2%).
- What the mistakes are. For Gemini-3.1-Pro, Gemini-4-Argon, GPT-5.5, and Qwen3-Omni-Instruct, 43–46% of the mistakes pick something that another person in the video really said or did, that is, the right content attributed to the wrong person; 10–15% pick the right person at the wrong moment.
Dear Reviewer toJq,
As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We would greatly appreciate any feedback, and are happy to clarify anything further before the deadline. If our response resolves your concerns, we would be grateful if you would consider updating your score.
Thank you for your time.
Best,
LongCoref-AV Authors
Response to Reviewer toJq (1/2)
We thank the reviewer for the review and the questions. We rebuilt the question set so that it cannot be answered without the video, reran every evaluation on it, and ran new experiments for the reviewer's questions. The revised set has 572 questions from 450 videos, and the revised paper reports its results on it. Please see the updated PDF, where all changes from the submission are highlighted.
W1. Is identity coreference more than diarization + re-identification + audio-visual association?
Our goal is to measure whether multimodal LLMs can perform identity coreference as one unified task, end to end, from raw video and audio. A question names a person only by a description and asks what that person said or did elsewhere in the video. Answering requires three abilities:
Diarization, re-identification, and audio-visual association each contribute to these abilities, but none of them is the task, and none of them connects its output to the person the question describes. We tested this directly: giving GPT-5.5 what such a pipeline produces (a speaker-attributed transcript and a face-track timeline, as text, in addition to its frames) raises accuracy only from 80.4% to 85.0%, and no model given this text clearly exceeds its own video-and-audio score. The tools supply anonymous clusters and tracks; the link from the described person to a face and from that face to a voice is still left to the model.
The experiments split the task into steps to diagnose where models fail, not to define it. The task definition, with the three abilities above, is now stated at the start of the introduction.
Q1. Cues that fit several people, and the effect of the number of people
Multiple correct answers. Uniqueness is enforced at two stages. When the attribute inventory is built, each visual cue carries a flag for whether it fits exactly one person in the video; the generator writes cues from this inventory, and cues include disambiguating detail where needed, as in "the man who had been seated on the viewer's left". Human reviewers then checked correctness and uniqueness for every submitted question, and a question whose cue fit more than one person, or whose answer was not unique, was repaired or rejected. The revised set keeps these human-verified target people and evidence moments. Rewritten questions were checked again against the video.
Number of people. Accuracy by the number of audible speakers in the video (93, 157, 100, and 222 questions):
The task gets harder with more people: from two speakers to five or more, every model loses 6–11 points. The decline is steady for Gemini-4-Argon and not monotonic for Gemini-3.1-Pro, which is lowest at three speakers.