- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PERSISTQA studies whether video question-answering models connect observations about the same individual across time. It contains 3,299 eight-option questions from 409 videos and evaluates 27 models. The paper complements overall accuracy with identity-pair judgments, fixed-budget evidence insertion and removal, restricted-input experiments, and exploratory frame selection. The results show low QA accuracy, substantial sensitivity to evidence availability, and a gap between frozen-feature matching and explicit identity judgments, while acknowledging that these diagnostics do not fully isolate the cause of answering errors.
Critical strengths
-
The paper asks a useful diagnostic question rather than treating low aggregate accuracy as a sufficient explanation. Holding the model and eight-frame budget fixed while replacing one frame is especially informative. Table 3 reports insertion gains of 5.00 points for InternVL3-14B and 9.67 for Ovis2.5-9B, with both remaining significant after Holm correction. Video-bootstrap intervals and paired tests are useful safeguards, and the paper distinguishes these supported effects from nonsignificant comparisons.
-
The discussion explicitly limits several tempting causal interpretations. It notes that CLIP matching can exploit clothing or backgrounds, that answer windows may omit referent context, and that restricting inputs changes several factors together. The combination of correspondence probes and temporal interventions is a useful starting point for investigating why long-video QA fails.
Critical weaknesses
-
The available evidence does not sufficiently establish the validity of identity-grounded questions and their answer labels. Section 3 describes model-assisted construction, an adversarial language-only filter involving model families also evaluated, and manual verification of candidate-answer windows. The paper itself notes that challenge labels do not verify the necessary reasoning for each answer. Verification of a window is not equivalent to independent confirmation that the described person is uniquely identified and that exactly one answer is correct. The 60% human no-look-back result is also reported without the sample size, annotator count, or uncertainty in the supplied main text. These details matter because ambiguity, filtering bias, and memory limitations could contribute to the observed human-model gap. The referenced appendices may address some of this, but they are absent from the supplied 15-page PDF and must be checked before treating the concern as unresolved.
-
The diagnostics do not yet establish identity association as the specific cause of the remaining QA errors. The CLIP-versus-VLM comparison uses isolated crop pairs and different decision procedures, while the restricted-input experiment may remove the observations needed to identify the referent. The paper acknowledges both issues. Moreover, the referent-plus-answer controls summarized on page 7 do not exceed a repeated answer-image control. Consequently, the results demonstrate correspondence and evidence-access difficulties, but do not quantitatively separate identity association from perception, lost context, and answer selection. That limits the strength of the benchmark's central diagnostic contribution.
Questions for authors
-
Clarification: What independent human checks verify referent uniqueness, cross-time identity, and answer correctness? Please provide the human-baseline sample size, number of raters, uncertainty, and the location of the supporting supplementary results.
-
Clarification: Which existing controlled result isolates an identity-association error after both referent and answer evidence are verified as sufficient? How should the restricted-input error rate be interpreted given the referent-plus-answer and repeated-answer-image controls?
4 comments
Response (2/2)
Summary. One annotator verified every generated question, a different annotator conducted the audit, and the replay study's 93.1% accuracy has a 95% participant-level interval of 87.7–97.7%. The person-crop control comes closest to the requested isolation, and a verified image of the person adds at most one correct answer.
Q1. Independent checks, and human-baseline details and location
Which independent human checks verify referent uniqueness, cross-time identity, and answer correctness? Please give the human-baseline sample size, number of raters, uncertainty, and the location of the supporting supplementary results.
After generation, one annotator manually verified every question. A different annotator conducted the additional audit, which covered 100 questions from 20 videos and asked three separate questions. Does the description identify one person? Is that the same person at the answer moment? Does the video support one answer? The checks pass on 99, 93 and 82 questions, and 77 pass all three. Rechecking the auditor's answer choices against the videos found four designated answers that the video does not support, all from one option-cleanup step. The reference controls use the corrected keys, and correcting them changes each model's benchmark accuracy by at most 0.12 points. This audit is separate from the annotation of answer windows.
The replay study involved 10 participants, each answering a different set of 13 questions, for 130 distinct questions and responses, and reached 93.1% accuracy. Individual participants scored between 76.9% and 100%, and resampling participants gives a 95% bootstrap interval of 87.7–97.7%. Video replay and audio were available, but subtitles were not. The replay study and the audit use different questions and measure different properties, namely answering accuracy and the three checks above.
The manual checks are described in Appendix A.2, p. 17, and the human-study protocol with its participant-level interval in Appendix B.5, p. 21.
Q2. A control that isolates association once evidence is verified
Which existing controlled result isolates an identity-association error after both referent and answer evidence are verified as sufficient? How should the restricted-input error rate be interpreted given the referent-plus-answer and repeated-answer-image controls?
The new person-crop control comes closest. It holds the selected answer frames fixed and changes only the person image. An author checked and cropped the target person and a different person for 10 questions from 10 videos, and eight of these questions also pass the checks that the selected frames provide sufficient evidence and support one answer. The question set and these checks were fixed before the model runs.
The first three result columns change only the reference image, to the target-person crop, the different-person crop or a blank gray image. The last column keeps the target crop but replaces the answer frames with eight frames spaced evenly across the whole video. Every input has eight video frames and one reference image.
Correct answers out of each question set when the person image or the video frames change
| Model | Question set | Target person | Different person | Blank image | Whole video + target |
|---|---|---|---|---|---|
| InternVL3-14B | All 10 questions | 3/10 | 3/10 | 2/10 | 1/10 |
| Qwen2.5-VL-7B | All 10 questions | 0/10 | 1/10 | 1/10 | 3/10 |
| Gemma3-12B | All 10 questions | 5/10 | 4/10 | 5/10 | 2/10 |
| InternVL3-14B | 8 passing checks | 3/8 | 2/8 | 2/8 | 1/8 |
| Qwen2.5-VL-7B | 8 passing checks | 0/8 | 1/8 | 1/8 | 3/8 |
| Gemma3-12B | 8 passing checks | 5/8 | 4/8 | 5/8 | 2/8 |
On the eight questions that pass both checks, the target crop adds one correct answer over the different-person crop for InternVL and Gemma and loses one for Qwen. A verified image of the person therefore adds at most one correct answer, so the remaining errors are not explained by identity association alone.
The restricted-input test reached up to 31.9% accuracy. Restricting the input can also remove earlier frames needed to identify the person. The repeated answer-image control shows the candidate answer frame eight times, while the referent-plus-answer input shows it once among full scenes, so the two inputs differ in repetition, context and distraction as well as in the referent cue. We therefore read the restricted-input score as performance under that viewing setup, with errors that can involve identifying the person, recognizing the event or selecting the answer. The crop experiment and its settings are in Appendix J.4, pp. 41–43.
Please let us know if you have any further questions. We will be happy to answer.
Thank you for your response. I will maintain my score.
We sincerely thank the reviewer for taking the time to read our response. We have addressed each point raised in the review, including the label checks and the 100-question audit, the replay-enabled human study, the appendices in the supplementary PDF, and the new person-crop and full-frame reference controls. We would be grateful if the reviewer could indicate which concerns remain, or which results are not yet convincing, so that we may address them and convince the reviewer of the merits of our work.
Response (1/2)
Summary. Every generated question was manually verified, a separate 100-question audit checks referent uniqueness, cross-time identity and answer uniqueness, and the supplementary PDF contains all appendices. A replay-enabled human study replaces the 60% pilot. We describe each diagnostic by what it measures, and the new person-crop control comes closest to the isolation the reviewer asks for.
We thank the reviewer for the detailed feedback. We clarify the manual checks, explain what each experiment measures, and give the locations of the supporting results.
W1. Validity of labels, human-baseline uncertainty, and appendices
We agree that marking when an answer appears does not check the whole question. All generated questions were manually verified, and the separate 100-question audit records whether the person is uniquely identified, whether the same person appears at the answer moment, and whether the video supports one answer (Q1 below and Appendix A.2, p. 17). The 60% figure came from an earlier no-look-back pilot under a separate viewing condition. The revised paper replaces it with the replay-enabled human study detailed in Q1 (Sec. 4 and Appendix B.5).
The supplementary PDF contains all appendices, with page numbers that continue from the main paper. The identity-pair probe is in Appendix D.1 (pp. 22–23), the frame-replacement tests in D.2 (pp. 23–25), the filtering experiment in J.2 (pp. 39–40) and the new reference controls in J.3–J.4 (pp. 40–43).
W2. Identity association as the specific cause of the errors
We agree, and the revision describes each diagnostic by what it measures rather than treating every incorrect answer as a failure to connect a person across time.
Identity-pair probe. On 2,848 balanced same-person and different-person pairs, frozen CLIP features reach 85.2% accuracy, while explicit VLM judgments reach 50.6–62.6%. Visual correspondence can therefore be recovered from frozen features even where explicit judgments are unreliable. We report the gap as a property of the two decision procedures, not as a count of QA errors.
Frame replacement. For each of 300 questions, we replaced one of eight frames with a frame from the candidate-answer window, keeping the question, options, model and other seven frames fixed.
Effect of replacing one of eight frames, 300 questions per model
One answer frame raises accuracy for all three models, and the gains for InternVL3-14B and Ovis2.5-9B remain significant after Holm correction. The added frame can help the model recognize the person, see the action or distinguish the options, so we report it as the effect of the added observation. Q2 examines a change that varies only the person image. The statistical details are in Table 3 and Appendix D.2, pp. 23–25.