acceptodds
Under review as a conference paper at ICLR 2027

PERSISTQA: DIAGNOSING IDENTITY ASSOCIATION IN LONG-VIDEO QUESTION ANSWERING

Abstract

Long videos often contain people who reappear across different scenes and moments. Answering a question about one person therefore requires more than finding a relevant event. A model must recognize the intended person across time and associate the correct observations with that person. However, long-video question answering is typically evaluated using aggregate answer accuracy, which does not reveal where this process fails. We introduce PersistQA, a diagnostic benchmark for studying identity grounding in long-video question answering. PersistQA contains 3,299 eight-option questions over 409 videos. Each question identifies a person through visual or relational cues and asks about the same individual at another moment. Across 27 models, the best open-weight and closed-source systems achieve only 27.2% and 29.3% accuracy, compared with 60% for a human no-look-back baseline. Our diagnostics reveal several sources of error. Frozen CLIP features distinguish same-person from different-person pairs with 85.2% accuracy, while VLMs explicitly judging identity reach only 50.6-62.6%. Eight uniformly sampled frames miss approximately 53% of candidate-answer windows, and replacing one frame with candidate evidence improves accuracy by 4.3-9.7 percentage points. Even when candidate evidence is directly provided, 61-78% of questions remain incorrect. These results highlight identity association, evidence access, and evidence use as distinct challenges for long-video question answering.

Primary area:
datasets and benchmarks
Keywords:
Video QA, Long Videos, Identity Grounding, VLMs, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Lpw7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PERSISTQA studies whether video question-answering models connect observations about the same individual across time. It contains 3,299 eight-option questions from 409 videos and evaluates 27 models. The paper complements overall accuracy with identity-pair judgments, fixed-budget evidence insertion and removal, restricted-input experiments, and exploratory frame selection. The results show low QA accuracy, substantial sensitivity to evidence availability, and a gap between frozen-feature matching and explicit identity judgments, while acknowledging that these diagnostics do not fully isolate the cause of answering errors.

Critical strengths
  1. The paper asks a useful diagnostic question rather than treating low aggregate accuracy as a sufficient explanation. Holding the model and eight-frame budget fixed while replacing one frame is especially informative. Table 3 reports insertion gains of 5.00 points for InternVL3-14B and 9.67 for Ovis2.5-9B, with both remaining significant after Holm correction. Video-bootstrap intervals and paired tests are useful safeguards, and the paper distinguishes these supported effects from nonsignificant comparisons.

  2. The discussion explicitly limits several tempting causal interpretations. It notes that CLIP matching can exploit clothing or backgrounds, that answer windows may omit referent context, and that restricting inputs changes several factors together. The combination of correspondence probes and temporal interventions is a useful starting point for investigating why long-video QA fails.

Critical weaknesses
  1. The available evidence does not sufficiently establish the validity of identity-grounded questions and their answer labels. Section 3 describes model-assisted construction, an adversarial language-only filter involving model families also evaluated, and manual verification of candidate-answer windows. The paper itself notes that challenge labels do not verify the necessary reasoning for each answer. Verification of a window is not equivalent to independent confirmation that the described person is uniquely identified and that exactly one answer is correct. The 60% human no-look-back result is also reported without the sample size, annotator count, or uncertainty in the supplied main text. These details matter because ambiguity, filtering bias, and memory limitations could contribute to the observed human-model gap. The referenced appendices may address some of this, but they are absent from the supplied 15-page PDF and must be checked before treating the concern as unresolved.

  2. The diagnostics do not yet establish identity association as the specific cause of the remaining QA errors. The CLIP-versus-VLM comparison uses isolated crop pairs and different decision procedures, while the restricted-input experiment may remove the observations needed to identify the referent. The paper acknowledges both issues. Moreover, the referent-plus-answer controls summarized on page 7 do not exceed a repeated answer-image control. Consequently, the results demonstrate correspondence and evidence-access difficulties, but do not quantitatively separate identity association from perception, lost context, and answer selection. That limits the strength of the benchmark's central diagnostic contribution.

Questions for authors
  1. Clarification: What independent human checks verify referent uniqueness, cross-time identity, and answer correctness? Please provide the human-baseline sample size, number of raters, uncertainty, and the location of the supporting supplementary results.

  2. Clarification: Which existing controlled result isolates an identity-association error after both referent and answer evidence are verified as sufficient? How should the restricted-input error rate be interpreted given the referent-plus-answer and repeated-answer-image controls?

4 comments
AuthorsOpenReview ↗

Response (1/2)

Summary. Every generated question was manually verified, a separate 100-question audit checks referent uniqueness, cross-time identity and answer uniqueness, and the supplementary PDF contains all appendices. A replay-enabled human study replaces the 60% pilot. We describe each diagnostic by what it measures, and the new person-crop control comes closest to the isolation the reviewer asks for.

We thank the reviewer for the detailed feedback. We clarify the manual checks, explain what each experiment measures, and give the locations of the supporting results.

W1. Validity of labels, human-baseline uncertainty, and appendices

Verifying a window is not the same as independently confirming that the described person is uniquely identified and that exactly one answer is correct. The 60% human no-look-back result is reported without a sample size, annotator count, or uncertainty, and the referenced appendices are absent from the supplied PDF.

We agree that marking when an answer appears does not check the whole question. All generated questions were manually verified, and the separate 100-question audit records whether the person is uniquely identified, whether the same person appears at the answer moment, and whether the video supports one answer (Q1 below and Appendix A.2, p. 17). The 60% figure came from an earlier no-look-back pilot under a separate viewing condition. The revised paper replaces it with the replay-enabled human study detailed in Q1 (Sec. 4 and Appendix B.5).

The supplementary PDF contains all appendices, with page numbers that continue from the main paper. The identity-pair probe is in Appendix D.1 (pp. 22–23), the frame-replacement tests in D.2 (pp. 23–25), the filtering experiment in J.2 (pp. 39–40) and the new reference controls in J.3–J.4 (pp. 40–43).

W2. Identity association as the specific cause of the errors

The CLIP-versus-VLM comparison uses isolated crop pairs and different decision procedures, and the restricted-input experiment may remove the observations needed to identify the referent. The referent-plus-answer control does not exceed a repeated answer-image control. The results therefore do not quantitatively separate identity association from perception, lost context, and answer selection.

We agree, and the revision describes each diagnostic by what it measures rather than treating every incorrect answer as a failure to connect a person across time.

Identity-pair probe. On 2,848 balanced same-person and different-person pairs, frozen CLIP features reach 85.2% accuracy, while explicit VLM judgments reach 50.6–62.6%. Visual correspondence can therefore be recovered from frozen features even where explicit judgments are unreliable. We report the gap as a property of the two decision procedures, not as a count of QA errors.

Frame replacement. For each of 300 questions, we replaced one of eight frames with a frame from the candidate-answer window, keeping the question, options, model and other seven frames fixed.

Effect of replacing one of eight frames, 300 questions per model

ModelNet additional correct answersAccuracy increase (percentage points)
InternVL3-14B15 out of 3005.00
Ovis2.5-9B29 out of 3009.67
Qwen2.5-VL-7B13 out of 3004.33

One answer frame raises accuracy for all three models, and the gains for InternVL3-14B and Ovis2.5-9B remain significant after Holm correction. The added frame can help the model recognize the person, see the action or distinguish the options, so we report it as the effect of the added observation. Q2 examines a change that varies only the person image. The statistical details are in Table 3 and Appendix D.2, pp. 23–25.

AuthorsOpenReview ↗

Response (2/2)

Summary. One annotator verified every generated question, a different annotator conducted the audit, and the replay study's 93.1% accuracy has a 95% participant-level interval of 87.7–97.7%. The person-crop control comes closest to the requested isolation, and a verified image of the person adds at most one correct answer.

Q1. Independent checks, and human-baseline details and location

Which independent human checks verify referent uniqueness, cross-time identity, and answer correctness? Please give the human-baseline sample size, number of raters, uncertainty, and the location of the supporting supplementary results.

After generation, one annotator manually verified every question. A different annotator conducted the additional audit, which covered 100 questions from 20 videos and asked three separate questions. Does the description identify one person? Is that the same person at the answer moment? Does the video support one answer? The checks pass on 99, 93 and 82 questions, and 77 pass all three. Rechecking the auditor's answer choices against the videos found four designated answers that the video does not support, all from one option-cleanup step. The reference controls use the corrected keys, and correcting them changes each model's benchmark accuracy by at most 0.12 points. This audit is separate from the annotation of answer windows.

The replay study involved 10 participants, each answering a different set of 13 questions, for 130 distinct questions and responses, and reached 93.1% accuracy. Individual participants scored between 76.9% and 100%, and resampling participants gives a 95% bootstrap interval of 87.7–97.7%. Video replay and audio were available, but subtitles were not. The replay study and the audit use different questions and measure different properties, namely answering accuracy and the three checks above.

The manual checks are described in Appendix A.2, p. 17, and the human-study protocol with its participant-level interval in Appendix B.5, p. 21.

Q2. A control that isolates association once evidence is verified

Which existing controlled result isolates an identity-association error after both referent and answer evidence are verified as sufficient? How should the restricted-input error rate be interpreted given the referent-plus-answer and repeated-answer-image controls?

The new person-crop control comes closest. It holds the selected answer frames fixed and changes only the person image. An author checked and cropped the target person and a different person for 10 questions from 10 videos, and eight of these questions also pass the checks that the selected frames provide sufficient evidence and support one answer. The question set and these checks were fixed before the model runs.

The first three result columns change only the reference image, to the target-person crop, the different-person crop or a blank gray image. The last column keeps the target crop but replaces the answer frames with eight frames spaced evenly across the whole video. Every input has eight video frames and one reference image.

Correct answers out of each question set when the person image or the video frames change

ModelQuestion setTarget personDifferent personBlank imageWhole video + target
InternVL3-14BAll 10 questions3/103/102/101/10
Qwen2.5-VL-7BAll 10 questions0/101/101/103/10
Gemma3-12BAll 10 questions5/104/105/102/10
InternVL3-14B8 passing checks3/82/82/81/8
Qwen2.5-VL-7B8 passing checks0/81/81/83/8
Gemma3-12B8 passing checks5/84/85/82/8

On the eight questions that pass both checks, the target crop adds one correct answer over the different-person crop for InternVL and Gemma and loses one for Qwen. A verified image of the person therefore adds at most one correct answer, so the remaining errors are not explained by identity association alone.

The restricted-input test reached up to 31.9% accuracy. Restricting the input can also remove earlier frames needed to identify the person. The repeated answer-image control shows the candidate answer frame eight times, while the referent-plus-answer input shows it once among full scenes, so the two inputs differ in repetition, context and distraction as well as in the referent cue. We therefore read the restricted-input score as performance under that viewing setup, with errors that can involve identifying the person, recognizing the event or selecting the answer. The crop experiment and its settings are in Appendix J.4, pp. 41–43.

Please let us know if you have any further questions. We will be happy to answer.

Thank you for your response. I will maintain my score.

We sincerely thank the reviewer for taking the time to read our response. We have addressed each point raised in the review, including the label checks and the 100-question audit, the replay-enabled human study, the appendices in the supplementary PDF, and the new person-crop and full-frame reference controls. We would be grateful if the reviewer could indicate which concerns remain, or which results are not yet convincing, so that we may address them and convince the reviewer of the merits of our work.

Reviewer_HUhxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PersistQA introduces a new benchmark for studying whether VideoLLMs understand grounding identities in the given VideoQA. The authors extracted relevant knowledge from the video and tried to generate questions that link identities in the video. They also evaluated 27 models of which the best open-weight only achieve 27.2% with the best closed-source system of 29.3%, revealing how difficult the benchmark/task is for VideoLLMs today.

Critical strengths
  1. Interesting as well as useful benchmark. I do agree that videollms lack observing or linking specific users in the video at different times, which do make this task very interesting. Also, there aren't much benchmarks that do tackle this idea.

  2. Interseting analysis points: measuring how each evidence plays a role in answering the questions, as well as like single-frame insertion or removal experiments. Also, the authors evalauted under various family models with different configs (I think the main experiment was with like 8 frames, and then in the supplement they did add like effect of the number of frames (which definitly was something I was curious about while reading).

Critical weaknesses
  1. The benchmark’s central identity linkage is insufficiently specified. I am very concerned about this point, so I think in the paper the authors do specify they conducted manual verifications and how they gathered relevant information to create the benchmark. But the real question is how did they make sure the referent-alone moment is established to be the same inidivual at the answer moment? like the procedures on how they synthesized the questions, how the ground truth was established is not clear. This is a major part of the intent of the benchmark which this paper itself lacks an explanation.

  2. The failure cases in this paper, don't specifically decouple the failures come from the identity associations : I understand that the experiments conducted such as the identity correspondence and explicit judgements anlysis and the restricted inputs and internal allocation tries the diagnose the issues. But isn't there a possibility that restriction may remove the context needed to establish the referent? I think an better option is to conduct an oracle-reference condition (like few samples where humans verify the target crop).

  3. As this is a benchmark paper, I do want to know more specifics about the human baseline protocols: so, what was the annotator sample size? how many annotators conducted this experiment, how and what way did they observe the video?

Questions for authors

Please address the questions raised in the weaknesses.

6 comments
AuthorsOpenReview ↗

Response (1/2)

Summary. Every generated question was manually verified, and a separate 100-question audit checks the identity link between the two moments directly. Following the reviewer's suggestion, we added full-frame and human-checked person-crop reference controls with the answer frames held fixed. An image of the described person adds at most one correct answer per model, so the remaining errors also involve locating and using the answer evidence, which PersistQA's frame-replacement test measures.

We thank the reviewer for the careful review and the useful suggestion to provide a person image. We clarify how the questions were checked and report the completed experiments below.

W1. Specification of the identity linkage and its ground truth

The benchmark's central identity linkage is insufficiently specified; how was the referent-alone moment established to be the same individual as the answer moment, and how were the questions synthesized and the ground truth set? This is a major part of the benchmark's intent.

Gemini 3.1 Pro drafted each question, producing a referent description, the queried property, the designated answer and a textual temporal anchor. Each question describes a person at one moment and asks about that same person at another, and the prompt instructs the model to follow appearance, actions and interactions over time. A separate prompt generates seven distractors. All generated questions were then manually verified.

A separate audit of 100 questions from 20 videos checks the identity link directly. For each question it records whether the description identifies one person, whether the person at the answer moment is the same individual, and whether the video supports a single answer. The description identifies one person in 99 questions, the same individual appears at the answer moment in 93, and the video supports a single answer in 82. The 77 questions that pass all three checks form the sample of the full-frame reference control below (Appendix A.2, p. 17).

The auditor also selected an answer for each of the 77 questions. Rechecking the disagreements against the videos found four questions whose designated answer the video does not support, all from one option-cleanup step. The results below use the corrected keys, and correcting them changes each model's benchmark accuracy by at most 0.12 points.

W2. Decoupling identity-association failures and an oracle-reference control

The failure analyses do not decouple errors due to identity association, and restricting the input may remove the context needed to establish the referent. A better option is an oracle-reference condition, for example a few samples where humans verify the target crop.

Following this suggestion, we kept the question and video frames fixed and changed only an extra image shown to the model, which we call the reference image. The comparison measures whether an image of the person named in the question helps the model answer.

Full-frame references. Each of the 77 audited questions from W1 receives eight video frames and one reference image. The frames are either spaced evenly across the whole video (Whole video) or sampled from the annotated answer window (Answer window). The reference is the full frame from the moment the question uses to identify the target person (Target reference), a frame from another question (Other reference) or a plain gray image (Blank image).

QA accuracy (%) on the 77 audited questions with full-frame reference images

ModelEight video framesTarget referenceOther referenceBlank image
InternVL3-14BWhole video24.720.823.4
InternVL3-14BAnswer window28.627.328.6
Qwen2.5-VL-7BWhole video26.027.322.1
Qwen2.5-VL-7BAnswer window32.531.227.3
Gemma3-12BWhole video22.114.316.9
Gemma3-12BAnswer window32.533.833.8

With the answer-window frames fixed, the target reference adds one correct answer over the other reference for InternVL and Qwen and one fewer for Gemma. Moving the frames from the whole video to the answer window raises accuracy with the target reference for all three models, by 3.9, 6.5 and 10.4 points, although no contrast on 77 questions is significant after Holm correction. A full frame can include several people and background details, so we then tested person crops.

AuthorsOpenReview ↗

Response (2/2)

Summary. Human-checked person crops add at most one correct answer per model once the answer frames are fixed, so the remaining errors also involve locating and using the answer evidence. People with video replay answer 93.1% of the sampled questions.

Person crops. For 10 questions from 10 videos, an author marked the target person and a different person with a box in the reference frames. Eight of these questions also pass the checks that the selected answer frames contain sufficient evidence and support a single answer. Of the other two, one lacks sufficient evidence and one permits more than one answer. With the answer frames fixed, only the reference changes, to the target-person crop, the different-person crop or a blank image.

Correct answers with each reference, out of all 10 questions and out of the 8 that pass both checks

ModelQuestion setTarget personDifferent personBlank image
InternVL3-14BAll 10 questions3/103/102/10
Qwen2.5-VL-7BAll 10 questions0/101/101/10
Gemma3-12BAll 10 questions5/104/105/10
InternVL3-14B8 passing checks3/82/82/8
Qwen2.5-VL-7B8 passing checks0/81/81/8
Gemma3-12B8 passing checks5/84/85/8

On the eight questions that pass both checks, the target crop changes one incorrect answer to correct for InternVL and Gemma, and one correct answer to incorrect for Qwen.

What the controls show. Once the answer frames are fixed, a verified image of the described person adds at most one correct answer per model. The remaining errors therefore also involve locating and using the answer evidence, which PersistQA's frame-replacement test measures directly. In that test, one answer frame among eight raises accuracy by up to 9.7 points (Sec. 5). The full protocols and results are in Appendices J.3–J.4, pp. 40–43.

Q. Human-baseline protocol

What was the annotator sample size, how many annotators took part, and in what way did they observe the video?

Ten participants each answered a different set of 13 questions, giving 130 distinct questions and responses. They could watch and replay the video with audio, without subtitles, and reached 93.1% accuracy (95% participant-level bootstrap interval 87.7–97.7%), which shows that people can answer the sampled questions. Participants had broader temporal access than the eight-frame models, so we report this result as answerability rather than as a matched-input comparison between humans and models (Appendix B.5, p. 21).

Please let us know if you have any further questions. We will be happy to answer.

Thank you for the revision, the response did address some of my concerns, especially coming to the scope of the human study.

However, two issues remain unclear to me. First, the new 100-question audit reports that only 77 items pass all three checks (82 for the single-answer check), yet it looks like the benchmark keys remain unchanged..? To me, it is not clear how the items that did not pass are handled, or what this implies for the full set of 3,299 questions.. Second, it looks like the person-crop study uses the benchmark keys “with four audit corrections,” which suggests that some keys are known to be incorrect. The scope of these corrections, and why they are not applied to the released benchmark, should be clarified. As I personally think benchmark reliability is central, I will maintain my score.

As a minor note, just a suggestion that it would have been helpful if the revised parts of the manuscript had been highlighted like in a different color. This would have made the changes much easier to identify.

Reviewer_gj86OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PersistQA, a diagnostic benchmark for "identity grounding" in long-video question answering which comprises 3,299 eight-option questions over 409 videos sourced from CinePile, MovieChat-1k, and LVU. It also proposes the model-assisted question construction and manually verified candidate-answer temporal windows. The authors evaluate massive models on the benchmark, where all of them outputs a low score. The paper conducted three diagnostic analyses complement the leaderboard and concludes that identity association, evidence access, and evidence use are distinct, jointly unsolved challenges.

Critical strengths
  1. Well-motivated diagnostic decomposition. The claim that aggregating long-video QA accuracy conflates missing the evidence, misattributing it to the wrong person, and failing to use supplied evidence is clear and each failure mode is probed with a dedicated analysis.

  2. Breadth of evaluation. Extensive models spanning open-weight models, long-video architectures, and closed-source systems are evaluated under a common protocol with per-capability breakdowns), plus a shared-failure analysis.

  3. Controlled design features in the benchmark construction. The design of language-only shortcut filtering, answer-position redistribution, and withheld verified evidence windows address known failure modes of VQA-style benchmarks and is reasonable.

Critical weaknesses
  1. The language-only filter is adversarial against the evaluated model families, confounding "difficulty." Items are removed when at least five of twelve text-only committee models select the designated answer (or at least eight select the same wrong option), and the committee members "come from model families that are also evaluated". The first criterion selects, by construction, for questions those very families get wrong, deflating their reported scores relative to an unfiltered distribution. The paper acknowledges that post-filter committee accuracy is uninformative, but does not quantify how much of the observed near-chance performance is manufactured by this adversarial filtering versus intrinsic task difficulty.

  2. Novelty relative to recent entity-centric benchmarks is incremental. NarrativeTrack (entity continuity, 1,006 questions), Video-MME's cross-appearance person tracking, Video-MME-v2's entity persistence, and LongVideoBench's referring reasoning (Section 2, Table 1) all target overlapping abilities. The genuinely new elements are the standalone identity-pair probe and the frame-intervention protocol; the QA task itself is close to prior referring/tracking QA.

Questions for authors

The authors are expected to answer the proposed concerns in the weakness section.

3 comments

Summary. A matched study of 300 retained and removed question pairs shows that the filter removes questions answerable from text alone and retains questions on which video helps more. The strongest models on PersistQA are closed-source systems that the filter never consulted, and they stay below 30% accuracy. Among the benchmarks in Table 1, PersistQA is the only one that reports identity-pair probes, a referring design and fixed-budget evidence interventions together.

We thank the reviewer for the helpful review. We have added a direct test of the filtering step and clarified what PersistQA adds to earlier benchmarks.

W1. Language-only filter and manufactured difficulty

An item is removed when at least five of twelve text-only committee models select the designated answer, or at least eight agree on one incorrect option, and the committee members come from families that are also evaluated. The first rule selects, by construction, for questions those families get wrong, which may deflate their scores; the paper does not quantify how much of the near-chance performance is manufactured by this filtering versus intrinsic difficulty.

The filter debiases the benchmark by removing questions that models can answer without watching the video. It selects on answers given without video and never uses video accuracy.

To measure its effect on video accuracy, we sampled 300 videos and took one removed and one retained question of the same capability category from each. Each model answered all 600 questions twice, from the question and options alone (No video) and with eight frames from the source video (With video). Both questions in a pair receive exactly the same eight frames.

Accuracy on removed and retained questions (%)

ModelRemoved, No videoRemoved, With videoRetained, No videoRetained, With video
InternVL3-14B54.354.318.732.0
Qwen2.5-VL-7B51.748.713.324.0
Gemma3-12B47.339.718.023.3
Chance (random guess)12.512.512.512.5

Removed questions are answerable from text alone. All three models answer them far above the 12.5% chance level without video, and adding video does not raise their accuracy.

Video matters more on retained questions. The gain from video is 13.3, 13.7 and 13.0 points larger on retained than on removed questions for InternVL, Qwen and Gemma, and all three 95% paired bootstrap intervals lie above zero.

Retained questions are harder for these models. With video, their accuracy is 16–25 points lower than on removed questions, which quantifies the selection effect that accompanies debiasing (Appendix J.2, pp. 39–40).

The low scores extend beyond the filter's model families. The committee consists of InternVL3, Qwen2.5-VL, Qwen3-VL, Gemma3, Ovis2.5 and Video-LLaVA models. The strongest models on PersistQA are closed-source systems outside these families, led by GPT-5.6 Terra at 29.3% (Table 2). Four further models outside the committee score 12.4–14.2% from text alone on the questions the filter retained, close to the 12.5% chance level (Appendix A.1).

W2. Novelty relative to recent entity-centric benchmarks

NarrativeTrack (entity continuity, 1,006 questions), Video-MME's cross-appearance person tracking, Video-MME-v2's entity persistence, and LongVideoBench's referring reasoning target overlapping abilities. The genuinely new elements are the standalone identity-pair probe and the frame-intervention protocol; the QA task itself is close to prior referring/tracking QA.

We agree that asking about the same person across a video overlaps with NarrativeTrack, LongVideoBench, Video-MME and Video-MME-v2, and the revised Related Work states this. PersistQA's contribution is the set of diagnostics built around these questions. Among the benchmarks in Table 1, it is the only one that reports identity-pair probes, an explicit referring design and fixed-budget evidence interventions together.

The identity-pair probe asks whether two crops show the same individual. The candidate-answer windows mark where the answer evidence appears in the video. The frame-replacement test changes one frame while the question, options, model and frame budget stay fixed. Replacing one of eight frames with a frame from the answer window raises InternVL3-14B and Ovis2.5-9B accuracy by 5.00 and 9.67 percentage points, and both gains remain significant after Holm correction. These diagnostics show which inputs change a model's answer, which the final QA score alone cannot reveal.

Please let us know if you have any further questions. We will be happy to answer.

I sincerly thank the authors for their efforts in addressing the concerns provided. I have read all reviews and rebuttals, including the discussions with other reviewers. Based on the provided experiments of the authors, I believe that the concerns on data constuction and the novelty has been addressed. Therefore, I prefer keep my positive initial rating.

We sincerely thank the reviewer for reading all reviews and discussions, and for confirming that our experiments address the concerns about data construction and novelty. We greatly appreciate the reviewer's positive assessment and constructive feedback, which have strengthened the paper.

Comments on the paper

1 comment

Summary. The revision adds a matched test of the language-only filter, documents how each question and its identity link were checked, and completes the person-reference controls the reviewers suggested. Each individual response reports these results and what they show about answering questions about a person across time.

We thank the reviewers for their helpful feedback. The revision makes four additions, each detailed in the individual responses.

  • Matched filtering study (Appendix J.2). Pairs of retained and removed questions with identical eight-frame inputs show that the filter removes questions answerable from text alone and retains questions on which video helps more.
  • Recorded validity checks (Sec. 3, Appendix A.2). All generated questions were manually verified, and a separate 100-question audit checks referent uniqueness, cross-time identity and answer uniqueness.
  • Reference controls (Sec. 5, Appendices J.3–J.4). Full-frame and human-checked person-crop references test whether an image of the described person helps a model answer when the answer frames stay fixed.
  • Replay-enabled human study (Sec. 4, Appendix B.5). Ten participants answered 130 distinct questions with video replay and reached 93.1% accuracy, which shows that people can answer the questions.

Together, these results show that PersistQA's questions depend on the video, that people can answer them, and which inputs change a model's answer. In our controls, one answer frame among eight raises accuracy by up to 9.7 points, while an image of the described person adds at most one correct answer per model.

Please let us know if you have any further questions. We will be happy to answer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.