- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This review represents my own independent reading, technical critique, and final rating of Submission 29074. My evaluation balances the value of the same-video multi-persona diagnostic design and transparent failure reporting against concerns regarding the validity of synthetic references, weak cross-model title agreement, restrictive completion filtering, and baseline generation failures, issues only partially resolved by the rebuttal's limited human study.
An AI language model was used strictly for minor language polishing and grammar refinement of my drafted points. All substantive arguments and the final decision are entirely my own, and I take full responsibility for this review.
Critical strengths
-
The same-video, different-persona comparisons address an important question that ordinary reference agreement can obscure.
-
The diagnostics include repeated generations, two annotation models, source-level bootstrap intervals, and explicit controls for empty selections and missing outputs.
-
The paper is unusually transparent about synthetic references, weak cross-model agreement, baseline training overlap, and generation failures.
Critical weaknesses
-
My main concern is whether the references represent useful personalization. Sec.6 acknowledges the absence of human validation. Highlight annotations use frame captions, while title annotations use only speech transcripts. Reproducible persona effects demonstrate annotation consistency, but do not establish factual grounding or relevance to a viewer. This limits the benchmark's intended use.
-
The title metrics provide limited evidence about meaning. In Table 2, cross-model persona contrast is only 0.003 in METEOR, compared with 0.083 and 0.216 within models. The authors appropriately restrict this conclusion to model-specific wording. Still, lexical overlap cannot distinguish different expressions of relevant content from different persona interpretations. The scoring needs independent support beyond wording agreement.
-
The effective title evaluation is small and selective. Table 15 reports only 16 videos for full-pair results and 14 for paired contrasts. Requiring valid titles from both annotators for every interval excludes otherwise usable pairs. Together with the earlier interest screen, this restricts coverage. Baseline title results using all available interval-level references can clarify sensitivity to the completion rule.
-
Output failures and prior exposure limit absolute model comparisons. Qwen3-VL fails on 147/252 persona-conditioned outputs, and the longer-output control still fails on 115/252 (Table 18). QD-DETR and CG-DETR previously saw 17/27 test source videos. The paired persona contrasts remain informative, but these issues complicate comparisons between models.
Questions for authors
As shown in weaknesses.
Thank you for the detailed review and for recognizing the same-video comparisons, the diagnostics, and our transparency about limitations. Changes related to your review are shown in orange, or in purple where they also answer other reviewers, and are tagged G4Xh in the margin of the revised PDF.
W1. Do the references represent useful personalization rather than consistency only?
We agree that consistency does not establish relevance, and added a study testing whether matched viewers prefer their own-persona references in a selected GPT-OSS subset. Participants were recruited through CloudResearch Connect and matched to personas using demographic information and an interest checklist. The 192 participants, 12 per persona, watched each video and ranked the original annotation and the references of the two personas, on clips chosen where the two persona references differ. Matched viewers ranked their own persona's reference above the other persona's in 61.5% [57.5, 65.3] of highlight and 57.1% [52.4, 62.0] of chapter comparisons (chance 50%). A test over trials gives p < .001 and p = .018, and the preference is stronger where our metrics separate the two references (Spearman ρ = .57 and .61). The study concerns relevance, not factual accuracy. Construction requires each title to cite supporting ASR utterances and selects highlights from captioned units, and Sec. 6 states that we do not audit references for factual accuracy.
Revised: Sec. 5.4, Table 5, App. G, Sec. 6.
W2. Is there support for the title evaluation beyond wording overlap?
Yes. Recomputing Table 2(a) with sentence embeddings (all-mpnet-base-v2, bge-large-en-v1.5) and rescaled BERTScore gives the same pattern, as the cross-model contrast stays positive but 5 to 27 times smaller than within each model (BERTScore .093 and .132 within, .005 across). Baseline title contrasts also remain near zero under these measures. Independently of any text metric, matched viewers preferred their own persona's titles in 57.1% of comparisons, more so where the METEOR distance between the two lists is larger (ρ = .61).
Revised: Sec. 4.4, App. H.5, Sec. 5.4.
W3. Is the title evaluation too small and selective?
Scoring every interval with an available reference raises the paired evaluation from 14 to 22 source videos (full results from 16 to 30), and target − other METEOR stays between +.001 and +.006 for all models and both references. Individual estimates shift, for example Mistral-Nemo against the GPT-OSS references from +.011 to +.005. The sign reversal of Mistral-Nemo against the Gemma references disappears under interval-level evaluation, so we removed that statement. Sec. 4.3 now states that the interest screen limits which persona contrasts the benchmark can test.
Revised: Sec. 4.3, Sec. 5.1, Sec. 5.3, App. H.3.
W4. Do output failures and training exposure complicate the comparisons?
The target-versus-other conclusions do not change. Grammar-constrained decoding removes all Qwen3-VL failures (155 of 280 outputs to none) with no reliable persona gain (target − other −.003 [−.025, .017] in base F1 and .000 [−.012, .012] in expanded F1), and the persona-free gaps of Qwen3-VL and Chapter-Llama shrink once failures are handled. Table 28 now reports absolute scores for both runs. For exposure, the 8 test videos excluded from QVHighlights training show weak contrasts for QD-DETR and CG-DETR (+.002 and +.006 base AP), so we applied the same protocol to all splits, on which no baseline is trained. On 56 excluded source videos, base AP target − other is +.031, +.030, +.022, and +.042 for CLIP, SigLIP2, QD-DETR, and CG-DETR, against +.008, +.013, +.025, and +.007 on the 90 included videos. Contrasts are thus not systematically larger on exposed videos, although QD-DETR's interval on excluded videos includes zero. The absolute target AP of QD-DETR and CG-DETR is similar on both groups (.219 and .230 excluded, .227 and .205 included). These videos belong to the QVHighlights validation set, so we call them excluded from supervised training rather than unseen.
Revised: Sec. 5.2, Sec. 5.3, App. H.1, App. H.4.
Presentation
We added a table that summarizes the tasks, references, conditions, contrasts, and metrics in one place, referenced from Sec. 5.1, and separated annotation inputs from baseline inputs there.
Revised: Table 14 in App. D, Sec. 5.1.
Summary
Key additions: a human preference study (Sec. 5.4, App. G), meaning-based title similarity (App. H.5), interval-level title evaluation (App. H.3), failure handling and exposure analyses (App. H.1, H.4), and a summary table (App. D). We thank the reviewer for the detailed weaknesses and hope the revision addresses the concerns. If it does, we would be grateful if you would consider updating your score.