acceptodds
Under review as a conference paper at ICLR 2027

One Video, Many Viewers: A Benchmark for Persona-Conditioned Video Highlights and Chapter Titles

Abstract

Highlights and chapters help viewers navigate long videos, yet they are usually produced once and shared by all viewers. Which moments matter and how each part should be described can differ between viewers, but existing evaluations rarely test whether these outputs fit a particular viewer. We study highlight selection and chapter titling conditioned on a viewer persona, a short description of a viewer's background and interests. Evaluating this setting requires annotations for many personas on the same video. Large language models make such annotations easy to produce, but differences between annotations may reflect run-to-run variation rather than the persona. We introduce One Video, Many Viewers, a benchmark with synthetic annotations for 259 source videos across highlight selection and chapter titling, built with 24 census-grounded synthetic personas. Highlight annotations are guided by a video-specific perspective derived from the persona, and chapter titles are generated for fixed intervals. Repeated-generation diagnostics show that persona-associated differences recur within an annotation model, whereas persona-specific title contrasts are much smaller across models than within each model. Evaluating existing models without benchmark-specific training, we find small but consistent gains over another persona in base AP for highlight models that score every unit. In contrast, we find no consistent gains for models that select units or for chapter titling. These results show that agreement with persona-conditioned references should be read alongside comparisons with other-persona and persona-free conditions. We release the annotations, personas, and perspectives for viewer-specific evaluation.

Primary area:
datasets and benchmarks
Keywords:
video highlight detection, video chaptering, personalization, persona conditioning, benchmark, synthetic annotations, LLM-generated annotations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G4XhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This review represents my own independent reading, technical critique, and final rating of Submission 29074. My evaluation balances the value of the same-video multi-persona diagnostic design and transparent failure reporting against concerns regarding the validity of synthetic references, weak cross-model title agreement, restrictive completion filtering, and baseline generation failures, issues only partially resolved by the rebuttal's limited human study.

An AI language model was used strictly for minor language polishing and grammar refinement of my drafted points. All substantive arguments and the final decision are entirely my own, and I take full responsibility for this review.

Critical strengths
  1. The same-video, different-persona comparisons address an important question that ordinary reference agreement can obscure.

  2. The diagnostics include repeated generations, two annotation models, source-level bootstrap intervals, and explicit controls for empty selections and missing outputs.

  3. The paper is unusually transparent about synthetic references, weak cross-model agreement, baseline training overlap, and generation failures.

Critical weaknesses
  1. My main concern is whether the references represent useful personalization. Sec.6 acknowledges the absence of human validation. Highlight annotations use frame captions, while title annotations use only speech transcripts. Reproducible persona effects demonstrate annotation consistency, but do not establish factual grounding or relevance to a viewer. This limits the benchmark's intended use.

  2. The title metrics provide limited evidence about meaning. In Table 2, cross-model persona contrast is only 0.003 in METEOR, compared with 0.083 and 0.216 within models. The authors appropriately restrict this conclusion to model-specific wording. Still, lexical overlap cannot distinguish different expressions of relevant content from different persona interpretations. The scoring needs independent support beyond wording agreement.

  3. The effective title evaluation is small and selective. Table 15 reports only 16 videos for full-pair results and 14 for paired contrasts. Requiring valid titles from both annotators for every interval excludes otherwise usable pairs. Together with the earlier interest screen, this restricts coverage. Baseline title results using all available interval-level references can clarify sensitivity to the completion rule.

  4. Output failures and prior exposure limit absolute model comparisons. Qwen3-VL fails on 147/252 persona-conditioned outputs, and the longer-output control still fails on 115/252 (Table 18). QD-DETR and CG-DETR previously saw 17/27 test source videos. The paired persona contrasts remain informative, but these issues complicate comparisons between models.

Questions for authors

As shown in weaknesses.

1 comment

Thank you for the detailed review and for recognizing the same-video comparisons, the diagnostics, and our transparency about limitations. Changes related to your review are shown in orange, or in purple where they also answer other reviewers, and are tagged G4Xh in the margin of the revised PDF.

W1. Do the references represent useful personalization rather than consistency only?

We agree that consistency does not establish relevance, and added a study testing whether matched viewers prefer their own-persona references in a selected GPT-OSS subset. Participants were recruited through CloudResearch Connect and matched to personas using demographic information and an interest checklist. The 192 participants, 12 per persona, watched each video and ranked the original annotation and the references of the two personas, on clips chosen where the two persona references differ. Matched viewers ranked their own persona's reference above the other persona's in 61.5% [57.5, 65.3] of highlight and 57.1% [52.4, 62.0] of chapter comparisons (chance 50%). A test over trials gives p < .001 and p = .018, and the preference is stronger where our metrics separate the two references (Spearman ρ = .57 and .61). The study concerns relevance, not factual accuracy. Construction requires each title to cite supporting ASR utterances and selects highlights from captioned units, and Sec. 6 states that we do not audit references for factual accuracy.

Revised: Sec. 5.4, Table 5, App. G, Sec. 6.

W2. Is there support for the title evaluation beyond wording overlap?

Yes. Recomputing Table 2(a) with sentence embeddings (all-mpnet-base-v2, bge-large-en-v1.5) and rescaled BERTScore gives the same pattern, as the cross-model contrast stays positive but 5 to 27 times smaller than within each model (BERTScore .093 and .132 within, .005 across). Baseline title contrasts also remain near zero under these measures. Independently of any text metric, matched viewers preferred their own persona's titles in 57.1% of comparisons, more so where the METEOR distance between the two lists is larger (ρ = .61).

Revised: Sec. 4.4, App. H.5, Sec. 5.4.

W3. Is the title evaluation too small and selective?

Scoring every interval with an available reference raises the paired evaluation from 14 to 22 source videos (full results from 16 to 30), and target − other METEOR stays between +.001 and +.006 for all models and both references. Individual estimates shift, for example Mistral-Nemo against the GPT-OSS references from +.011 to +.005. The sign reversal of Mistral-Nemo against the Gemma references disappears under interval-level evaluation, so we removed that statement. Sec. 4.3 now states that the interest screen limits which persona contrasts the benchmark can test.

Revised: Sec. 4.3, Sec. 5.1, Sec. 5.3, App. H.3.

W4. Do output failures and training exposure complicate the comparisons?

The target-versus-other conclusions do not change. Grammar-constrained decoding removes all Qwen3-VL failures (155 of 280 outputs to none) with no reliable persona gain (target − other −.003 [−.025, .017] in base F1 and .000 [−.012, .012] in expanded F1), and the persona-free gaps of Qwen3-VL and Chapter-Llama shrink once failures are handled. Table 28 now reports absolute scores for both runs. For exposure, the 8 test videos excluded from QVHighlights training show weak contrasts for QD-DETR and CG-DETR (+.002 and +.006 base AP), so we applied the same protocol to all splits, on which no baseline is trained. On 56 excluded source videos, base AP target − other is +.031, +.030, +.022, and +.042 for CLIP, SigLIP2, QD-DETR, and CG-DETR, against +.008, +.013, +.025, and +.007 on the 90 included videos. Contrasts are thus not systematically larger on exposed videos, although QD-DETR's interval on excluded videos includes zero. The absolute target AP of QD-DETR and CG-DETR is similar on both groups (.219 and .230 excluded, .227 and .205 included). These videos belong to the QVHighlights validation set, so we call them excluded from supervised training rather than unseen.

Revised: Sec. 5.2, Sec. 5.3, App. H.1, App. H.4.

Presentation

We added a table that summarizes the tasks, references, conditions, contrasts, and metrics in one place, referenced from Sec. 5.1, and separated annotation inputs from baseline inputs there.

Revised: Table 14 in App. D, Sec. 5.1.

Summary

Key additions: a human preference study (Sec. 5.4, App. G), meaning-based title similarity (App. H.5), interval-level title evaluation (App. H.3), failure handling and exposure analyses (App. H.1, H.4), and a summary table (App. D). We thank the reviewer for the detailed weaknesses and hope the revision addresses the concerns. If it does, we would be grateful if you would consider updating your score.

Reviewer_48D2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces One Video, Many Viewers, a benchmark for persona-conditioned video highlight selection and chapter titling. It constructs synthetic annotations across videos using viewer personas and evaluates their consistency through repeated generations and cross-model comparisons. Experiments with existing highlight detection and chapter generation models examine whether target-persona conditioning improves agreement over other-persona or persona-free conditions.

Critical strengths
  • The paper introduces a meaningful benchmark for persona-conditioned video highlighting and chapter titling, addressing the differences in viewer preferences for the same video.
  • The evaluation includes useful controls across target, other, and no-persona conditions, along with consistency analyses of synthetic annotations across repeated generations and different annotation models.
Critical weaknesses
  • The benchmark relies entirely on LLM generated annotations without human evaluation or user studies. While the paper examines annotation consistency across repeated generations and different LLMs, these analyses do not establish whether the references reflect actual viewer preferences.
  • Not all chapter intervals necessarily require persona specific titles. The low cross-model persona contrast in Table 2 may partly reflect such persona-insensitive intervals, potentially obscuring meaningful persona differences in the evaluation.
Questions for authors
  • Could the authors provide a human preference study to assess whether the synthetic highlights and chapter titles are actually preferred by viewers with the corresponding interests?
  • Could the authors analyze whether the low cross-model persona contrast in Table 2 is partly due to chapter intervals that do not require persona-specific titles?
3 comments

Thank you for the careful review and for recognizing the motivation of the benchmark and its controls. Changes related to your review are shown in green, or in purple where they also answer other reviewers, and are tagged 48D2 in the margin of the revised PDF.

W1 & Q1. Are the references preferred by viewers with the corresponding interests?

Yes, for the selected GPT-OSS references we tested. We ran the suggested study with 192 participants recruited through CloudResearch Connect and matched to one persona of 8 persona pairs (12 per persona) using demographic information and an interest checklist. Participants watched each video and ranked the original annotation and the GPT-OSS references of the two personas. Matched viewers ranked their own persona's reference above the other persona's in 61.5% [57.5, 65.3] of highlight and 57.1% [52.4, 62.0] of chapter comparisons (chance 50%). A test over trials, which treats the selected clips as a sample, gives p < .001 and p = .018. Own-persona references received higher scores than other-persona references in all 16 viewer groups for highlights and in 12 for titles. The preference rises with the persona difference that our metrics measure (Spearman ρ = .57 and .61). The study covers 16 personas, 40 trials chosen where the two persona references differ, and the GPT-OSS references, and interest matching was moderate (7-point similarity 3.85 to the own persona against 3.27), as Sec. 6 and App. G state.

Revised: Sec. 5.4, Table 5, App. G (Table 24), Sec. 6.

W2 & Q2. Is the low cross-model title contrast due to intervals that need no persona-specific title?

Partly. Identical titles for the two compared personas are concentrated in Gemma (44% of test intervals, GPT-OSS 4%). An identical title shows that the annotator did not personalize that interval, which can reflect content that needs no personalization or a missed opportunity, and the near-identical repeated Gemma generations explain why its within-model contrast is larger. Using this sensitivity as a proxy, measured in one generation with the contrast computed in another, the cross-model contrast stays small in every tertile (point estimates at most +.008 METEOR, against .083 and .216 within models), and baseline contrasts stay near zero on intervals whose persona references differ. In the human study, chapter trials whose two lists share a title were at chance (50.3%, 6 trials) against 61.1% for the other 10 (difference 10.8 points, 95% CI [2.8, 19.9]), consistent with your point that such intervals dilute persona differences. The remaining cross-model gap is therefore not explained by this proxy alone, and we keep the two annotators' references separate.

Revised: Sec. 4.4, App. H.6, App. G.

Summary

Key additions: a human preference study (Sec. 5.4, App. G) and an analysis of persona-insensitive intervals (Sec. 4.4, App. H.6). We thank the reviewer for both questions, which strengthened the paper, and hope the revision addresses the concerns. If it does, we would be grateful if you would consider updating your score.

Thank you for the detailed rebuttal and additional experiments, which have addressed several of my original concerns.

However, as raised by Reviewer BoZ7 and acknowledged by the authors, the human study does not establish whether automatic benchmark scores align with human judgments of baseline outputs. Given the relatively modest human preference results, I remain uncertain whether the proposed evaluation reliably captures meaningful persona-specific preferences.

While I appreciate the additional experiments and the importance of the this benchmark, I believe its automatic evaluation remains insufficiently established. I will therefore maintain my original score.

Thank you for the follow-up and for acknowledging the additional experiments.

We agree that the study does not validate automatic scores of baseline outputs against human judgments, and we state this scope in Sec. 6. Our main conclusions, however, rest on paired contrasts, which compare a model's outputs for the target and another persona against the same references, rather than on absolute scores. Such a contrast is meaningful when the references differ between personas in ways that viewers perceive, and this is what the study supports. Matched viewers preferred their own persona's references, and the preference was stronger in trials where our metrics separate the two references more (Spearman ρ = .57 and .61).

The preference is moderate on average because it mixes content where personalization matters with content where it matters little. Chapter trials whose two lists share a title were at chance (50.3%), against 61.1% for the others (Appendix G). Collecting human judgments of baseline outputs is the natural next step, and we note it as future work.

We appreciate your engagement throughout the discussion.

Reviewer_BoZ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper introduces One Video, Many Viewers, a benchmark for persona-conditioned video highlight selection and chapter titling. It provides synthetic persona-conditioned references for 259 source videos and evaluates existing models under target-persona, other-persona, and persona-free settings. The paper also studies annotation consistency through repeated generations and cross-model comparisons.

Critical strengths
  • Meaningful benchmark design: Unlike conventional query-based benchmarks, this work uses reusable viewer personas rather than video-specific queries. Evaluating multiple personas on the same video with fixed temporal intervals enables controlled comparisons of how general viewer interests influence highlight selection and chapter titling.

  • Careful diagnostic design: The target-persona, other-persona, and persona-free controls are useful. The annotation analysis also reveals an important limitation: persona effects are consistent within each annotation model, but are poorly shared across models, especially for chapter titles.

  • Transparent reporting: The paper clearly reports annotation failures, incomplete samples, baseline failures, and training-data overlap rather than hiding these issues.

Critical weaknesses
  • No human validation of the benchmark references: The benchmark is evaluated entirely against synthetic annotations. Repeated agreement between LLM annotations does not gurantee that the selected highlights or titles are actually appropriate for the persona or grounded in the video. This is especially important for chapter titles, where different annotation models show very different persona effects.

  • Some baseline results are dominated by generation failures: Qwen3-VL fails on 58.3% of persona-conditioned highlight outputs, and still fails on 45.6% with a larger output budget. Chapter-Llama also has substantial format and output-length failures. These results mix personalization quality with output-format reliability, making the corresponding comparisons difficult to interpret.

  • Chapter evaluation is heavily filtered: A video–persona pair is kept only when both annotation models produce valid titles for every chapter. As a result, only 315 of 775 title pairs remain complete, and the paired title evaluation uses only 14 source videos. Many incomplete pairs miss only a small number of titles, so discarding the entire pair removes otherwise usable annotations. It is unclear whether the main conclusions remain unchanged under interval-level evaluation using all valid titles.

Questions for authors
  • Can the authors provide human evaluation of whether the synthetic highlights and titles are both video-grounded and appropriate for the target persona? How well do the automatic benchmark scores correlate with these judgments?

  • Do the main conclusions hold after addressing output-length and formatting failures for Qwen3-VL and Chapter-Llama?

  • Why require complete annotations for every chapter in a video–persona pair? Do the chapter-titling conclusions remain the same if evaluation is performed at the interval level using all valid references?

3 comments

Thank you for the careful review and for recognizing the benchmark design, the controls, and our transparent reporting. Changes related to your review are shown in blue, or in purple where they also answer other reviewers, and are tagged BoZ7 in the margin of the revised PDF.

W1 & Q1. Are the references appropriate for the persona, and do automatic scores track human judgments?

We added a human preference study with real viewers, using selected GPT-OSS references to assess whether matched viewers prefer their own-persona references. Participants were recruited through CloudResearch Connect and matched to personas using demographic information and an interest checklist. The 192 participants, 12 per persona, watched each video and ranked the original annotation and the references of the two personas, on clips chosen where the two persona references differ. Matched viewers ranked their own persona's reference above the other persona's in 61.5% [57.5, 65.3] of highlight and 57.1% [52.4, 62.0] of chapter comparisons (chance 50%). A test over trials, which treats the selected clips as a sample, gives p < .001 and p = .018. At the trial level, metric distance between the two persona references was associated with own-persona preference (24 highlight trials: Spearman ρ = .57, permutation p = .004; 16 chapter trials: ρ = .61, p = .012). This association does not establish whether automatic scores of baseline outputs agree with human judgments, because participants ranked references rather than baseline outputs, and Sec. 6 now states this. For grounding, construction requires each title to cite its supporting ASR utterances and selects highlights from captioned units, and Sec. 6 also states that we do not audit each reference for factual accuracy.

Revised: Sec. 5.4, Table 5, App. G, Sec. 6.

W2 & Q2. Do the conclusions hold once Qwen3-VL and Chapter-Llama failures are removed?

For the target-versus-other conclusions, yes. Grammar-constrained decoding with the same checkpoint, frames, prompt, and sampling removes all Qwen3-VL failures (155 of 280 outputs to none). We detect no reliable target-persona gain (target − other −.003 [−.025, .017] in base F1 and .000 [−.012, .012] in expanded F1), and the deficit against persona-free outputs shrinks from −.049 to −.010 [−.032, .012] once failures are removed. For Chapter-Llama, we recovered all 287 failed outputs by their first title, which changes output parsing rather than rerunning the model. This keeps target − other near zero and shrinks its advantage over persona-free outputs from +.018 to +.009 against GPT-OSS references and from +.022 to +.004 against Gemma references. Table 28 now also reports absolute scores and expanded F1.

Revised: Sec. 5.2, Sec. 5.3, App. H.4, Table 28.

W3 & Q3. Why complete pairs, and do the title conclusions hold at the interval level?

Complete pairs let us score every model on the same intervals against both annotation models. Scoring every interval with a valid reference raises the paired evaluation from 14 to 22 source videos (full results from 16 to 30), and target − other METEOR stays between +.001 and +.006 for all models and both references. Individual estimates shift, for example Mistral-Nemo against the GPT-OSS references from +.011 to +.005. The sign reversal of Mistral-Nemo against the Gemma references disappears under interval-level evaluation, so we removed that statement.

Revised: Sec. 5.1, Sec. 5.3, App. H.3.

Summary

Key additions: a human preference study with a test of whether our metrics' persona differences match the viewers' (Sec. 5.4, App. G), failure handling with absolute scores (App. H.4), and interval-level title evaluation (App. H.3). We thank the reviewer for these questions, which led directly to the human study, and hope the revision addresses the concerns. If it does, we would be grateful if you would consider updating your score.

Thanks for the rebuttal. While additional results and human preference experiments have been provided, I still have major concerns about the study. In particular, I find the experimental design of randomly assigning each participant to one of two predefined personas insufficiently justified. Although demographic information and an interest checklist are collected, there is still a substantial chance that neither persona accurately represents the participant's characteristics, potentially introducing considerable noise into the study. The ranking results also suggest only moderate correlations. Furthermore, the newly added material contains too many details, making it difficult to grasp the main findings. I will maintain my weak reject rating for now.

Thank you for the follow-up. We clarify the three points briefly.

Assignment. Participants were not assigned at random. Each pair contrasts personas from different archetypes, and each participant was assigned to the persona in the pair that better matched their screening answers. We agree that neither persona may fit a participant closely, and we report the moderate matching in Sec. 6. Such mismatch adds noise that pulls preferences toward chance, so it can weaken but not create an own-persona preference, which makes the test conservative. Because both groups of a pair rank the same options, differences in option quality or length cannot produce the effect either.

Effect size. The preference is moderate, 61.5% and 57.1% against a 50% chance level, but consistent, holding in all 16 viewer groups for highlights and in 12 for titles. The average also mixes content where personalization matters with content where it matters little. The preference is stronger in trials whose two references differ more, and chapter trials whose two lists share a title were at chance (50.3% against 61.1% for the others, Appendix G). Identifying, for highlights and chapters separately, where personalization is most useful is a natural direction for future work.

Main findings. In short, (1) viewers matched to a persona prefer its references over another persona's (Table 5, first row), (2) this preference is stronger when our metrics separate the two references more, and (3) the evidence covers selected GPT-OSS references and does not validate scores of baseline outputs. All other details are in Appendix G.

We appreciate your engagement and hope this clarifies the study.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank you for your time and careful reviews. We are glad that the reviewers found the benchmark design meaningful, in particular comparing personas on the same video (BoZ7, 48D2, G4Xh), considered the target, other-persona, and persona-free controls and the diagnostics useful (BoZ7, 48D2, G4Xh), and appreciated our transparent reporting of synthetic references, failures, and training overlap (BoZ7, G4Xh).

We know your time is limited, so we summarize the main revisions below, and each review also has its own response.

In the revision, we have

  • added a human preference study in which 192 viewers matched to the personas ranked selected GPT-OSS references (Sec. 5.4, Table 5, App. G), as raised by all reviewers. Matched viewers ranked their own persona's reference above the other persona's in 61.5% of highlight and 57.1% of chapter comparisons (chance 50%), more so where our metrics separate the two references.
  • tested whether the persona differences our metrics measure match the viewers' judgments (Table 5, App. G), as asked by reviewer BoZ7.
  • handled the generation failures of Qwen3-VL and Chapter-Llama and reported absolute scores before and after (Sec. 5.2, Sec. 5.3, App. H.4), as raised by reviewers BoZ7 and G4Xh.
  • evaluated titles at the interval level without the completion filter (Sec. 5.1, Sec. 5.3, App. H.3), as raised by reviewers BoZ7 and G4Xh.
  • stated how the interest screen limits the testable persona contrasts (Sec. 4.3), as raised by reviewer G4Xh.
  • analyzed intervals that receive identical titles for both personas (Sec. 4.4, App. H.6), as raised by reviewer 48D2.
  • recomputed the title diagnostics with meaning-based similarity (Sec. 4.4, App. H.5), as raised by reviewer G4Xh.
  • evaluated highlights on all splits and by training exposure (Sec. 5.2, App. H.1), as raised by reviewer G4Xh.
  • added a table that summarizes the tasks, references, conditions, and contrasts (Table 14 in App. D), as suggested by reviewer G4Xh.

In the revised PDF, changes are colored by reviewer, blue for reviewer BoZ7, green for reviewer 48D2, and orange for reviewer G4Xh, and changes that answer several reviewers are purple, with the reviewer IDs in the margin. A table of all revisions precedes the appendices.

We hope these revisions address your concerns, and if they do, we would be grateful if you would consider updating your score. Thank you again for your time and insightful comments.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.