acceptodds
Under review as a conference paper at ICLR 2027

PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?

Abstract

Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether their judgments remain reliable when the evidence is buried in day-long videos has yet to be established. Existing benchmarks cannot answer this. Their videos are typically only a few minutes long, many answer pairs can be separated from the transcript alone, and collecting human judgments does not scale to ultra-long videos. We introduce PlaylistEval, an agentic framework that builds video-language judge benchmarks over -hour playlist collection without human annotation. It automatically generates questions with paired answers whose differences are controlled by causal degradation, so that every pair demands retrieval across the collection. The resulting benchmark contains pairs across seven domains spanning both static and dynamic knowledge, and on a stratified subset of pairs it agrees with human judgments % of the time (IAA ). Evaluating omnimodal and multimodal models from eight families reveals that frontier judges reach only % pairwise accuracy, while open-source judge models perform far behind. We further show that both retrieval and final judgment depend on using multiple modalities, and that judge accuracy degrades as the playlist set grows. We will release our pipeline, benchmark, and evaluation code upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
VLM-as-a-Judge, Meta-Evaluation, Video Understanding, Retrieval Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_YQHDOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PlaylistEval, an automated framework for constructing benchmarks that evaluate video-language judges over large video collections. It generates questions grounded in two video segments, creates graded answer variants with controlled visual errors, and applies verification gates and regeneration feedback to improve data quality. The resulting PlaylistBench contains 630 preference pairs across seven domains, each with approximately 100 hours of source videos. Evaluation covers 17 judges and four retrievers, with analyses of collection size, input modalities, answer position, and inference budgets. Human validation supports the reliability of the generated preferences, while the model results reveal substantial limitations in retrieving and evaluating the relevant evidence.

Critical strengths
  1. A useful benchmark design with substantial attention to data quality. Evaluating judges over video collections of this scale is a meaningful extension of short-video preference evaluation. The combination of evidence-grounded questions, graded visual perturbations, verification across model families, and regeneration feedback provides a practical way to construct challenging examples.
  2. The experiments provide informative diagnosis beyond a model leaderboard. The paper evaluates a broad range of judges and retrievers and systematically varies retrieval scope, modalities, answer order, reasoning budget, and visual input budget. The consistent decline as the search collection grows, together with the error analysis distinguishing retrieval, frame sampling, perception, and reasoning failures, helps explain where current systems struggle. This makes the benchmark useful for guiding improvements in evidence retrieval and judging.
Critical weaknesses

Human annotators receive selected evidence clips and optional highlighting of answer differences (Appendices F and G.3), whereas the main model evaluation uses retrieved or uniformly sampled evidence. The human results are strong evidence of label quality, but the numerical gap should not be interpreted as a comparison under identical evidence conditions.

Questions for authors
  1. Are model results available on the same 152 human-evaluated pairs using the same selected evidence clips, to help interpret the reported human-model gap?

  2. Do the existing text-only ranking outputs indicate how often the final selected pairs can be ordered correctly without video, even when the full five-answer ordering is incorrect?

Please address these questions in the rebuttal.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YQHD (1/3)

We appreciate your strong support. Thank you so much for recognizing our benchmark design and its attention to data quality and diagnostic analysis.

We address your questions and weaknesses below:

Evidence-matched comparison of model performance

(Question 1) Are model results available on the same 152 human-evaluated pairs using the same selected evidence clips, to help interpret the reported human-model gap?

Here are the results in matched conditions for human-validated 147 pairs (5 were overturned by the annotators out of 152) in oracle evidence-moments settings:

JudgeAgreement
Qwen-3.7-Flash90.8
Gemini-3.5-Flash-Lite90.5
Gemma-4-26B-A4B86.1
Human annotators95.4

The performance of the Judge model is within 10% of human agreement when the gold evidence moments are provided. The reason is our benchmark covers long-form videos where the number of entities and events changes less rapidly within the short span. Thus, the model can decode the masked entity in the question from the transcript and/or video and provide the correct judgement. However, this becomes increasingly difficult as the database size grows to 1 hour or 1 day.

We derived the ground truth evidence to keep the human annotation feasible. The human agreement reported in the paper (93.0% over all 152 pairs; 95.4% per rating on these 147) was therefore measured with the evidence in hand, so we use it as a reference point for judges rather than as a matched baseline. We will clarify this in the Introduction and Results sections.

Our framework is intended to be multi-skill similar to any large database multi-hop QA benchmark, such as HotpotQA [1] over Wikipedia database. Even if sampling is perfect the models still can suffer long-context degradation at a few-minutes to hour-level. For example, in LongVideoBench [2] Table 5, the performance is far from perfect accuracy even with 256 frames for few minutes videos for strong model families like GPT-4o.

In our paper, as mentioned in Section 4.4 and Appendix C.2, even with agentic tool use in gold segments of 10 to 30-minutes where the agent collects its own frames, the performance gain is only about 2 points with 3 steps (median) of search and reasoning. The number of steps required for the agentic model is given below (median = 3 steps):

Tool-use stepsPairsPercentage
0-1355.6%
220833.0%
3-429246.3%
5 or more9515.1%

The goal of our benchmark is to go beyond a few-minutes level of video-language judgement. Though the deciding moments may be shorter than 1 hour in total, they are scattered across the 100-hour database. Therefore, our framework poses a unique challenge that is absent in previous short-video judge benchmarks.

AuthorsOpenReview ↗

Response to Reviewer YQHD (2/3)

Ordering verification and Text-only solvability of pairs

(Question 2) Do the existing text-only ranking outputs indicate how often the final selected pairs can be ordered correctly without video, even when the full five-answer ordering is incorrect?

We use the strongest model of the family for text-only ranking prediction in phase 2: Gemini-3.1-Pro and GPT-5.4. This ordering capability also includes the parametric knowledge of the models. As our dataset includes factual domains like History and Art, even without text-only shortcuts, one model can predict order partially, while other can fail. Again, as the parametric knowledge of different models are different within the family and sub-family, it is practically challenging to eliminate this effect completely without biasing the benchmark against particular families.

During question generation in Phase 1, we verified that the two entities in questions are difficult to infer without watching the video (Example in Figure 1). Again, during distractor generation in Phase 2, we prompted the Generator model (Gemini-3-Flash) to output meta-data for its pieces of modification. Moreover, in the structural check in Phase 2, our pipeline verifies there is at least one visual-only change for each of all 4 generated answers. If the generator model has strong enough video understanding capability, the chances of text-only shortcut can be prevented even before going to the ranking prediction stage.

Moreover, during final pair sampling as described in Section 3.3 ("Controllable Pair Selection"), we employ two small models with 3B and 8B active parameters and ensure at least one of them fails for the sampled pairs. If the pairs are trivially solvable only by looking at the text and transcript, we expect them to be solvable by these small models.

Reference judgesPairsPercentage
Both fail27143.0%
Only Qwen3-VL fails17728.1%
Only InternVL fails18228.9%

Finally, during human annotation, the annotators spent about 7 to 11 minutes (median) as discussed in Appendix F. If the ranking can be predicted just looking at the text difference, we expect this duration to be much shorter.

AuthorsOpenReview ↗

Response to Reviewer YQHD (3/3)

Question and answer-only results

JudgeVideoRewardBench long, original orderOurs, original orderVideoRewardBench long, right in both ordersOurs, right in both ordersGap in both ordersSame letter in both orders, VideoRewardBench / ours
Gemini-3.7-Flash70.361.067.548.4-19.17.8% / 29.5%
Qwen-3.8-Max63.159.449.143.5-5.625.8% / 33.2%
Qwen-3.5-9B65.050.549.134.8-14.331.1% / 37.3%
GLM-4.6V-Flash67.849.453.016.8-36.232.2% / 73.0%
Gemma-4-26B-A4B71.450.265.033.7-31.310.2% / 38.4%

Here, we evaluate top 2 API models, top 2 open-weights models and one additional model (GLM-4.6V-Flash) in question and answer-only setup. We compare our results with another video-language judge benchmark (VideoRewardBench [3], long-form perception subset). Our benchmark is less prone to both parametric and text shortcuts. For example, Qwen-3.5-9B scores only 50.5% while it can achieve 65.0% in VideoRewardBench. For large models, Gemini-3.7-Flash reaches 70.3% in VideoRewardBench, while it is only 61.0% in our case. Moreover, when the model must be right in both orders, the accuracy drops by 12.6% while it only drops about 2.8% for VideoRewardBench in both orders. Therefore, our benchmark mitigates text-only and parametric shortcuts within the current setup in the paper.

Again, educated guessing in text-only setup does not imply strong performance in the final benchmark automatically. The final performance of top 3 models is less than 76% in both benchmarks as given below.

JudgeVideoRewardBench long-form perceptionOurs
Gemini-3.7-Flash74.475.4
Qwen-3.8-Max74.473.9
GPT-5.6-Terra73.172.5

Moreover, our benchmark results in identical ranking in LVBench, a popular long-video understanding benchmark as discussed in Appendix C.3 of our manuscript.

JudgeLVBenchOurs
Gemini-3.7-Flash85.475.4
Qwen-3.8-Max81.873.9
GPT-5.6-Terra78.972.5

Finally, in this work, we propose a minimal setup to demonstrate the effectiveness of the components of our framework where the final pairs are determined by multiple models in different complementary stages. It can be extended to support complex workflows, for example, allowing higher threshold of "two" gap-1 discordant pairs (currently one) or adding more models and ensure that all of them fails, and so on. There are possible tradeoffs for those parameters, such as, with higher discordant pair threshold, the benchmark might be biased by used models in that gate.

In this paper, we aim to study models of diverse parameter size from 2B to 2T+ parameters, hence we chose a smaller pair sampler of 3B and 8B active parameters. If the goal is to study only large models, such as 100B+ parameters, the pair sampler could be scaled accordingly to decrease the baseline performance.

With our experiments and analysis, we conclude that our current framework results in a consistent benchmark with human-curated ones, like VideoRewardBench and LVBench - similar ranking of top-3 models and lower text-only score.

[1] Yang, Zhilin, et al. "HotpotQA: A dataset for diverse, explainable multi-hop question answering." Proceedings of the 2018 conference on empirical methods in natural language processing. 2018.

[2] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." Advances in Neural Information Processing Systems 37 (2024): 28828-28857.

[3] Zhang, Zhihong, et al. "Videorewardbench: Comprehensive evaluation of multimodal reward models for video understanding." arXiv preprint arXiv:2509.00484 (2025).

The authors have addressed my concerns, I'll keep my rating.

Reviewer_LB4UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper mainly studies whether video-language models can reliably compare answers when the supporting evidence is scattered across a large video collection. It introduces an automated pipeline to generate questions grounded in two video segments, constructs candidate answers with different level of degraded visual errors, and filters them using several verification stages. The benchmark contains 630 preference pairs across seven domains, each with about 100 hours of video. The paper evaluates 17 models and examines the effects of retrieval, collection size, input modality, and answer order.

Critical strengths
  1. The paper addresses an interesting problem. As model-based judges become more common in video evaluation, understanding when their judgments fail is important, particularly when relevant evidence must first be retrieved.

  2. The answer construction is reasonably controlled. Modifying visual details while keeping the writing style and structure similar helps reduce some confounding factors in answer comparison. The graded errors also provide a way to vary difficulty.

  3. The evaluation covers multiple relevant settings, including retrieval versus uniform sampling, different search scopes, modality ablations, and answer-order sensitivity. The human study also provides evidence for the label quality: most judgments agree with the intended preference on 147/152 sampled pairs.

Critical weaknesses
  1. The text-only filtering in Section 3.2 does not directly test whether the final answer pairs require video evidence. A set is rejected only when both text-only judges correctly rank all five answers. Suppose the intended order is A > B > C > D > E, but a text-only judge predicts A > B > D > C > E. This fails the full-ordering check, so the set can survive filtering, even though the judge correctly resolves nine of the ten possible pairs without video. A pair such as A versus E could therefore remain solvable from text alone. Moreover, a set can survive even if one judge recovers the entire ordering, provided the other does not. This does not prove that the final benchmark contains substantial textual shortcuts, but the filtering rule does not rule them out. The modality ablation shows that video helps on average; checking shortcuts on the final selected pairs would better establish whether it is necessary.

  2. A question requiring two segments does not guarantee that comparing its answers requires both. If two answers differ only in a visual detail from one segment, that segment alone may determine the preference. A first-segment-only / second-segment-only / both-segments ablation would help verify the claimed need for multi-segment reasoning.

  3. The main accuracy conflates retrieval, frame selection, and judgment errors. Even the oracle setting can miss decisive moments when sampling from the two gold segments. The comparison with human agreement also uses different evidence conditions, since humans receive selected clips and optional highlighting of answer differences.

Questions for authors
  1. What are the question-and-answers-only and transcript-only accuracies on the final 630 pairs? Would using the shortcut filter at the pair level substantially change the retained dataset?

  2. How does performance change when a judge receives only the first gold segment, only the second, or both? What proportion of pairs genuinely requires evidence from both? Can you do some ablations about this?

  3. Can you evaluate judges and humans under matched evidence conditions, using the same verified evidence clips? This would help separate judgment failures from evidence-selection failures.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LB4U (1/4)

Thank you very much for insightful questions and suggestions. We address your questions and concerns below:

Ordering verification and Text-only solvability of pairs

(Weakness 1) The text-only filtering in Section 3.2 does not directly test whether the final answer pairs require video evidence. A set is rejected only when both text-only judges correctly rank all five answers. Suppose the intended order is A > B > C > D > E, but a text-only judge predicts A > B > D > C > E. This fails the full-ordering check, so the set can survive filtering, even though the judge correctly resolves nine of the ten possible pairs without video. A pair such as A versus E could therefore remain solvable from text alone. Moreover, a set can survive even if one judge recovers the entire ordering, provided the other does not. This does not prove that the final benchmark contains substantial textual shortcuts, but the filtering rule does not rule them out. The modality ablation shows that video helps on average; checking shortcuts on the final selected pairs would better establish whether it is necessary.

(Question 1) What are the question-and-answers-only and transcript-only accuracies on the final 630 pairs? Would using the shortcut filter at the pair level substantially change the retained dataset?

We use the strongest model of the family for text-only ranking prediction in Phase 2: Gemini-3.1-Pro and GPT-5.4. This ordering capability also includes the parametric knowledge of the models. As our dataset includes factual domains like History and Art, even without text-only shortcut, one model can predict order partially, while other can fail. Again, as the parametric knowledge of different models are different within the family and sub-family, it is practically challenging to eliminate this effect completely without biasing the benchmark against particular families.

During question generation in Phase 1, we verified that the two entities in questions are difficult to infer without watching the video (Example in Figure 1). Again, during distractor generation in Phase 2, we prompted the Generator model (Gemini-3-Flash) to output meta-data for its pieces of modification. Moreover, in the structural check in Phase 2, our pipeline verifies there is at least one visual-only change for each of all 4 generated answers. If the generator model has strong enough video understanding capability, the chances of text-only shortcut can be prevented even before going to the ranking prediction stage.

Moreover, during final pair sampling as described in Section 3.3 ("Controllable Pair Selection"), we employ two small models with 3B and 8B active parameters and ensure at least one of them fails for the sampled pairs. If the pairs are trivially solvable only by looking at the text and transcript, we expect them to be solvable by these small models.

Reference judgesPairsPercentage
Both fail27143.0%
Only Qwen3-VL fails17728.1%
Only InternVL fails18228.9%

Finally, during human annotation, the annotators spent about 7 to 11 minutes (median) as discussed in Appendix F. If the ranking can be predicted just looking at the text difference, we expect this duration to be much shorter.

AuthorsOpenReview ↗

Response to Reviewer LB4U (2/4)

W1-1: Question and answer-only results

JudgeVideoRewardBench long, original orderOurs, original orderVideoRewardBench long, right in both ordersOurs, right in both ordersGap in both ordersSame letter in both orders, VideoRewardBench / ours
Gemini-3.7-Flash70.361.067.548.4-19.17.8% / 29.5%
Qwen-3.8-Max63.159.449.143.5-5.625.8% / 33.2%
Qwen-3.5-9B65.050.549.134.8-14.331.1% / 37.3%
GLM-4.6V-Flash67.849.453.016.8-36.232.2% / 73.0%
Gemma-4-26B-A4B71.450.265.033.7-31.310.2% / 38.4%

Here, we evaluate top 2 API models, top 2 open-weights models and one additional model (GLM-4.6V-Flash) in question and answer-only setup. We compare our results with another video-language judge benchmark (VideoRewardBench [1], long-form perception subset). Our benchmark is less prone to both parametric and text shortcuts. For example, Qwen-3.5-9B scores only 50.5% while it can achieve 65.0% in VideoRewardBench. For large models, Gemini-3.7-Flash reaches 70.3% in VideoRewardBench, while it is only 61.0% in our case. Moreover, when the model must be right in both orders, the accuracy drops by 12.6% while it only drops about 2.8% for VideoRewardBench in both orders. Therefore, our benchmark mitigates text-only and parametric shortcuts within the current setup in the paper.

Again, educated guessing in text-only setup does not imply strong performance in the final benchmark automatically. The final performance of top 3 models is less than 76% in both benchmarks as given below.

JudgeVideoRewardBench long-form perceptionOurs
Gemini-3.7-Flash74.475.4
Qwen-3.8-Max74.473.9
GPT-5.6-Terra73.172.5

Moreover, our benchmark results in identical ranking in LVBench, a popular long-video understanding benchmark as discussed in Appendix C.3 of our manuscript.

JudgeLVBenchOurs
Gemini-3.7-Flash85.475.4
Qwen-3.8-Max81.873.9
GPT-5.6-Terra78.972.5

W1-2: Transcript-only results

We discuss this modality ablation in Section 4.3 and Figure 5 for 3 models. The transcript can provide hints to the model that can help resolve the entities by guessing. We notice a 3-6% gap from this result. However, watching video does not guarantee that the model can use that information effectively. We provide error analysis of such cases in Section 4.5.

JudgeTranscript onlyVideo onlyVideo + transcript
Qwen-3.7-Flash61.7 (−5.8)65.7 (−1.8)67.5
Gemini-3.5-Flash-Lite59.0 (−3.1)59.8 (−2.3)62.1
Gemma-4-26B-A4B55.4 (−4.3)56.3 (−3.4)59.7

W1-3: Alternative setups

Finally, in this work, we propose a minimal setup to demonstrate the effectiveness of the components of our framework where the final pairs are determined by multiple models in different complementary stages. It can be extended to support complex workflows, for example, allowing higher threshold of "two" gap-1 discordant pairs (currently one) or adding more models and ensuring that all of them fails, and so on.

There are possible tradeoffs for those hyper-parameters, such as, with higher discordant pair threshold, the benchmark might be biased by used models in that gate. In this paper, we aim to study models of diverse parameter size from 2B to 2T+ parameters, hence we chose a smaller pair sampler of 3B and 8B active parameters. If the goal is to study only large models, such as 100B+ parameters, the pair sampler could be scaled accordingly to decrease the baseline performance.

With our experiments and analysis, we conclude that our current framework results in a consistent benchmark with human-curated ones, like VideoRewardBench and LVBench - similar ranking of top-3 models and lower text-only score.

AuthorsOpenReview ↗

Response to Reviewer LB4U (3/4)

Multi-segment requirement for pairwise evaluation

(Weakness 2) A question requiring two segments does not guarantee that comparing its answers requires both. If two answers differ only in a visual detail from one segment, that segment alone may determine the preference. A first-segment-only / second-segment-only / both-segments ablation would help verify the claimed need for multi-segment reasoning.

(Question 2) How does performance change when a judge receives only the first gold segment, only the second, or both? What proportion of pairs genuinely requires evidence from both? Can you do some ablations about this?

Multi-segment requirement annotation from our pipeline

During QA and distractor generation in both phases, we asked the Generator model (Gemini-3-Flash) to output meta-data for the segment labels of the entities in the question and the modifications in the answer. Here is the segment-level breakdown of answer differences in visual evidence.

Answers differ inPairsPercentage
Both segments42968.1%
Segment 2 only11418.1%
Segment 1 only8713.8%

In about 68% of our final pairs, both segments are modified with at least one visual evidence in each by the generator model in Phase 2.

Downstream effect of multi-segment presence

We also provide oracle performance of Qwen-3.7-Flash model in the oracle setup with 10 to 30-minutes segments.

Qwen-3.7-Flash evidenceAccuracy
Gold segment 1 only64.5
Gold segment 2 only67.6
Both gold segments73.6

With both segments the performance is highest in comparison to using only one segment.

AuthorsOpenReview ↗

Response to Reviewer LB4U (4/4)

Evidence retrieval failure and judgement failure

(Weakness 3) The main accuracy conflates retrieval, frame selection, and judgment errors. Even the oracle setting can miss decisive moments when sampling from the two gold segments. The comparison with human agreement also uses different evidence conditions, since humans receive selected clips and optional highlighting of answer differences.

(Question 3) Can you evaluate judges and humans under matched evidence conditions, using the same verified evidence clips? This would help separate judgment failures from evidence-selection failures.

Here are the results in matched conditions for human-validated 147 pairs in oracle evidence-moments settings:

JudgeAgreement
Qwen-3.7-Flash90.8
Gemini-3.5-Flash-Lite90.5
Gemma-4-26B-A4B86.1
Human annotators95.4

The performance of the Judge model is within 10% of human agreement when the gold evidence moments are provided. The reason is our benchmark covers long-form videos where the number of entities and events changes less rapidly within the short span. Thus, the model can decode the masked entity in the question from the transcript and/or video and provide the correct judgement. However, this becomes increasingly difficult as the database size grows to 1 hour or 1 day.

We derived the ground truth evidence to keep the human annotation feasible. The human agreement reported in the paper (93.0% over all 152 pairs; 95.4% per rating on these 147) was therefore measured with the evidence in hand, so we use it as a reference point for judges rather than as a matched baseline. We will clarify this in the Introduction and Results sections.

Our framework is intended to be multi-skill similar to any large database multi-hop QA benchmark, such as HotpotQA[2] over Wikipedia database. Even if sampling is perfect the models still can suffer long-context degradation at a few-minutes to hour-level. For example, in LongVideoBench[3] Table 5, the performance is far from perfect accuracy even with 256 frames for few minutes videos for strong model families like GPT-4o. In our paper, as mentioned in Section 4.4 and Appendix C.2, even with agentic tool use in gold segments of 10 to 30-minutes, the performance gain is only about 2 points with 3 steps (median) of search and reasoning. The number of steps required for the agentic model is given below (median = 3 steps):

Tool-use stepsPairsPercentage
0-1355.6%
220833.0%
3-429246.3%
5 or more9515.1%

The goal of our benchmark is to go beyond a few-minutes level of video-language judgement. Though the deciding moments may be shorter than 1 hour in total, they are scattered across the 100-hour database. Therefore, our framework poses a unique challenge that is absent in previous short-video judge benchmarks.

[1] Zhang, Zhihong, et al. "Videorewardbench: Comprehensive evaluation of multimodal reward models for video understanding." arXiv preprint arXiv:2509.00484 (2025).

[2] Yang, Zhilin, et al. "HotpotQA: A dataset for diverse, explainable multi-hop question answering." Proceedings of the 2018 conference on empirical methods in natural language processing. 2018.

[3] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." Advances in Neural Information Processing Systems 37 (2024): 28828-28857.

Reviewer_e32AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether video-language models can reliably judge the quality of long-form answers over large video collections. The authors propose PlaylistEval, an automated framework for building evaluation data. It first generates questions and reference answers with evidence citations from two related video clips. It then creates incorrect answers by adding visual errors of different severity. Finally, it uses checks across different model families, feedback from failed checks, and hard-example filtering to build a preference benchmark. The resulting PlaylistBench covers seven domains, with 327 questions and 630 answer pairs. Each domain has around 100 hours of video.

The authors evaluate 17 models. The best model reaches 75.4% accuracy on answer pairs, while several specialized video judges perform close to random guessing. Ablation studies on retrieval scope and input modalities further show how evidence retrieval and multimodal checking affect the results. The main contributions are an automated and reproducible evaluation pipeline, a preference dataset, and a set of systematic analyses.

Critical strengths

1. The problem is important and relevant to the research community. Video judges are used to evaluate models, select answers, and potentially provide reward signals. Their reliability can affect model rankings and how models are improved. By including evidence retrieval and answer verification over large video collections, this paper provides a concrete way to study this problem.

2. The benchmark construction pipeline is reusable and allows controlled comparisons. By adding visual errors of different severity to the same reference answer, the authors can create challenging answer pairs while keeping their length, tone, and structure relatively similar. Evidence citations, edit records, automatic checks, and feedback work together to help researchers rebuild the benchmark on new video collections. The pipeline also makes it possible to study the effects of search scope, input modalities, and differences in answer quality.

3. The experiments are fairly comprehensive, and the synthetic labels are checked by humans. The paper evaluates models from several families and includes ablation studies on retrieval, input modalities, and search scope. In a human review of 152 answer pairs, 93.0% of individual ratings agree with the intended preference, and 147 pairs agree with the intended label based on majority vote. These results provide evidence for the quality of the dataset and its usefulness in analyzing model behavior.

Critical weaknesses
  • The current filtering criteria do not fully ensure that each answer pair requires visual evidence. The no-video check tests whether a judge can recover the full intended ranking of five answers, while the final task only asks the judge to compare two answers. Failing to recover the full ranking does not necessarily mean the judge cannot make accurate pairwise comparisons. Also, the first-stage check only tests whether a question can be answered from transcripts. It does not directly show that transcripts alone are insufficient to compare two candidate answers. Therefore, the claim that every answer pair requires watching the video is not yet fully supported.

  • It is unclear how well the synthetic errors represent real model errors. Most incorrect answers are created by the same generator, which adds visual errors based on predefined attributes. This helps control the experiments, but these errors may differ from those made by real video models, such as missing details, incorrect attribution, object confusion, and errors in reasoning across clips. The current experiments do not clearly show whether judges that perform better on synthetic answer pairs can also evaluate naturally generated answers more reliably.

  • The need for cross-clip verification is not directly tested. Although the questions and reference answers involve two clips, this does not mean that choosing the better answer always requires both clips. In some cases, the key difference between two answers may come from only one clip, and checking that detail may be enough to decide which answer is better. This limits what the results can tell us about cross-clip verification. It also raises questions about how well the metric of "retrieving both reference clips" matches what is actually needed for the final judgment.

Questions for authors
  • Regarding Weakness 1: Beyond the existing modality ablations, could the authors directly examine a random sample of final answer pairs to check whether the details that determine the preference are already present in the transcripts, or whether other text-based shortcuts exist? If a more direct pairwise validation criterion is used, would the main model rankings and length effects still hold?

  • Regarding Weakness 2: Could the authors add a small-scale experiment using answers naturally generated by real video question-answering models? These answers could be used to build human-verified preference pairs, allowing the authors to test whether judge performance on PlaylistBench transfers to naturally occurring errors.

  • Regarding Weakness 3: Could the authors compare three settings: providing only the first clip, only the second clip, and both clips? This would help test how much the final preference judgment actually depends on having both clips. If some answer pairs only require one clip, how would the authors explain the need for cross-clip verification and the related retrieval metrics?

3 comments
AuthorsOpenReview ↗

Response to Reviewer e32A (1/2)

Thank you so much for your insightful questions and suggestions. We address your questions and weaknesses below:

Transcript/text-based shortcuts and pairwise validation

(Question 1) Beyond the existing modality ablations, could the authors directly examine a random sample of final answer pairs to check whether the details that determine the preference are already present in the transcripts, or whether other text-based shortcuts exist? If a more direct pairwise validation criterion is used, would the main model rankings and length effects still hold?

Explanation with specific example

We describe the complete example of Figure 1 in Appendix B.7. For the Rating 2 vs 1 pair, GPT-5.6-Terra focuses on the wrong "clock" brought by the retriever and reaches the wrong conclusion. A "Rotterdam" narration that echoes segment 2 (skyscrapers, a giant clock, a port) outranks the true Tacoma tower, and GPT-5.6-Terra follows it to the wrong answer. Therefore, it is challenging to perform evidence retrieval and judging at 100 hours scale as it requires finding the correct entities and events from many scenarios.

Pairwise validation

In our pipeline, we use the strongest model of the family for text-only ranking prediction in phase 2: Gemini-3.1-Pro and GPT-5.4. This ordering capability also includes the parametric knowledge of the models. As our dataset includes factual domains like History and Art, even without text-only shortcut, one model can predict order partially, while other can fail. Again, as the parametric knowledge of different models are different within the family and sub-family, it is practically challenging to eliminate this effect completely without biasing the benchmark against particular families.

During question generation in Phase 1, we verified that the two entities in questions are difficult to infer without watching the video (Example in Figure 1). Again, during distractor generation in Phase 2, we prompted the Generator model (Gemini-3-Flash) to output meta-data for its pieces of modification. Moreover, in the structural check in Phase 2, our pipeline verifies there is at least one visual-only change for each of all 4 generated answers. If the generator model has strong enough video understanding capability, the chances of text-only shortcut can be prevented even before going to the ranking prediction stage.

Moreover, during final pair sampling as described in Section 3.3 ("Controllable Pair Selection"), we employ two small models with 3B and 8B active parameters and ensure at least one of them fails for the sampled pairs. If the pairs are trivially solvable only by looking at the text and transcript, we expect them to be solvable by these small models.

Reference judgesPairsPercentage
Both fail27143.0%
Only Qwen3-VL fails17728.1%
Only InternVL fails18228.9%

Next, during human annotation, the annotators spent about 7 to 11 minutes (median) as discussed in Appendix F. If the ranking can be predicted just looking at the text difference, we expect this duration to be much shorter.

Transcript-only results

We discuss this modality ablation in Section 4.3 and Figure 5 for 3 models. The transcript can provide hints to the model that can help resolve the entities by guessing. We notice 3-6% gap from this result. However, watching video does not guarantee that the model can use that information effectively. We provide error analysis of such cases in Section 4.5.

JudgeTranscript onlyVideo onlyVideo + transcript
Qwen-3.7-Flash61.7 (−5.8)65.7 (−1.8)67.5
Gemini-3.5-Flash-Lite59.0 (−3.1)59.8 (−2.3)62.1
Gemma-4-26B-A4B55.4 (−4.3)56.3 (−3.4)59.7

Finally, in this work, we propose a minimal setup to demonstrate the effectiveness of the components of our framework where the final pairs are determined by multiple models in different complementary stages. It can be extended to support complex workflows, for example, allowing higher threshold of "two" gap-1 discordant pairs (currently one) or adding more models and ensure that all of them fails, and so on. There are possible tradeoffs for those parameters, such as, with higher discordant pair threshold, the benchmark might be biased by used models in that gate. In this paper, we aim to study models of diverse parameter size from 2B to 2T+ parameters, hence we chose a smaller pair sampler of 3B and 8B active parameters. If the goal is to study only large models, such as 100B+ parameters, the pair sampler could be scaled accordingly to decrease the baseline performance.

AuthorsOpenReview ↗

Response to Reviewer e32A (2/2)

Performance on naturally generated answers

(Question 2) Could the authors add a small-scale experiment using answers naturally generated by real video question-answering models? These answers could be used to build human-verified preference pairs, allowing the authors to test whether judge performance on PlaylistBench transfers to naturally occurring errors.

It is infeasible to generate new benchmark during rebuttal time as collecting reliable human annotation requires substantial effort. Here, we compare our results with another related video-language judge benchmark (VideoRewardBench [1], long-form perception subset). In this dataset, responses are derived from five real video-language models (LLaVA-Video-72B, Qwen2.5-VL-72B and three proprietary models), and the preferred response is set by a majority vote of human annotators.

JudgeVideoRewardBench long-form perceptionOurs
Gemini-3.7-Flash74.475.4
Qwen-3.8-Max74.473.9
GPT-5.6-Terra73.172.5

Moreover, our benchmark results in identical ranking in LVBench, a popular long-video understanding benchmark as discussed in Appendix C.3 of our manuscript.

JudgeLVBenchOurs
Gemini-3.7-Flash85.475.4
Qwen-3.8-Max81.873.9
GPT-5.6-Terra78.972.5

With our experiments and analysis, we conclude that our current framework results in a consistent benchmark with human-curated long-video judge and understanding benchmarks, such as VideoRewardBench and LVBench - similar ranking of top-3 models.

Multi-segment necessity

(Question 3) Could the authors compare three settings: providing only the first clip, only the second clip, and both clips? This would help test how much the final preference judgment actually depends on having both clips. If some answer pairs only require one clip, how would the authors explain the need for cross-clip verification and the related retrieval metrics?

Multi-segment requirement annotation from our pipeline

During QA and distractor generation in both phases, we asked Generator model (Gemini-3-Flash) to output meta-data for the segment labels of the entities in the question and the modifications in the answer. Here is the segment-level breakdown of answer differences in visual evidence.

Answers differ inPairsPercentage
Both segments42968.1%
Segment 2 only11418.1%
Segment 1 only8713.8%

In about 68% of our final pairs, both segments are modified with at least one visual evidence in each by the generator model in Phase 2.

Downstream effect of multi-segment presence

We also provide oracle performance of Qwen-3.7-Flash model in the oracle setup with 10 to 30-minutes segments.

Qwen-3.7-Flash evidenceAccuracy
Gold segment 1 only64.5
Gold segment 2 only67.6
Both gold segments73.6

With both segments the performance is highest in comparison to using only one segment. Therefore, even if answering one question may be possible with only one 10 to 30-minutes segments, during retrieving at 30-second chunks, the judge systems are still highly likely to require multiple 30-second chunks which is challenging.

[1] Zhang, Zhihong, et al. "Videorewardbench: Comprehensive evaluation of multimodal reward models for video understanding." arXiv preprint arXiv:2509.00484 (2025).

Thanks for your reply. Generally I think this is a good paper and I decided to maintain my score.

Reviewer_gP5wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PlaylistEval. Its is a fully automated agentic pipeline for building a long horizon video-language evaluation benchmark. The authors claim the benchmark evaluates reasoning and retrieval of evidence that spans across a ~100 hour Youtube playlist collection in seven domains. Phase I generates questions whose evidence is spread across distant segments, and also generates timestamped gold answers. Phase II generates four wrong answers of graded severity by injecting visual-only errors that can’t be detected from the transcript. Cross-family model gates and a feedback loop reject invalid items. The benchmark in total contains 630 preference pairs, with a 93% agreement with human annotator judgements. They evaluate across 17 judge Video Language models and conclude that they fall far behind humans for retrieval and reasoning across long videos, with the best models reaching only 75.4% against humans agreement of 93%.

Critical strengths
  1. It is scalable because humans are only required to find a playlist; the rest of the pipeline is automated and validated. According to the authors it costs about $1 per question, which is reasonable and about 8 times cheaper than a human. The QA pairs are further refined or rejected by automated cross-family validation and feedback loops.

  2. The preference rating is human validated -- human annotators agreed with the intended preferences in 93.0% of cases (alpha=0.781).

  3. It presents a controlled long-horizon testbed that isolates failure modes.

    • Gold answers are grounded in two verified segments, where the distractors differ only in visual details. This lets the authors vary haystack size, input modality and pair difficulty independently.
    • The judge benchmark to extend beyond hour-long video, and the sweep from the two gold segments up to the full ~100-hour domain shows judge accuracy falling steadily as the collection grows, which short-clip benchmarks cannot reveal.
    • The analysis traces the bottleneck to retrieval rather than perception and shows that judges fine-tuned on short video do not transfer.
  4. These are actionable findings for anyone using judges or reward models on long video.

Critical weaknesses
  1. The benchmark is built and filtered by the same model families that perform the evaluation: *Gemini-3-Flash is used to create all the questions, gold answers and distractors, *Gemini-3.1-Pro is also on the C2 and D3 verification panels, *Qwen models show up nearly everywhere else. etc.

Overall, Gemini and Qwen are heavily used in both the QA synthesis and distractor synthesis. So it is natural that the two best judges are Gemini-3.7-Flash and Qwen-3.8-Max.

  1. The difficulty filter also removes pairs that Qwen3-VL and InternVL solve, which seems to work against the evaluated Qwen-3.5 models, so I’m not convinced by the claim that it doesn’t bias the benchmark toward any evaluated model. The authors note generator bias but give no ablation, such as distractors from another family or results by family. Since the main output is a ranking of judges, this makes the obtained ranking dubious.

  2. The claim that the video is necessary isn’t actually checked for the distractors. The paper says the wrong answers can’t be told apart from the gold “on the transcript alone”. But the D3 gate gives its judges only the question and the answers, with no transcript. It also passes an item as soon as one of the two judges fails to recover the full 5-way order, which is a low bar for a pairwise benchmark. The results highlight the anomaly : removing frames costs judges only 2–7 points, so transcript-only judges stay well above chance on pairs that are supposed to be visual-only. A pairwise transcript-only baseline would settle this. Without one, the central grounding claim isn’t established.

Questions for authors
  1. Can you report pairwise accuracy for a transcript-only judge (no frames) on all 630 pairs? The D3 gate never shows the judges the transcript, so I’d also like to know how many pairs a text-only model gets right when it is given the transcript. If transcript-only accuracy is well above chance, the claim that the benchmark requires the video doesn’t hold.

  2. Would the rankings hold if the distractors came from a non-Gemini model, or if the D4 and difficulty gates used something other than Qwen? Even a small re-generated subset would help show whether Gemini-3.7-Flash and Qwen-3.8-Max are on top because they’re better judges or because the items were built by their own model families.

  3. The human annotators didn’t search the playlist: they were shown short clips picked by Qwen-3.7-Plus, with the differences between answers underlined. Models instead had to find the evidence across ~100 hours. Wouldn’t it be fairer to compare the 93.0% human figure with model accuracy when models are also given the correct segments (the oracle setting in Fig. 4)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gP5w (1/3)

We are grateful for your insightful questions and suggestions. We address your questions and weaknesses below:

Bias caused by Gemini and Qwen Families

(Weakness 1) The benchmark is built and filtered by the same model families that perform the evaluation: *Gemini-3-Flash is used to create all the questions, gold answers and distractors, *Gemini-3.1-Pro is also on the C2 and D3 verification panels, *Qwen models show up nearly everywhere else. etc. Overall, Gemini and Qwen are heavily used in both the QA synthesis and distractor synthesis. So it is natural that the two best judges are Gemini-3.7-Flash and Qwen-3.8-Max.

(Question 2) Would the rankings hold if the distractors came from a non-Gemini model, or if the D4 and difficulty gates used something other than Qwen? Even a small re-generated subset would help show whether Gemini-3.7-Flash and Qwen-3.8-Max are on top because they’re better judges or because the items were built by their own model families.

In this paper, we aim to push the limit of video-language judge models to 100-hour playlists. We have some requirements to achieve this. First, the generator and verifier model should have the ability to process up to 1 hour video reliably and generate long-form outputs without major hallucination. Second, these models have to be cheap to achieve the first goal for 100-hour video databases. We provide LVBench scores from the Qwen-3.7-Plus blog [1] and LongVideoBench scores for Gemini-3-Pro and Gemini-3-Flash from ByteDance's Seed2.0 model page [2], as we found no published LVBench score for Gemini-3-Flash.

ModelLVBenchInput / output ($ per 1M tokens)
GPT-5.4 (xhigh)77.42.50 / 15.00
Qwen-3.7-Plus76.20.40 / 1.60
Gemini-3.1-Pro75.12.00 / 12.00
Qwen-3.6-Plus74.80.50 / 3.00
Claude Opus 4.6 (Max)63.05.00 / 25.00
ModelLongVideoBenchInput / output ($ per 1M tokens)
Gemini 3 Pro (High)78.22.00 / 12.00
Gemini 3 Flash (High)74.50.50 / 3.00

Popular models like GPT and Claude model have image limitations of 1500 and 600 frames respectively. Moreover, we could not find any official public score of long-video understanding video benchmark for these models provided by OpenAI and Anthropic. While pilot testing for this work, we find GPT-5.4-mini struggle to understand 10-minute of video and generate/verify coherent questions and answers without hallucination. Therefore, we choose Gemini-3-Flash and Qwen-3.7-Plus for coherent video understanding capability at a manageable price. Appendix A (Generator bias) acknowledges this limitation.

We agree that regenerating the benchmark with other model families would be the most direct test. However, their long-video understanding may not yet be adequate for generating and verifying such data at a feasible and scalable cost, as discussed above. Instead, we compare our results with an independently built benchmark (VideoRewardBench [3], long-form perception subset), where preference labels come from human annotators and the answers are naturally generated by different models.

As this benchmark does not go through our generation and filtering pipeline, any advantage that Gemini and Qwen judges gain from our pipeline should not carry over to it. If the ranking on our benchmark were driven by such an advantage, we would expect a different trend there.

JudgeVideoRewardBench long-form perceptionOurs
Gemini-3.7-Flash74.475.4
Qwen-3.8-Max74.473.9
GPT-5.6-Terra73.172.5

Moreover, our benchmark results in identical ranking in LVBench, a popular long-video understanding benchmark as discussed in Appendix C.3 of our manuscript.

JudgeLVBenchOurs
Gemini-3.7-Flash85.475.4
Qwen-3.8-Max81.873.9
GPT-5.6-Terra78.972.5

With our experiments and analysis, we conclude that our current framework results in a consistent benchmark with human-curated long-video judge and understanding benchmarks, such as VideoRewardBench and LVBench - similar ranking of top-3 models.

AuthorsOpenReview ↗

Response to Reviewer gP5w (2/3)

Transcript-only accuracy

(Question 1) Can you report pairwise accuracy for a transcript-only judge (no frames) on all 630 pairs? The D3 gate never shows the judges the transcript, so I’d also like to know how many pairs a text-only model gets right when it is given the transcript. If transcript-only accuracy is well above chance, the claim that the benchmark requires the video doesn’t hold.

We discuss this modality ablation in Section 4.3 and Figure 5 for 3 models.

JudgeTranscript onlyVideo onlyVideo + transcript
Qwen-3.7-Flash61.7 (−5.8)65.7 (−1.8)67.5
Gemini-3.5-Flash-Lite59.0 (−3.1)59.8 (−2.3)62.1
Gemma-4-26B-A4B55.4 (−4.3)56.3 (−3.4)59.7

Transcript-only accuracy is above chance (55.4-61.7), which is expected as the transcript is part of each video and carries some of its context. Adding the video still raises every judge, by 3.1-5.8 points. The transcript route is also screened for every question in Phase I where Gemini-3-Flash answers from the transcript alone, shuffled among windows from another video of the same playlist, and GPT-5.4-mini rejects the question if that answer matches the gold (Section 3.1). However, watching video does not guarantee that the model can use that visual information effectively. We provide error analysis of such cases in Section 4.5.

Difficulty filter bias

(Weakness 2) The difficulty filter also removes pairs that Qwen3-VL and InternVL solve, which seems to work against the evaluated Qwen-3.5 models, so I’m not convinced by the claim that it doesn’t bias the benchmark toward any evaluated model. The authors note generator bias but give no ablation, such as distractors from another family or results by family. Since the main output is a ranking of judges, this makes the obtained ranking dubious.

During final pair sampling as described in Section 3.3 ("Controllable Pair Selection"), we employ two small models with 3B and 8B active parameters and ensure at least one of them fails for the sampled pairs. These models are also released relatively earlier than the models evaluated in our paper.

Reference judgesPairsPercentage
Both fail27143.0%
Only Qwen3-VL fails17728.1%
Only InternVL fails18228.9%

Here is the performance breakdown of two open-weights families by these cases in the following. If the Qwen3-VL filter hurt Qwen-3.5, Qwen-3.5 would do worse on the pairs only Qwen3-VL failed. It does the opposite; on those pairs, every Qwen-3.5 model scores above its overall accuracy (70.6, 58.2 and 69.5 vs 51.6, 56.2 and 56.7), and so does every Gemma model. Hence, we find no conclusive evidence that filtering on Qwen3-VL hurts the Qwen-3.5 family specifically.

JudgeAll pairs (630)Both fail (271)Only Qwen3-VL fails (177)Only InternVL fails (182)
Qwen-3.5-2B51.645.470.642.3
Qwen-3.5-4B56.248.358.265.9
Qwen-3.5-9B56.746.969.558.8
Gemma-4-E2B46.043.248.048.4
Gemma-4-E4B44.335.857.144.5
Gemma-4-26B-A4B59.752.067.863.2

Moreover, Qwen3-VL [4] and Qwen3.5 [5] differ substantially, such as in terms of architecture and tokenizer. Qwen3-VL uses a Qwen3 decoder with full attention in every layer. Qwen3.5 is a hybrid: three Gated DeltaNet (linear attention) layers for every gated full-attention layer. Qwen3.5 tokenizer is retrained, with a 248K vocabulary against 152K of Qwen3-VL.

Thus, we conclude using Qwen3-VL is distant enough for pair selection without measurably penalizing Qwen3.5 series.

AuthorsOpenReview ↗

Response to Reviewer gP5w (3/3)

Model performance in human annotation setup

(Question 3) The human annotators didn’t search the playlist: they were shown short clips picked by Qwen-3.7-Plus, with the differences between answers underlined. Models instead had to find the evidence across ~100 hours. Wouldn’t it be fairer to compare the 93.0% human figure with model accuracy when models are also given the correct segments (the oracle setting in Fig. 4)?

Here are the results in matched conditions for human-validated 147 pairs in oracle evidence-moments settings:

JudgeAgreement
Qwen-3.7-Flash90.8
Gemini-3.5-Flash-Lite90.5
Gemma-4-26B-A4B86.1
Human annotators95.4

The performance of the Judge model is within 10% of human agreement when the gold evidence moments are provided. The reason is our benchmark covers long-form videos where the number of entities and events changes less rapidly within the short span. Thus, the model can decode the masked entity in the question from the transcript and/or video and provide the correct judgement. However, this becomes increasingly difficult as the database size grows to 1 hour or 1 day.

We derived the ground truth evidence to keep the human annotation feasible. The human agreement reported in the paper (93.0% over all 152 pairs; 95.4% per rating on these 147) was therefore measured with the evidence in hand, so we use it as a reference point for judges rather than as a matched baseline. We will clarify this in the Introduction and Results sections.

Our framework is intended to be multi-skill similar to any large database multi-hop QA benchmark, such as HotpotQA[6] over Wikipedia database. Even if sampling is perfect the models still can suffer long-context degradation at a few-minutes to hour-level. For example, in LongVideoBench[7] Table 5, the performance is far from perfect accuracy even with 256 frames for few minutes videos for strong model families like GPT-4o.

In our paper, as mentioned in Section 4.4 and Appendix C.2, even with agentic tool use in gold segments of 10 to 30-minutes, the performance gain is only about 2 points with 3 steps (median) of search and reasoning. The number of steps required for the agentic model is given below (median = 3 steps):

Tool-use stepsPairsPercentage
0-1355.6%
220833.0%
3-429246.3%
5 or more9515.1%

The goal of our benchmark is to go beyond a few-minutes level of video-language judgement. Though the deciding moments may be shorter than 1 hour in total, they are scattered across the 100-hour database. Therefore, our framework poses a unique challenge that is absent in previous short-video judge benchmarks.

[1] "Qwen3.7-Plus: Multimodal Agent Intelligence." Qwen, 21 May 2026, https://qwen.ai/blog?id=qwen3.7-plus. Accessed 15 Nov. 2026.

[2] "Seed2.0." ByteDance Seed, https://seed.bytedance.com/en/seed2. Accessed 15 Nov. 2026.

[3] Zhang, Zhihong, et al. "Videorewardbench: Comprehensive evaluation of multimodal reward models for video understanding." arXiv preprint arXiv:2509.00484 (2025).

[4] Bai, Shuai, et al. "Qwen3-vl technical report." arXiv preprint arXiv:2511.21631 (2025).

[5] "Qwen3.5: Towards Native Multimodal Agents." Qwen, 14 Feb. 2026, https://qwen.ai/blog?id=qwen3.5. Accessed 15 Nov. 2026.

[6] Yang, Zhilin, et al. "HotpotQA: A dataset for diverse, explainable multi-hop question answering." Proceedings of the 2018 conference on empirical methods in natural language processing. 2018.

[7] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." Advances in Neural Information Processing Systems 37 (2024): 28828-28857.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses

Dear Area Chair,

We appreciate your time, and we thank all the reviewers again for their insightful questions and thoughtful suggestions.

The reviewers found the problem “important and relevant to the research community” (Reviewer e32A) and the answer construction “reasonably controlled” (Reviewer LB4U), and described the results as “actionable findings for anyone using judges or reward models on long video” (Reviewer gP5w) and an “informative diagnosis beyond a model leaderboard” (Reviewer YQHD).

Here we summarize the reviewers' comments and our corresponding responses:

Ordering verification and text-only solvability of pairs. (Reviewers LB4U, gP5w, e32A, YQHD) In the structural check in Phase 2, our pipeline verifies there is at least one visual-only change for each of all 4 generated answers. Again, during final pair sampling as described in Section 3.3 ("Controllable Pair Selection"), we utilize two small models with 3B and 8B active parameters and ensure at least one of them fails for the sampled pairs. In Question-and-answers only setup, Gemini-3.7-Flash scores 61.0 on our pairs vs 70.3 on VideoRewardBench (48.4 vs 67.5 when it must be right in both answer orders). Moreover, Qwen-3.5-9B scores only 50.5% while it can achieve 65.0% in VideoRewardBench. In Transcript only setups, adding video raises three judges by 3.1-5.8 points. Both difficulty-gate models fail 43% of pairs. Reviewer YQHD acknowledged their concerns to be resolved and maintained positive rating.

Multi-segment necessity in final benchmark. (Reviewers LB4U, e32A) In our pipeline, we have the segment-level annotation while generating 4 ratings in Phase 2. 68.1% of pairs differ in both segments. In oracle 10-30 minute segments, Qwen-3.7-Flash achieves 64.5 / 67.6 in one segment scenario and 73.6 when both segments provided. Reviewer e32A acknowledged our response and maintained positive rating.

Human vs model under the same evidence. (Reviewers LB4U, gP5w, YQHD) On the 147 validated pairs with the same evidence moments, three judges score 86.1-90.8 against 95.4 for humans. With whole gold segments they score 62.9-73.6 on all the 630 pairs, and with retrieval 59.5-67.5. The gap gradually increase with the increasing size of the retrieval corpus of 100 hours. Reviewer YQHD acknowledged their concerns to be resolved and maintained positive rating.

Answers written by real models. (Reviewer e32A) We utilize VideoRewardBench’s long-form subset where answers are from five real models and labelled by human vote. Top three models in our benchmark demonstrates similar ranking. Moreover, our benchmark results in identical ranking in LVBench, a popular long-video understanding benchmark as discussed in Appendix C.3. Reviewer e32A acknowledged our response and maintained positive rating.

Bias caused by Gemini/Qwen families. (Reviewer gP5w) As mentioned in the paragraph above, top three models achieve similar ranking in two independent and human-curated benchmarks - VideoRewardBench and LVBench. Again, the Qwen3-VL difficulty filter does not penalise Qwen-3.5 - on the pairs only Qwen3-VL failed, each Qwen-3.5 model scores above its overall accuracy.

Changes in the revised paper. All changes are marked in blue.

  • Appendix C.7 to C.10 add the new analyses: question and answers only and transcript only (C.7), one or both gold segments (C.8), judges and humans on the same evidence clips (C.9), and the comparison with VideoRewardBench (C.10).
  • Appendix C.11 explains how the structural check and the pair sampler cover the pairs that the text-only ranking lets through, and why the two samplers are small.
  • Appendix A (Generator bias) now explains why we chose Gemini-3-Flash and Qwen-3.7-Plus: the GPT and Claude APIs cap a request at 1,500 and 600 images, and Table 4 compares long-video scores and prices. It also reports the check against LVBench and VideoRewardBench.
  • The Introduction and Section 4.1 now state that the annotators judged from the evidence clips, and point to the matched comparison in Appendix C.9.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.