- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces PlaylistEval, an automated framework for constructing benchmarks that evaluate video-language judges over large video collections. It generates questions grounded in two video segments, creates graded answer variants with controlled visual errors, and applies verification gates and regeneration feedback to improve data quality. The resulting PlaylistBench contains 630 preference pairs across seven domains, each with approximately 100 hours of source videos. Evaluation covers 17 judges and four retrievers, with analyses of collection size, input modalities, answer position, and inference budgets. Human validation supports the reliability of the generated preferences, while the model results reveal substantial limitations in retrieving and evaluating the relevant evidence.
Critical strengths
- A useful benchmark design with substantial attention to data quality. Evaluating judges over video collections of this scale is a meaningful extension of short-video preference evaluation. The combination of evidence-grounded questions, graded visual perturbations, verification across model families, and regeneration feedback provides a practical way to construct challenging examples.
- The experiments provide informative diagnosis beyond a model leaderboard. The paper evaluates a broad range of judges and retrievers and systematically varies retrieval scope, modalities, answer order, reasoning budget, and visual input budget. The consistent decline as the search collection grows, together with the error analysis distinguishing retrieval, frame sampling, perception, and reasoning failures, helps explain where current systems struggle. This makes the benchmark useful for guiding improvements in evidence retrieval and judging.
Critical weaknesses
Human annotators receive selected evidence clips and optional highlighting of answer differences (Appendices F and G.3), whereas the main model evaluation uses retrieved or uniformly sampled evidence. The human results are strong evidence of label quality, but the numerical gap should not be interpreted as a comparison under identical evidence conditions.
Questions for authors
-
Are model results available on the same 152 human-evaluated pairs using the same selected evidence clips, to help interpret the reported human-model gap?
-
Do the existing text-only ranking outputs indicate how often the final selected pairs can be ordered correctly without video, even when the full five-answer ordering is incorrect?
Please address these questions in the rebuttal.
4 comments
Response to Reviewer YQHD (2/3)
Ordering verification and Text-only solvability of pairs
(Question 2) Do the existing text-only ranking outputs indicate how often the final selected pairs can be ordered correctly without video, even when the full five-answer ordering is incorrect?
We use the strongest model of the family for text-only ranking prediction in phase 2: Gemini-3.1-Pro and GPT-5.4. This ordering capability also includes the parametric knowledge of the models. As our dataset includes factual domains like History and Art, even without text-only shortcuts, one model can predict order partially, while other can fail. Again, as the parametric knowledge of different models are different within the family and sub-family, it is practically challenging to eliminate this effect completely without biasing the benchmark against particular families.
During question generation in Phase 1, we verified that the two entities in questions are difficult to infer without watching the video (Example in Figure 1). Again, during distractor generation in Phase 2, we prompted the Generator model (Gemini-3-Flash) to output meta-data for its pieces of modification. Moreover, in the structural check in Phase 2, our pipeline verifies there is at least one visual-only change for each of all 4 generated answers. If the generator model has strong enough video understanding capability, the chances of text-only shortcut can be prevented even before going to the ranking prediction stage.
Moreover, during final pair sampling as described in Section 3.3 ("Controllable Pair Selection"), we employ two small models with 3B and 8B active parameters and ensure at least one of them fails for the sampled pairs. If the pairs are trivially solvable only by looking at the text and transcript, we expect them to be solvable by these small models.
| Reference judges | Pairs | Percentage |
|---|---|---|
| Both fail | 271 | 43.0% |
| Only Qwen3-VL fails | 177 | 28.1% |
| Only InternVL fails | 182 | 28.9% |
Finally, during human annotation, the annotators spent about 7 to 11 minutes (median) as discussed in Appendix F. If the ranking can be predicted just looking at the text difference, we expect this duration to be much shorter.
Response to Reviewer YQHD (3/3)
Question and answer-only results
| Judge | VideoRewardBench long, original order | Ours, original order | VideoRewardBench long, right in both orders | Ours, right in both orders | Gap in both orders | Same letter in both orders, VideoRewardBench / ours |
|---|---|---|---|---|---|---|
| Gemini-3.7-Flash | 70.3 | 61.0 | 67.5 | 48.4 | -19.1 | 7.8% / 29.5% |
| Qwen-3.8-Max | 63.1 | 59.4 | 49.1 | 43.5 | -5.6 | 25.8% / 33.2% |
| Qwen-3.5-9B | 65.0 | 50.5 | 49.1 | 34.8 | -14.3 | 31.1% / 37.3% |
| GLM-4.6V-Flash | 67.8 | 49.4 | 53.0 | 16.8 | -36.2 | 32.2% / 73.0% |
| Gemma-4-26B-A4B | 71.4 | 50.2 | 65.0 | 33.7 | -31.3 | 10.2% / 38.4% |
Here, we evaluate top 2 API models, top 2 open-weights models and one additional model (GLM-4.6V-Flash) in question and answer-only setup. We compare our results with another video-language judge benchmark (VideoRewardBench [3], long-form perception subset). Our benchmark is less prone to both parametric and text shortcuts. For example, Qwen-3.5-9B scores only 50.5% while it can achieve 65.0% in VideoRewardBench. For large models, Gemini-3.7-Flash reaches 70.3% in VideoRewardBench, while it is only 61.0% in our case. Moreover, when the model must be right in both orders, the accuracy drops by 12.6% while it only drops about 2.8% for VideoRewardBench in both orders. Therefore, our benchmark mitigates text-only and parametric shortcuts within the current setup in the paper.
Again, educated guessing in text-only setup does not imply strong performance in the final benchmark automatically. The final performance of top 3 models is less than 76% in both benchmarks as given below.
| Judge | VideoRewardBench long-form perception | Ours |
|---|---|---|
| Gemini-3.7-Flash | 74.4 | 75.4 |
| Qwen-3.8-Max | 74.4 | 73.9 |
| GPT-5.6-Terra | 73.1 | 72.5 |
Moreover, our benchmark results in identical ranking in LVBench, a popular long-video understanding benchmark as discussed in Appendix C.3 of our manuscript.
| Judge | LVBench | Ours |
|---|---|---|
| Gemini-3.7-Flash | 85.4 | 75.4 |
| Qwen-3.8-Max | 81.8 | 73.9 |
| GPT-5.6-Terra | 78.9 | 72.5 |
Finally, in this work, we propose a minimal setup to demonstrate the effectiveness of the components of our framework where the final pairs are determined by multiple models in different complementary stages. It can be extended to support complex workflows, for example, allowing higher threshold of "two" gap-1 discordant pairs (currently one) or adding more models and ensure that all of them fails, and so on. There are possible tradeoffs for those parameters, such as, with higher discordant pair threshold, the benchmark might be biased by used models in that gate.
In this paper, we aim to study models of diverse parameter size from 2B to 2T+ parameters, hence we chose a smaller pair sampler of 3B and 8B active parameters. If the goal is to study only large models, such as 100B+ parameters, the pair sampler could be scaled accordingly to decrease the baseline performance.
With our experiments and analysis, we conclude that our current framework results in a consistent benchmark with human-curated ones, like VideoRewardBench and LVBench - similar ranking of top-3 models and lower text-only score.
[1] Yang, Zhilin, et al. "HotpotQA: A dataset for diverse, explainable multi-hop question answering." Proceedings of the 2018 conference on empirical methods in natural language processing. 2018.
[2] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." Advances in Neural Information Processing Systems 37 (2024): 28828-28857.
[3] Zhang, Zhihong, et al. "Videorewardbench: Comprehensive evaluation of multimodal reward models for video understanding." arXiv preprint arXiv:2509.00484 (2025).
The authors have addressed my concerns, I'll keep my rating.
Response to Reviewer YQHD (1/3)
We appreciate your strong support. Thank you so much for recognizing our benchmark design and its attention to data quality and diagnostic analysis.
We address your questions and weaknesses below:
Evidence-matched comparison of model performance
Here are the results in matched conditions for human-validated 147 pairs (5 were overturned by the annotators out of 152) in oracle evidence-moments settings:
The performance of the Judge model is within 10% of human agreement when the gold evidence moments are provided. The reason is our benchmark covers long-form videos where the number of entities and events changes less rapidly within the short span. Thus, the model can decode the masked entity in the question from the transcript and/or video and provide the correct judgement. However, this becomes increasingly difficult as the database size grows to 1 hour or 1 day.
We derived the ground truth evidence to keep the human annotation feasible. The human agreement reported in the paper (93.0% over all 152 pairs; 95.4% per rating on these 147) was therefore measured with the evidence in hand, so we use it as a reference point for judges rather than as a matched baseline. We will clarify this in the Introduction and Results sections.
Our framework is intended to be multi-skill similar to any large database multi-hop QA benchmark, such as HotpotQA [1] over Wikipedia database. Even if sampling is perfect the models still can suffer long-context degradation at a few-minutes to hour-level. For example, in LongVideoBench [2] Table 5, the performance is far from perfect accuracy even with 256 frames for few minutes videos for strong model families like GPT-4o.
In our paper, as mentioned in Section 4.4 and Appendix C.2, even with agentic tool use in gold segments of 10 to 30-minutes where the agent collects its own frames, the performance gain is only about 2 points with 3 steps (median) of search and reasoning. The number of steps required for the agentic model is given below (median = 3 steps):
The goal of our benchmark is to go beyond a few-minutes level of video-language judgement. Though the deciding moments may be shorter than 1 hour in total, they are scattered across the 100-hour database. Therefore, our framework poses a unique challenge that is absent in previous short-video judge benchmarks.