- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Meta-Video, a compact video understanding benchmark constructed by filtering and sampling questions from five existing benchmarks. The authors also investigate how different evaluation configurations affect model performance and propose a standardized evaluation protocol. Overall, the main contribution is an efficiency-oriented subset of existing benchmarks, together with an empirical analysis of evaluation settings, rather than fundamentally new evaluation tasks or methodologies.
Critical strengths
-
The paper highlights the sensitivity of video understanding benchmark results to evaluation configurations, an important issue for reproducibility and fair model comparison.
-
The analysis covers several practical evaluation factors, including frame sampling, video decoding, inference backends, input modalities, and prompting. These observations may serve as useful references for researchers conducting video understanding evaluations.
Critical weaknesses
-
The manuscript contains numerous grammatical errors, unclear expressions, and incomplete sentences that significantly affect readability. The Conclusion (Lines 477–485) is particularly problematic, with sentences such as "We propose Meta-Video, a compact video understanding benchmarking which is efficient, and version of the source benchmarks." Similar issues appear throughout the paper, along with unresolved references and placeholders, including "Table ref" (Line 324), "yy points lower" (Line 910), and "Table ??" (Line 946). These are not isolated typos but reflect serious problems with the overall writing quality. The manuscript requires substantial revision and does not meet the presentation standards expected at ICLR.
-
Meta-Video is essentially a smaller subset constructed by filtering and sampling questions from five existing benchmarks. While improving evaluation efficiency is useful, the methodological novelty is limited. Moreover, the paper lacks comparisons with random or stratified sampling baselines to demonstrate the advantages of its construction strategy. The claim that Meta-Video preserves model rankings is not supported by rigorous statistical analysis, and ranking reversals can be observed in Table 4. Additionally, the blind filtering strategy may introduce model-dependent selection bias, but its effectiveness is not independently validated through human annotations or evaluations on unseen model families. These limitations weaken the paper's main contributions and conclusions.
Questions for authors
Q1. The authors claim that Meta-Video preserves model rankings, but Table 4 shows inconsistent results. For example, with 4 input frames, InternVL3.5-8B outperforms Qwen3-VL-8B on the average of the seven benchmarks (54.5% vs. 50.5%), whereas their ranking is reversed on Meta-Video (35.0% vs. 35.9%). Could the authors explain this inconsistency and provide stronger evidence to support the claim that Meta-Video reliably preserves model rankings?
Q2. Given that Meta-Video is primarily constructed by filtering and sampling existing benchmark questions, what is its main advantage over a randomly sampled subset with the same size and task distribution? A comparison against random and stratified sampling baselines would help establish the necessity of the proposed construction method.
Q3. How do the authors verify that questions removed by blind filtering genuinely do not require visual information, rather than being answered correctly through language priors or guessing? Could the authors provide human validation or evaluate the resulting benchmark on model families not used during filtering?
Q4. The manuscript reports different numbers of unique videos (791 in the Abstract versus 788 in Section 3.2), and contains several unresolved references and numerical placeholders. Could the authors clarify these inconsistencies and confirm that all reported statistics and experimental results have been checked?
4 comments
Dear Reviewer,
Thank you once again for your constructive feedback. As we approach the end of the discussion period, we wanted to reiterate that we have provided clarification on the typos, on construction of Meta-Video and we have provided comparison with randomly sampled subsets. We have also provided human check used in the construction of Meta-Video, along with experiments for generalization on new model family: GLM-4.6V-Flash.
With less than 24 hours remaining in the discussion period, we would greatly appreciate your feedback on our rebuttal and additional experiments. Please let us know whether our responses affect your assessment or if any concerns remain unresolved, so that we can provide further clarification before the deadline. We believe these efforts address the concerns raised in your original reviews and have improved the paper.
Thank you again for your time and consideration. Even a brief update would be very helpful.
Sincerely,
The Authors
Thank you for the rebuttal and additional experiments. While I appreciate the authors' efforts, my concerns regarding the limited novelty, insufficient benchmark validation, and overall writing quality remain. The manuscript contains substantial grammatical errors, unclear expressions, and poorly organized discussions that require more than minor proofreading. Therefore, I will maintain my original rating.
Re: Official Comment by Reviewer q9uY
Re: Official Comment by Reviewer q9uY
We are happy to correct the four (!) glitches the reviewer pointed out for the camera-ready version. Just to reiterate, those were:
- "Conclusion (Lines 477–485) is particularly problematic, with sentences such as "We propose Meta-Video, a compact video understanding benchmarking which is efficient, and version of the source benchmarks."
- "Table ref" (Line 324)
- "yy points lower" (Line 910)
- "Table ??" (Line 946)
Thanks, actually, for reading the paper so carefully up to the end, and even pointing to a missing value and a missing reference in the supplementary on pages 17 and 18 (Line 910 and Line 946) .
We didn't comment on that because we took it for granted that we will resolve all those issues for the camera-ready version. We don't think that those issues justify rejection.
We also want to point out that the paper was written by real humans, so yes, humans make mistakes. We will obviously fix those, but we don't see how this would devalue the paper's contribution, especially given that we seem to have addressed all the questions listed in the initial review.
Response to reviewer q9uY
We thank the reviewer for their constructive feedback and positive assessment. Specifically, the paper highlights the sensitivity of video understanding benchmark results to evaluation configs, which is an important issue for reproducibility and fair model comparison, and that our analysis covers several practical evaluation factors which serve as useful references for researchers conducting video understanding evaluations.
The following are answers to the questions. We hope our responses below address your questions.
Q1. We will refine our statement regarding model preservation to largely preserve rankings, and also report rank correlations. As in section 5.1, models differ based on which frame budget suits them, so a fair comparison evaluates each model at its best budget. Meta-Video selects the same optimal budget as the average of benchmarks for most of the models, except InternVL3.5-8b where it differs by only 0.2 points. Meta-Video is therefore not intended to reproduce the average exactly, but it aims to give a cheaper signal that agrees with similar conclusions.
Q2. We compared Meta-Video against three 1,000-question baselines, each averaged over 500 draws: (i) random sampling, and (ii) stratified sampling matching Meta-Video’s source and duration distribution. Our main advantage is that every question requires visual input. About 1/3rd of a matched stratified subset is answerable without the video (stratified: 31.9 ± 1.3% and random: 33.4 ± 1.5%), and nearly 30% of evaluation compute is spent on these questions. For Meta-Video, it contains none. Therefore, the accuracy gain from seeing the video is 36.4 points on Meta-Video vs. 21.1 ± 1.1 for stratified and 19.4 ± 1.8 for random subsets. Cost is similar for all subsets of the same size (~4,700–4,800 s). The speedup comes from subset size, and our curation ensures this compute budget is spent only on questions that measure video understanding. We will add this experiment to table 5.
Q3. We want to clarify that blind question filtering means that questions that require no visual information and also questions for which the model answers correctly through language priors is also intended. A question answerable from its text alone (through world knowledge, common sense or option cues) does not measure video understanding even if the video is also informative.
Furthermore, to validate this, in a text-only human review, we ask an annotator to look at 100 questions (75 - blind filtered, 25 - retained in meta-video). Based on this study, 96% (24/25) of Meta-Video questions were judged to require the video. For the filtered-out questions, 64% were answerable at least without videos or through general knowledge or cues in the question or options, or language priors. The remaining 36% were judged to need the video, which reflects that the filter is conservative: it may discard some video-dependent questions, but it does not admit blind-answerable ones. This indicates that most of the questions filtered did not measure the video understanding aspect of the benchmark. We will add this study to the appendix.
To test whether Meta-Video generalizes beyond these families, we evaluated GLM-4.6V-Flash, another model family that was not used for blind filtering, at 4–64 frames under the same protocol:
GLM follows the same trend as the other models: accuracy increases with frame count and saturates around 16-32 frames (Table 4). Its scores are similar and are within the range of the evaluated models (e.g., 51.4 at 64 frames vs. 39.5–57.4 for the other five models). This shows that Meta-Video gives a meaningful signal for a family it was not built from. We will add GLM results on the full source benchmarks and a larger-scale model in the revision.
Q4. We apologize for this. The two video counts refer to different things. Meta-Video contains 788 unique video files. A few videos appear in more than one source benchmark with different questions, and counting each (source, video) pair separately gives 791 videos. We will use one definition consistently (788 unique videos) and explain the overlap in Sec. 3.2. We clarify and reassure that all the reported statistics are checked and will be refined in the final version. We will also resolve the references and placeholders.