acceptodds
Under review as a conference paper at ICLR 2027

Meta-Video: Efficient Standardized Evaluation for Video Understanding

Abstract

Video understanding benchmarks are the primary yardstick for measuring progress in video-language modeling, and a growing body of work relies on them to claim improvements. The reported benchmark scores in these works, however, are almost always produced by frameworks (e.g., lmms eval) used as black boxes: certain evaluation choices like frame sampling, preprocessing, and input formatting are inherited from preset defaults and seldom inspected carefully. We show these unexamined choices are a major source of variance. Even for the same model checkpoint, evaluation configuration alone can shift accuracy by up to 19.5pp. Hence, it is unclear whether reported gains reflect genuine improvements in capability or differences in evaluation setup. Moreover, many benchmark samples offer little evidence of visual capability, since they can be answered without the video, and the full suites are expensive to evaluate. We address these issues in two parts. First, we curate existing suites for discriminability and efficiency, producing Meta-Video, a balanced 1,000-question benchmark over 791 videos spanning diverse tasks and durations, which preserves model rankings, at 5× inference speedup. Second, we establish a standardized evaluation protocol by sweeping the configuration space to isolate choices that measurably affect results, distill them into best practices and reporting standards, and release reproducible reference scores across five models and seven benchmarks. We hope that the proposed evaluation framework and the Meta-Video benchmark will support video understanding research by providing a reproducible and efficient foundation to measure progress in a reliable way.

Primary area:
datasets and benchmarks
Keywords:
Video Understanding, MLLM, Evaluations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q9uYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Meta-Video, a compact video understanding benchmark constructed by filtering and sampling questions from five existing benchmarks. The authors also investigate how different evaluation configurations affect model performance and propose a standardized evaluation protocol. Overall, the main contribution is an efficiency-oriented subset of existing benchmarks, together with an empirical analysis of evaluation settings, rather than fundamentally new evaluation tasks or methodologies.

Critical strengths
  1. The paper highlights the sensitivity of video understanding benchmark results to evaluation configurations, an important issue for reproducibility and fair model comparison.

  2. The analysis covers several practical evaluation factors, including frame sampling, video decoding, inference backends, input modalities, and prompting. These observations may serve as useful references for researchers conducting video understanding evaluations.

Critical weaknesses
  1. The manuscript contains numerous grammatical errors, unclear expressions, and incomplete sentences that significantly affect readability. The Conclusion (Lines 477–485) is particularly problematic, with sentences such as "We propose Meta-Video, a compact video understanding benchmarking which is efficient, and version of the source benchmarks." Similar issues appear throughout the paper, along with unresolved references and placeholders, including "Table ref" (Line 324), "yy points lower" (Line 910), and "Table ??" (Line 946). These are not isolated typos but reflect serious problems with the overall writing quality. The manuscript requires substantial revision and does not meet the presentation standards expected at ICLR.

  2. Meta-Video is essentially a smaller subset constructed by filtering and sampling questions from five existing benchmarks. While improving evaluation efficiency is useful, the methodological novelty is limited. Moreover, the paper lacks comparisons with random or stratified sampling baselines to demonstrate the advantages of its construction strategy. The claim that Meta-Video preserves model rankings is not supported by rigorous statistical analysis, and ranking reversals can be observed in Table 4. Additionally, the blind filtering strategy may introduce model-dependent selection bias, but its effectiveness is not independently validated through human annotations or evaluations on unseen model families. These limitations weaken the paper's main contributions and conclusions.

Questions for authors

Q1. The authors claim that Meta-Video preserves model rankings, but Table 4 shows inconsistent results. For example, with 4 input frames, InternVL3.5-8B outperforms Qwen3-VL-8B on the average of the seven benchmarks (54.5% vs. 50.5%), whereas their ranking is reversed on Meta-Video (35.0% vs. 35.9%). Could the authors explain this inconsistency and provide stronger evidence to support the claim that Meta-Video reliably preserves model rankings?

Q2. Given that Meta-Video is primarily constructed by filtering and sampling existing benchmark questions, what is its main advantage over a randomly sampled subset with the same size and task distribution? A comparison against random and stratified sampling baselines would help establish the necessity of the proposed construction method.

Q3. How do the authors verify that questions removed by blind filtering genuinely do not require visual information, rather than being answered correctly through language priors or guessing? Could the authors provide human validation or evaluate the resulting benchmark on model families not used during filtering?

Q4. The manuscript reports different numbers of unique videos (791 in the Abstract versus 788 in Section 3.2), and contains several unresolved references and numerical placeholders. Could the authors clarify these inconsistencies and confirm that all reported statistics and experimental results have been checked?

4 comments
AuthorsOpenReview ↗

Response to reviewer q9uY

We thank the reviewer for their constructive feedback and positive assessment. Specifically, the paper highlights the sensitivity of video understanding benchmark results to evaluation configs, which is an important issue for reproducibility and fair model comparison, and that our analysis covers several practical evaluation factors which serve as useful references for researchers conducting video understanding evaluations.

The following are answers to the questions. We hope our responses below address your questions.

Q1. We will refine our statement regarding model preservation to largely preserve rankings, and also report rank correlations. As in section 5.1, models differ based on which frame budget suits them, so a fair comparison evaluates each model at its best budget. Meta-Video selects the same optimal budget as the average of benchmarks for most of the models, except InternVL3.5-8b where it differs by only 0.2 points. Meta-Video is therefore not intended to reproduce the average exactly, but it aims to give a cheaper signal that agrees with similar conclusions.

Q2. We compared Meta-Video against three 1,000-question baselines, each averaged over 500 draws: (i) random sampling, and (ii) stratified sampling matching Meta-Video’s source and duration distribution. Our main advantage is that every question requires visual input. About 1/3rd of a matched stratified subset is answerable without the video (stratified: 31.9 ± 1.3% and random: 33.4 ± 1.5%), and nearly 30% of evaluation compute is spent on these questions. For Meta-Video, it contains none. Therefore, the accuracy gain from seeing the video is 36.4 points on Meta-Video vs. 21.1 ± 1.1 for stratified and 19.4 ± 1.8 for random subsets. Cost is similar for all subsets of the same size (~4,700–4,800 s). The speedup comes from subset size, and our curation ensures this compute budget is spent only on questions that measure video understanding. We will add this experiment to table 5.

ModelRandomStratified randomMeta-V
InternVL3.5-4B54.5 ± 2.055.1 ± 1.345.5
Qwen3-VL-4B53.4 ± 1.954.5 ± 1.450.0
Qwen2.5-VL-7B46.1 ± 1.946.2 ± 1.234.9
InternVL3.5-8B57.3 ± 2.357.3 ± 1.550.4
Qwen3-VL-8B54.8 ± 2.255.6 ± 1.455.0

Q3. We want to clarify that blind question filtering means that questions that require no visual information and also questions for which the model answers correctly through language priors is also intended. A question answerable from its text alone (through world knowledge, common sense or option cues) does not measure video understanding even if the video is also informative.

Furthermore, to validate this, in a text-only human review, we ask an annotator to look at 100 questions (75 - blind filtered, 25 - retained in meta-video). Based on this study, 96% (24/25) of Meta-Video questions were judged to require the video. For the filtered-out questions, 64% were answerable at least without videos or through general knowledge or cues in the question or options, or language priors. The remaining 36% were judged to need the video, which reflects that the filter is conservative: it may discard some video-dependent questions, but it does not admit blind-answerable ones. This indicates that most of the questions filtered did not measure the video understanding aspect of the benchmark. We will add this study to the appendix.

To test whether Meta-Video generalizes beyond these families, we evaluated GLM-4.6V-Flash, another model family that was not used for blind filtering, at 4–64 frames under the same protocol:

Frames
Model48163264
GLM-4.6V-Flash37.842.848.748.351.4

GLM follows the same trend as the other models: accuracy increases with frame count and saturates around 16-32 frames (Table 4). Its scores are similar and are within the range of the evaluated models (e.g., 51.4 at 64 frames vs. 39.5–57.4 for the other five models). This shows that Meta-Video gives a meaningful signal for a family it was not built from. We will add GLM results on the full source benchmarks and a larger-scale model in the revision.

Q4. We apologize for this. The two video counts refer to different things. Meta-Video contains 788 unique video files. A few videos appear in more than one source benchmark with different questions, and counting each (source, video) pair separately gives 791 videos. We will use one definition consistently (788 unique videos) and explain the overlap in Sec. 3.2. We clarify and reassure that all the reported statistics are checked and will be refined in the final version. We will also resolve the references and placeholders.

Dear Reviewer,

Thank you once again for your constructive feedback. As we approach the end of the discussion period, we wanted to reiterate that we have provided clarification on the typos, on construction of Meta-Video and we have provided comparison with randomly sampled subsets. We have also provided human check used in the construction of Meta-Video, along with experiments for generalization on new model family: GLM-4.6V-Flash.

With less than 24 hours remaining in the discussion period, we would greatly appreciate your feedback on our rebuttal and additional experiments. Please let us know whether our responses affect your assessment or if any concerns remain unresolved, so that we can provide further clarification before the deadline. We believe these efforts address the concerns raised in your original reviews and have improved the paper.

Thank you again for your time and consideration. Even a brief update would be very helpful.

Sincerely,

The Authors

Thank you for the rebuttal and additional experiments. While I appreciate the authors' efforts, my concerns regarding the limited novelty, insufficient benchmark validation, and overall writing quality remain. The manuscript contains substantial grammatical errors, unclear expressions, and poorly organized discussions that require more than minor proofreading. Therefore, I will maintain my original rating.

AuthorsOpenReview ↗

Re: Official Comment by Reviewer q9uY

Re: Official Comment by Reviewer q9uY

We are happy to correct the four (!) glitches the reviewer pointed out for the camera-ready version. Just to reiterate, those were:

  • "Conclusion (Lines 477–485) is particularly problematic, with sentences such as "We propose Meta-Video, a compact video understanding benchmarking which is efficient, and version of the source benchmarks."
  • "Table ref" (Line 324)
  • "yy points lower" (Line 910)
  • "Table ??" (Line 946)

Thanks, actually, for reading the paper so carefully up to the end, and even pointing to a missing value and a missing reference in the supplementary on pages 17 and 18 (Line 910 and Line 946) .

We didn't comment on that because we took it for granted that we will resolve all those issues for the camera-ready version. We don't think that those issues justify rejection.

We also want to point out that the paper was written by real humans, so yes, humans make mistakes. We will obviously fix those, but we don't see how this would devalue the paper's contribution, especially given that we seem to have addressed all the questions listed in the initial review.

Reviewer_3c4TOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper has two main contributions. (1) It introduces Meta-Video, a compact benchmark constructed by selecting questions from five existing video benchmarks and filtering questions that models can answer without visual input. (2) It studies how evaluation choices, including frame count, sampling, resolution, input format, prompting, inference backend, batch size, and video decoder, affect reported performance. Based on this analysis, the authors propose a standardized evaluation protocol and evaluation cards to report the settings used.

Critical strengths

(1) While several evaluation factors are already well known, I found the analysis of the inference backend and batch size in Table 2(a), and the video decoder in Table 2(d), informative and novel. These are less commonly discussed details that can affect evaluation results.

(2) Although the model coverage is limited, the paper examines a broad range of evaluation settings across multiple benchmarks. This detailed experimental analysis helps explain how implementation choices affect reported performance.

Critical weaknesses
  1. One of the main contributions of this paper is a study of evaluation settings and a proposal for a standardized evaluation protocol. (i) The effects of frame count, frame sampling, resolution, and prompting are already well known, and existing benchmark papers already investigate several of these factors. (ii) Evaluating a proposed method and its baselines/competing methods under the same, and clearly reporting the inference settings, is also an established practice, and these are details that reviewers specifically check. While the backend and decoder observations may be useful (and new info), I am not clear what substantial new insight the broader standardization study provides beyond these existing findings and practices.

  2. The argument for standardization is incomplete and weak (particularly how Tables 1 and 8 are used to support this argument).

    (a) Table 1 shows that different papers report different performance for the same baseline. However, it does not examine whether each paper compared its method and the baseline under the same setting and clearly reported that setting. Different papers can reasonably choose different configurations. If they rerun the baseline and competing methods under the same configuration, they already follow the principle recommended here. A difference from the original Qwen score or the reproduction in this paper does not make those comparisons unfair or their gains invalid. Using these cross-paper differences to cast doubt on the reported improvements is therefore potentially misleading without examining the original evaluation protocols. The later controlled experiments provide separate evidence. Instead, the paper should clarify what is missing in existing evaluations and what its proposed standardization would correct.

    (b) Comparing all methods under the same settings is a valid way to make a controlled comparison, and many papers already follow this practice. In that case, the baseline scoring differently from its original paper is not itself a problem. However, this does not mean that one common configuration is always the most appropriate way to compare different approaches. A setting that works well for one method may disadvantage another because of how it was trained or designed to run. The other option is to evaluate each method under its own recommended settings. This can better reflect its intended performance, but those results may involve different inputs or computational costs and cannot be compared without accounting for these differences. Both approaches have advantages and limitations. The appropriate choice depends on what is being compared and what conclusion is being drawn. As this paper specifically studies evaluation protocols, there must be a clear discussion of both sides.

    (c) Table 8 in the appendix is an example of why this distinction matters. The authors evaluate reasoning models against Qwen2.5-VL-7B without chain-of-thought and with only 16 output tokens, then use their limited gains to question whether reasoning models are needed. But this only tests how these models perform when reasoning is removed. It does not test whether that procedure provides useful improvements when it is actually used. Existing work already distinguishes where its benefits appear: for example, Video-R2 reports comparable performance to the base model on generic benchmarks, stronger performance on reasoning benchmarks, and improvements in reasoning quality. Questioning the usefulness of such models requires examining those claims under the relevant evaluation settings and the capabilities they target. As a paper focused on fair evaluation, these distinctions must be addressed before drawing such conclusions.

  3. The method of black-frame used for filtering samples does not establish that the remaining questions require video understanding. The paper removes a question only when all three filtering models answer it correctly with black frames. A question can therefore remain simply because one model fails to answer it due to limited knowledge, reasoning errors, or ambiguity in the question, and does not necessarily mean that watching the video is required. A human check of whether the retained questions can be answered without watching the video, and whether they can be answered reliably after watching it would have been a stronger approach. This would provide stronger evidence that the filtering actually selects questions that depend on visual information. The current procedure is a useful initial check, but is not sufficient to establish this.

  4. The claim that Meta-Video preserves model rankings is not supported well enough. For example, in Table 4, at the same budget of 32 frames, InternVL3.5-4B performs better than Qwen3-VL-4B on the seven-benchmark average, with 59.5 versus 57.6. However, the ranking is reversed on Meta-Video, with 45.5 versus 50.0. There are several other patterns like this, and the results do not consistently lead to the same conclusion about which model performs better. The paper summarizes these results by saying that Meta-Video broadly follows the same pattern as the full benchmarks. The stronger claim that Meta-Video preserves model rankings is therefore overstated.

  5. The model selection is too narrow to establish broad benchmark validity. The main comparison includes five models in the 4-8B range from only two model families, Qwen and InternVL. What about large-scale models? Audio and subtitles are also insufficiently studied.

  6. I am not fully convinced by the motivation for combining existing benchmarks into a smaller set primarily for quicker evaluation. During development, we can use dedicated validation sets or a few selected benchmarks to test ideas and refine the training recipe. Once the model is finalized, broader evaluation serves to assess generalization across different tasks and domains. The goal should not be to repeatedly tune against all these benchmarks. Is Meta-Video intended as a development tool or a substitute for broader evaluation? If it is the latter, it is unclear whether this smaller set can provide the same evidence of generalization as evaluating across the original benchmarks. The paper therefore needs to better justify why the reduction in evaluation cost warrants reducing the coverage offered by the original benchmarks.

Questions for authors

Please address the questions and concerns raised in the weaknesses above.

7 comments
AuthorsOpenReview ↗

Response to reviewer 3c4T (1/3)

We thank the reviewer for their detailed feedback for noting that the analysis on the inference backend and video decoder is informative and novel, and that the paper covers a broad range of evaluation settings across multiple benchmarks, and how implementation choices affect reported performance.

The following are answers to the questions. We hope our responses below address your questions.

Q1. We agree that the influence of frame count, sampling, resolution and prompting are expected to influence results in practice. However, these influences have not been studied systematically for video understanding benchmarks. Benchmark papers typically ablate one factor (usually frame count) on their own benchmark for a few models. Systematic sensitivity studies such as Sober Reasoning (Hochlehnert et al., 2025) target language/math, not video. However, no prior work (i) varies backend, batch size, decoder, sampling, resolution, input modality, prompt/layout and temperature under one controlled protocol, (ii) relates their magnitude to the claims being made in the literature, or (iii) turns the result into a reporting standard.

We believe the parameters reported above can produce varied results, and although different methods can use different settings, it is at least important that these settings are reported. Also, it is important to note that some of the parameters can vary the performance by a larger magnitude. For the same checkpoint, Qwen2.5-VL-7b, with everything fixed, frame count changes by 15 points, the backend changes the performance ~6 points on MVBench. Each is comparable to, or larger than, the gains reported in works that build on top of these. Several of these settings (backend, decoder, input modality) are almost never reported, and some differences can arise due to implementation issues (ex: vLLM not doing spatio-temporal pooling) that the matched settings don’t cover. Some of the papers (ex. Video-r2) report frames used, but other parameters are unknown. Therefore, by measuring which settings actually matter and how much they matter, evaluation cards, providing fixed protocol, and efficient benchmarks give the community a concrete way to make these evaluations comparable and reproducible.

Q2 (a). We agree that cross-paper differences alone do not invalidate a paper’s gains, and that a paper which re-evaluates its baseline and other models under one stated configuration follows principles we recommend. We will revise the text around Table 1 to make this distinction clear. However, it shows that this principle is sometimes not followed in practice. Moreover, some of the aspects such as backend and input modality were never studied to know that these parameters also matter a lot especially in conditions that look similar, i.e. 128 frames with .mp4 is not same as .jpg. Among the 12 papers reported for Qwen2.5-vl-7b model, most papers state the number of input frames (11/12), but only a few specify the remaining factors. Only 3/12 report frame sampling rate and evaluation toolkit, 4/12 report decoding parameters. None of the works say whether videos are used as .mp4 or as .jpg, or which video decoding backend was used. Only 3/12 explicitly re-evaluate the base model under their own setting, 4/12 suggest it without stating, 5/12 give no information on where the base model numbers come from.

Our standardization addresses this in three ways: Same configuration: the baseline and the method are evaluated under one stated configuration including the frame budget, Full reporting: an evaluation card records every setting that affects scores, including settings that are rarely reported, such as backend, decoder and input modality and fixed implementation issues: we fix issues in the standard eval frameworks, such as spatio-temporal pooling for vLLM, which is an important aspect of model evaluation.

Q2 (b). We agree that both approaches are evaluated in the same settings, or under a method’s recommended setting. We are not necessarily favoring one over the other. It depends on which setting is varied, and in either case the setting should be reported. Our main goal is transparency, not a compulsory configuration. With respect to the two settings, we want the implementation-level settings to be at least reported. These include inference backend, video decoder, batch size, input modality, and decoding temperature. For some of these like inference backend, the methods are not designed to depend on them, yet they shift scores substantially (ex: ~5points for the backend in Table 2). For method-level settings, like frame budget, prompt etc can follow a how a model was trained on, but they must be reported. Our protocol supports both. For frame budget, we sweep 4-128 frames. For prompts, each model can use its intended prompt (next answer Q2. c). Since different settings can mean different inputs, we report inputs and runtime for both and also report the inputs in the evaluation card.

AuthorsOpenReview ↗

Response to reviewer 3c4T (2/3)

Q2 (c). We will fix the paragraph about reasoning models, and agree that table 8 only measures these models with reasoning disabled (direc answer, 16 tokens). We also want to clarify that our protocol does not standardize prompts, but the important aspect is that it is reported in the evaluation card for reproducibility, as the benchmarks accuracies are sensitive to the prompts. Following the reviewers suggestion, we evaluated each reasoning model with its own recommended prompt. We use the recommended settings for each model.

MMVUSciVideoBench
ModelPrompt32641283264128
Qwen2.5-VL-7Bdefault63.3665.9264.6426.126.927.9
Video-R1-7Bdefault55.2053.7655.5222.425.026.4
video r164.1664.9664.4826.728.827.9
Video-R2default63.8464.1664.8025.827.728.6
video r266.7264.9665.2827.826.327.7
VideoChat-R1.5-7Bdefault64.4866.2464.1626.827.328.4
video chat r1564.8064.9665.1228.327.827.4

On MMVU, the best-reasoning model score (Video-R2, 66.7 at 32 frames) and for SciVideoBench, Video-R2 with default prompt performs best at 128 frames. Our claim is not that the reasoning models are unnecessary. It is that their accuracy gains on these benchmarks are of the same order as the evaluation setting, so it is advised to report these settings carefully.

Q3. The filter is designed as a check that is necessary but need not be sufficient, i.e. it removes questions that are answerable from text alone, but it is not meant to certify that the remaining questions depend on visual information. Its purpose is to remove text-answerable questions that add no signal about video understanding, and to avoid compute on them. The reason we remove questions answered by these models is because with mcq with 4 options, any single model answers a fraction of questions correctly by chance. On the other hand, requiring agreement across models reduces the false removals. Since, the retained questions come from benchmarks designed and validated to require video with human annotations targeting specific video capabilities, our filter only removes questions that are easy and doesn’t replace that.

Human check. Furthermore, to validate this, in a text-only human review, we ask an annotator to look at 100 questions (75 – blind filtered, 25 – retained in meta-video). Based on this study, 96% (24/25) of Meta-Video questions were judged to require the video. For the filtered-out questions, 64% were answerable at least partly without videos, through general knowledge, cues in the question or options, or language priors. The remaining 36% were judged to need the video, which reflects that the filter is conservative: it may discard some video-dependent questions, but it does not admit blind-answerable ones. We will add this study to the appendix.

Q4. We will refine our statement regarding model preservation to largely preserves rankings, and also report rank correlations. In Table 4, at each model’s best setting, Meta-V recovers the ranking and the optimal budget. As presented in Sec. 5.1, models differ in which frame budget suits them, we compare each model at its best budget. Meta-V selects the same optimal budget as the seven-benchmark average for 4 of 5 models: InternVL3.5-4B peaks at 32 frames on both, and Qwen2.5-VL-7B, Qwen3-VL-4B and Qwen3-VL-8B peak at 128 on both. For InternVL3.5-8B the two differ (64 vs. 128), but its Meta-V scores at these budgets differ by only 0.2 points. However, it is to be noted that the benchmark average is over seven benchmarks which includes few that are not part of Meta-Video sources. Meta-Video is therefore not intended to reproduce the average exactly, but it aims to give a cheaper signal that agrees with similar conclusions.

AuthorsOpenReview ↗

Response to reviewer 3c4T (3/3)

Q5. We focus on the 4-8b range specifically as it is where most current video understanding works build upon. Nearly all recent open video reasoning models are finetuned from QwenVL or InternVL backbones at this scale (eg. VideoR1, VideoRFT, VideoR2, etc.). These backbones are therefore the baselines that published comparisons depend on. Audio and subtitles: Our scope is currently restricted to the video understanding with these components, as standardizing the video aspect is important and further other modalities can be included in future studies. We will state the scope of the paper explicitly in the revised version.

To test whether Meta-Video generalizes beyond these families, we evaluated GLM-4.6V-Flash, another model family that was not used for blind filtering, at 4–64 frames under the same protocol:

Model48163264
GLM-4.6V-Flash37.842.848.748.351.4

GLM follows the same trend as the other models: accuracy increases with frame count and saturates around 16-32 frames (Table 4). Its scores are similar and are within the range of the evaluated models (e.g., 51.4 at 64 frames vs. 39.5-57.4 for the other five models). This shows that Meta-Video gives a meaningful signal for a family it was not built from. We will add GLM results on the full source benchmarks and a larger-scale model in the revision.

Q6. Meta-Video is intended as a development tool, not a substitute for full evaluation. We will state this explicitly in Sec. 3 and the abstract. A single benchmark covers a single capability. Development decisions made on it can improve one axis while degrading another. Meta-Video covers five complementary axes at roughly the cost of one benchmark. Since evaluating models across settings, especially frames, is an essential part of model building, doing it on all datasets with all possible settings multiplies the cost. For example, the frame sweep in Table 2(b) took 3,675 GPU-minutes on four benchmarks vs. 695 on Meta-Video. Meta-Video makes such sweeps affordable during development, and its per-duration split shows where a change helps (Table 4). Once a model improves on Meta-Video, full runs on the relevant source benchmarks will help confirm the gains.

Reviewer_GJ7fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how configuration and implementation choices affect video-understanding evaluation. The authors construct Meta-Video, a compact benchmark of 1,000 question–answer pairs selected from five existing benchmarks, to reduce evaluation cost. They also examine inference backends, frame sampling, input representations, and prompts. The paper diagnoses and fixes an input-processing discrepancy in the evaluated HF/vLLM wrappers, provides reference results for five models across seven benchmarks, and proposes evaluation cards for reporting configurations.

Critical strengths
  1. The paper addresses a practical problem that is easy to overlook when comparing video models. Examining the different stages of the evaluation pipeline helps explain discrepancies between reported results.

  2. I found the HF/vLLM diagnosis particularly useful. The authors trace the discrepancy to how video inputs reach the model and show that correcting this brings the results closer together (Appendix A.3, Table 6). This provides concrete guidance for improving existing evaluation implementations.

  3. A compact evaluation set is a useful resource, and Table 4 provides a helpful reference for comparing several backbones and their responses to different frame budgets. The evaluation cards are also a sensible reporting convention that could make subsequent comparisons more transparent.

Critical weaknesses
  1. I am less convinced about the added value of the curation procedure itself (§3.1, Appendix A.5). The experiments show that a smaller evaluation can reduce cost while retaining useful performance trends, but they do not directly establish how much the black-frame filtering and sample selection contribute beyond a similarly sized subset with comparable source and category coverage. This makes the benchmark-construction contribution harder to assess.

  2. The evidence for using Meta-Video during model development is somewhat indirect. Table 4 mainly compares existing backbones and frame budgets, which is useful for model and inference-configuration selection. Method development also often involves comparing a shared backbone before and after a training or methodological change. Table 8 considers base and fine-tuned models, but reports these comparisons only on Video-MME. It remains less clear how useful Meta-Video would be for this type of comparison.

  3. A smaller concern is the source of the gain from single-letter prompting (§§4.3, 5.2; Table 5). Section 4.3 acknowledges that prompts affect both predictions and answer parsing, but Table 5 does not separate these effects. I would like to understand how much of the improvement comes from more accurate answers and how much comes from more reliable formatting and extraction. This would help interpret the reported end-to-end accuracy gains.

Questions for authors
  1. Could you provide a representative comparison with an equally sized, randomly sampled subset matched for source and category coverage, to clarify what the curation adds? Existing per-example predictions could be used where available.

  2. Could you include a representative base-versus-fine-tuned comparison on Meta-Video, for example using models already considered in Table 8, and discuss how it compares with the full-benchmark result?

  3. How are final answers extracted under the different prompts in Table 5? Do differences in output formatting or answer extraction contribute to the reported gap?

2 comments
AuthorsOpenReview ↗

Response to reviewer GJ7f

We thank the reviewer for their constructive feedback and positive assessment of the work. Especially highlighting that this paper addresses practical problems which are easy to overlook, it mentions that evaluating different stages of the pipeline explain the discrepancies, different configs like hf/vllm and meta-video as a useful compact evaluation along with the evaluation cards as sensible reporting convention.

The following are answers to the questions. We hope our responses below address your questions.

Q1. Using our existing per-example predictions, we compared Meta-Video against equally sized (1,000-question) subsets: stratified random samples matched for source and category coverage (500 draws). About 1/3rd of a matched stratified subset is answerable without the video (stratified: 31.9 ± 1.3%), and ~30% of evaluation compute is spent on these questions. For Meta-Video, it contains none. Therefore, the accuracy gain from seeing the video is 36.4 points on Meta-Video vs. 21.1 ± 1.1 for stratified. Cost is similar for all subsets of the same size (~4,700–4,800 s). The speedup comes from subset size, and our curation ensures this compute budget is spent only on questions that measure video understanding. We will add this experiment to table 5.

ModelStratified randomMeta-V
InternVL3.5-4B55.1 ± 1.345.5
Qwen3-VL-4B54.5 ± 1.450.0
Qwen2.5-VL-7B46.2 ± 1.234.9
InternVL3.5-8B57.3 ± 1.550.4
Qwen3-VL-8B55.6 ± 1.455.0

Q2. We evaluated Qwen2.5-VL-7B and the three reasoning models fine-tuned from it (Video-R1, VideoChat-R1.5, Video-R2) on Meta-Video. We used the same protocol as Table 8 (direct-answer prompt) and, following our response on reasoning models, also each model’s own recommended prompt, at 16, 32, 64 frames. We will add performance for 128 frames in the revised version.

ModelPrompt16 frames32 frames64 frames
Qwen2.5-VL-7B (base)direct32.535.939.2
Video-R1direct30.732.434.2
own31.133.036.6
VideoChat-R1.5direct36.438.839.7
own34.736.938.7
Video-R2direct34.535.939.1
own36.438.240.6
  • Similar conclusion to VideoMME benchmark: Both VideoMME, and Meta-Video lead to the same overall conclusion which is under a controlled protocol, finetuned models only outperform their base model by a small margin. Both VideoChat-R1.5 and Video-R2 models perform consistently better than the base model at lower frame count.
  • It is important to note that Meta-Video is more discriminative than VideoMME. Under the direct-answer setting, VideoChat-R1.5 gains +3.9, +2.9 points over its base at 16, 32 frames respectively on Meta-Video, but is within ±1 points on VideoMME. A probable explanation is that around 1/3rd of the questions of VideoMME are answerable without video.
  • Effect of prompt: For VideoR2, with its own prompt, it improves over the base model in all frame settings with +3.9,+2.3,+1.4 points for 16,32,64 frames respectively. VideoR1 improves with its own prompt but still remains below the base model, and VideoChatR1.5 performs better with the direct prompt.

Q3. All prompts were scored with each source benchmark's own lmms-eval answer parser (e.g., a ranked option-letter extractor for Video-MME, the first numeric token for VSI-Bench); outputs with no valid answer count as wrong. Only for P1 (think step-by-step prompt) failures include no truncation, i.e. 7% of failures hit the 1,024-token limit. For P* and P0, all answers are parsed successfully.

Thanks for the additional comparisons. The matched random subsets show that the curated questions better capture the benefit of video input. I value the paper’s practical contribution in reducing evaluation costs and making results easier to compare. These are useful improvements to existing evaluation practice, though I consider the methodological advance incremental. I’ll therefore keep my score at 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow up comments for reviewers

Dear Reviewers,

Thank you again for your time and constructive feedback. In response to your comments, we have provided the following experiments and clarifications:

Reviewer GJ7f: added experiments on similar sized random stratified filtering to compare with Meta-Video, Meta-Video results for base-versus-finetuned-models (Video-R1, VideoChat-R1.5, Video-R2), and answer extraction.

Reviewer 3c4T: detailed explanation for why Meta-Video and standardization of evaluation are important, discussion about comparing all models under the same setting and evaluating each model under its own recommended setting, evaluation of two datasets (MMVU, SciVideoBench) on video reasoning models (Video-R1, VideoChat-R1.5, Video-R2) under their recommended settings, human check for the quality of blind-filtering, evaluation of GLM-4.6V-Flash on Meta-Video under different frame setting (4-64 frames), and a discussion on importance of Meta-Video as a development tool for models.

Reviewer q9uY: clarification on model preservation, experiment on stratified filtering compared to Meta-Video and human check on blind-filtering.

As the discussion period is now more than halfway through, we would greatly appreciate your feedback on whether these responses address your concerns and affect your overall assessment. If any points remain unresolved, please let us know so that we can address them during the remaining discussion period.

Thank you for your continued engagement. Even a brief update would be very helpful!

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.