acceptodds
Under review as a conference paper at ICLR 2027

RoSci-Video: Benchmarking Robust Scientific Understanding and Reasoning on Experimental Videos

Abstract

Scientific videos from laboratory experiments often contain missing, erroneous, or conflicting data across modalities. When models fail to recognize such degradation, they risk producing confidently wrong conclusions in scientific understanding and reasoning. Existing benchmarks for large multimodal models (LMMs) do not evaluate this capability. To bridge this gap, we introduce RoSci-Video, the first benchmark evaluating LMM robustness to information degradation in laboratory videos. It covers 3000 questions across 390 videos from 13 disciplines, with perturbations categorized as Information Error or Information Loss. We define Robustness Drop (RD) as the difference between reference accuracy and rejection rate. Evaluating 32 LMMs reveals three findings: (i) all models exhibit positive RD, indicating a universal robustness gap; (ii) information loss perturbations produce higher RD than information errors, showing models struggle more with missing information; (iii) the dominant failure mode is capability-present robustness failure: models answer correctly on clean references yet commit to incorrect answers on degraded inputs, accounting for the majority of cases. Within-family robustness trends are inconsistent, indicating that scale alone does not reliably predict degradation robustness.

Primary area:
datasets and benchmarks
Keywords:
Large multimodal models, Scientific videos, Information degradation, Robustness evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qXjLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the ability of multimodal large models to identify erroneous or missing information within scientific experimental videos. Existing benchmarks primarily focus on whether models can answer questions supported by complete evidence, while rarely testing their capacity to determine that a question is unanswerable when the evidence is unreliable. To address this, the authors introduce the RoSci-Video benchmark, which comprises 3,000 pairs of standard and "distractor" questions derived from 390 experimental videos across 13 scientific disciplines. These distractors incorporate factual errors, contradictions with the video content, or the omission of critical conditions or visual elements. The authors evaluate the models' resilience to information degradation by comparing their accuracy on standard questions against their refusal rate (i.e., correctly identifying that a question cannot be answered) on distractor questions. Testing 32 multimodal models reveals a significant robustness gap: while models often answer standard questions correctly, they frequently persist in providing an answer even when the evidence is flawed or insufficient. Identifying missing information proves particularly challenging; in a paired-question analysis, the most common failure mode is for a model to answer a standard question correctly while failing to refuse the corresponding distractor version. Although explicitly prompting models that an "unanswerable" response is permitted can increase refusal rates, it does not fully resolve the issue. The study concludes that reliable scientific video understanding requires not only the ability to answer questions but also the capacity to assess whether the available evidence is sufficient to support a conclusion.

Critical strengths
  1. Propose reviews that are closer to scientific experiments. It uses pairs of normal questions and interference questions to systematically test whether the model can identify information errors and missing information in experimental videos, supplementing existing evaluations that focus on answer accuracy.
  2. Clear directions for improvement were found. Experiments show that the model often "answers normal questions but does not stop when the evidence fails", and it is especially difficult to detect missing information. This suggests that follow-up work needs to strengthen the judgment of whether the evidence is sufficient, rather than just improving scientific knowledge or question-answering abilities.
Critical weaknesses
  1. The task design is somewhat one-dimensional and fails to adequately capture true robustness. The paper categorizes standard inputs as "answerable" and perturbed inputs as "must-refuse," lacking a control condition for cases where an input is perturbed yet can still be answered correctly. Consequently, it is difficult to determine whether the model truly understands which evidence influences the conclusion, or if it simply defaults to a refusal response whenever it encounters an anomalous input.
  2. The experimental coverage is wide, but the method comparison and cause analysis are not in-depth enough. Although 32 models were evaluated, the experiments mainly focused on default performance and simple refusal prompts, and lacked targeted evidence verification, confidence calibration and other method baselines. Therefore, the paper shows the poor performance of the model, but it has not yet clarified which link the failure mainly occurs and how much the existing methods can be improved. The guidance for subsequent research is still limited.
Questions for authors
  1. Could you evaluate some more recent models? The models currently assessed in the paper are rather dated.
  2. Have the authors considered adding "disturbed but still answerable" control samples, such as masking out the same number of non-keyframes, and comparing with keyframe absence? Such experiments can further test whether the model is able to determine the actual impact of interference on answers, rather than just a tendency to reject answers in response to anomalies in the input.
Reviewer_649ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoSci-Video evaluates whether multimodal models can recognize incorrect or insufficient information in experimental video questions and abstain. It contains 3,000 clean–degraded question pairs across 390 JoVE videos and six degradation types. The paper evaluates 32 models and introduces Robustness Drop (RD), defined as clean accuracy minus degraded-input rejection rate. Models frequently answer clean questions correctly but fail to reject their degraded counterparts.

Critical strengths
  1. The evaluation targets a relevant limitation. Recognizing unreliable evidence is important for scientific applications. The distinction between incorrect information and missing information provides a useful basis for analysis.

  2. The paired construction supports informative behavioral comparisons. Clean and degraded versions allow the authors to examine whether failures persist when a model can answer the original question. Expert review, human baselines, and prompt ablations strengthen the evaluation.

Critical weaknesses
  1. RD does not isolate robustness from task capability. Lower clean accuracy directly improves RD when rejection stays unchanged. In Table 2, Qwen2.5-VL-72B ranks above Gemini-3.1-Pro despite having both lower clean accuracy (84.6% vs. 99.2%) and lower degraded-input rejection (28.9% vs. 42.1%). This undermines the leaderboard and comparisons based on RD.

  2. The full-benchmark and subset results differ substantially without an adequate explanation. InternVL-3.5-38B has 21.6% rejection in Table 2 but only 0.3% on the claimed uniformly sampled 60% subset in Table 7. GPT-5.2 changes from 36.3% to 50.2%. These shifts require reconciliation before the paired analysis can support the conclusions.

Questions for authors
  1. How do you justify RD-based rankings when lower clean accuracy improves the score? Would rejection conditional on correct clean answers change the conclusions?

  2. Were Tables 2 and 7 computed from the same response logs and evaluation settings? Please explain the large rejection-rate differences and provide the subset identifiers and per-question outcomes.

  3. How are responses scored when they identify and correct a false premise, then provide the appropriate answer? Do such responses count as failures despite detecting the degradation?

Reviewer_1f8bOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RoSci-Video, a benchmark for evaluating whether large multimodal models appropriately abstain from answering scientific video questions when information is erroneous or incomplete. It pairs reference questions with degraded versions covering six degradation types and evaluates 32 models. The results show that even models with high accuracy on reference questions frequently continue answering rather than abstaining on degraded inputs.

Critical strengths
  1. The investigated problem is important that models should not only answer correctly but also recognize when the available evidence is insufficient or inconsistent.
  2. The paired question design can serve as a good diagnostic tool.
  3. The experiment covers 32 models, which makes the results more persuasive.
Critical weaknesses
  1. The criterion for unanswerability is insufficiently justified. The benchmark does not clearly distinguish between an incorrect statement in a question and insufficient evidence to answer the information actually requested.
  2. Robustness Drop is problematic as the primary ranking metric. The metricRD=acc_ref - rej_err, where lower is considered better, can favor models with lower reference accuracy. In Table 2, Gemini-3.1-Pro outperforms Qwen2.5-VL-72B in both reference accuracy (99.2% vs. 84.6%) and degraded-input rejection (42.1% vs. 28.9%), yet receives a worse reported RD (57.2 vs. 55.7).
  3. The reported results contain substantial discrepancies that require reconciliation. The paper states that the paired analysis uses a uniformly sampled 60% subset. However, the differences between the full-benchmark and subset results are difficult to reconcile with ordinary random subsampling of the same predictions. For example, InternVL-3.5-38B has a rejection rate of 21.6% on the full benchmark but only 0.3% on the subset.
Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.