- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the ability of multimodal large models to identify erroneous or missing information within scientific experimental videos. Existing benchmarks primarily focus on whether models can answer questions supported by complete evidence, while rarely testing their capacity to determine that a question is unanswerable when the evidence is unreliable. To address this, the authors introduce the RoSci-Video benchmark, which comprises 3,000 pairs of standard and "distractor" questions derived from 390 experimental videos across 13 scientific disciplines. These distractors incorporate factual errors, contradictions with the video content, or the omission of critical conditions or visual elements. The authors evaluate the models' resilience to information degradation by comparing their accuracy on standard questions against their refusal rate (i.e., correctly identifying that a question cannot be answered) on distractor questions. Testing 32 multimodal models reveals a significant robustness gap: while models often answer standard questions correctly, they frequently persist in providing an answer even when the evidence is flawed or insufficient. Identifying missing information proves particularly challenging; in a paired-question analysis, the most common failure mode is for a model to answer a standard question correctly while failing to refuse the corresponding distractor version. Although explicitly prompting models that an "unanswerable" response is permitted can increase refusal rates, it does not fully resolve the issue. The study concludes that reliable scientific video understanding requires not only the ability to answer questions but also the capacity to assess whether the available evidence is sufficient to support a conclusion.
Critical strengths
- Propose reviews that are closer to scientific experiments. It uses pairs of normal questions and interference questions to systematically test whether the model can identify information errors and missing information in experimental videos, supplementing existing evaluations that focus on answer accuracy.
- Clear directions for improvement were found. Experiments show that the model often "answers normal questions but does not stop when the evidence fails", and it is especially difficult to detect missing information. This suggests that follow-up work needs to strengthen the judgment of whether the evidence is sufficient, rather than just improving scientific knowledge or question-answering abilities.
Critical weaknesses
- The task design is somewhat one-dimensional and fails to adequately capture true robustness. The paper categorizes standard inputs as "answerable" and perturbed inputs as "must-refuse," lacking a control condition for cases where an input is perturbed yet can still be answered correctly. Consequently, it is difficult to determine whether the model truly understands which evidence influences the conclusion, or if it simply defaults to a refusal response whenever it encounters an anomalous input.
- The experimental coverage is wide, but the method comparison and cause analysis are not in-depth enough. Although 32 models were evaluated, the experiments mainly focused on default performance and simple refusal prompts, and lacked targeted evidence verification, confidence calibration and other method baselines. Therefore, the paper shows the poor performance of the model, but it has not yet clarified which link the failure mainly occurs and how much the existing methods can be improved. The guidance for subsequent research is still limited.
Questions for authors
- Could you evaluate some more recent models? The models currently assessed in the paper are rather dated.
- Have the authors considered adding "disturbed but still answerable" control samples, such as masking out the same number of non-keyframes, and comparing with keyframe absence? Such experiments can further test whether the model is able to determine the actual impact of interference on answers, rather than just a tendency to reject answers in response to anomalies in the input.