- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces VIALS, a VQA benchmark that focuses on evaluating the interpretation of scientific artifacts encountered in professional life sciences research. The images and questions are generated by professional annotators and are validated carefully to avoid existing in possible pretraining data of models. The authors evaluate 10 frontier VLMs and find that the strongest configurations achieve only 26.5% average accuracy, demonstrating a true difficulty in answering the questions in this benchmark. The authors report that most failures are associated with reading or interpreting information from scientific artifacts through error analysis. And they also conduct an evaluation with tool-assisted models which shows substantial performance improvements for several models at considerably higher computational costs.
Critical strengths
1. Expert-curated tasks of practically relevant scientific artifacts with sound examination and validation.
The benchmark is targeting questions and answers that are challenging and frequently-encountered in practical professional scenarios. This is essentially very different from existing work that has been mainly focusing sourcing from textbook materials. The curated questions are blinded-compared to existing benchmark that covers similar area and demonstrated clear practical motivation. The curation pipeline consists of independent expert attempts, additional review, and consensus procedures to establish reference answers, which all contribute to the quality of the examples.
2. Informative evaluation of frontier models and tool-assisted systems.
The authors systematically evaluate 10 models with repeated rollouts and reveal the substantial limitations of the models over the benchmark. The automatic judge for accuracy is also validated through an agreement test with human experts' judgement. The evaluation with tool-assisted systems demonstrate the trade-off between performance and computational costs.
Critical weaknesses
1. Error-type analysis is mainly done by a model and does not clearly receive the image, but 83–93% of the errors are attributed to categories associated with reading or interpreting the visual artifact.
The judge model for error types only receive the question, ground-truth answer, and the incorrect model response and generated reasoning. Without the image, attributing the mistake into error types like "Visual quantification error" which is described as "for errors in counting objects such as cells, colonies, or bands" might not be well-suited as the inputs do not reveal enough information to make judgements regarding the image. And it would be much more beneficial if the paper can include a comparison between model's error classification with expert's error classification.
Questions for authors
-
Could the authors provide a domain-expert audit of a representative sample of the automatically assigned failure categories, including inter-annotator agreement and agreement with the LLM classifier?
-
How should we interpret the difficulty of the benchmark? Can the author provide numbers like accuracies of human with different education backgrounds?
Response to Reviewer 9kpB
We appreciate the reviewer’s feedback, which we have used to improve our paper. We respond to the individual comments below.
Thank you for raising this concern. We agree that the automated labels alone cannot reliably separate difficulties in visual interpretation from gaps in scientific knowledge, and have clarified this in our revised paper. To assess their reliability, we asked domain experts to review 133 automatically labeled Gemini 3.7 Flash failure cases (all tasks where the model made an error in at least one rollout), where experts could also refer to the images during their review. The disagreement rate between human experts and our automated classifier was 12% (16 of 133 cases). For the final revision, we will ask a second expert to independently label the cases and report inter-rater disagreement, which we expect will be substantial as well.
VIALS focuses on everyday interpretation tasks that are straightforward for PhD-level life sciences researchers who have previous experience handling these artifacts. The questions are designed to be answered directly from the provided images and text without external tools, but generally appear difficult for someone outside the particular domain. As reported in Section 3.5, independent expert attempts took 16 minutes to complete the average task, including writing detailed justifications of their answer/process. We did not measure the time needed to provide the answer alone, but estimate that most of the time was spent writing the justifications, and that scientists can answer many VIALS questions within a few minutes.