acceptodds
Under review as a conference paper at ICLR 2027

VIALS: A Benchmark for Visual Interpretation of Artifacts in the Life Sciences

Abstract

In professional life sciences workflows, scientists routinely interpret visual artifacts (gel blots, microscopy images, plasmid maps, flow cytometry plots, molecular structures, ...) to inform research decisions. We introduce VIALS, a visual question-answering benchmark with 161 such interpretation tasks, spanning the types of artifacts examined throughout experimental workflows in the biotech industry (rather than polished figures from publications and textbooks). While frontier vision-language models can now fluently describe natural images, we find that they are unable to accurately interpret these scientific images, reflecting limitations in domain knowledge and domain-specific visual reasoning capabilities. In contrast, scientists with relevant domain expertise find these visual interpretation tasks straightforward. AI that cannot similarly interpret such images will have limited utility in professional life sciences workflows, where such artifacts are central to how scientists reason, communicate, and make decisions.

Primary area:
datasets and benchmarks
Keywords:
Vision-language models, visual question answering, scientific visual reasoning, benchmarks and evaluation, life sciences

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9kpBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VIALS, a VQA benchmark that focuses on evaluating the interpretation of scientific artifacts encountered in professional life sciences research. The images and questions are generated by professional annotators and are validated carefully to avoid existing in possible pretraining data of models. The authors evaluate 10 frontier VLMs and find that the strongest configurations achieve only 26.5% average accuracy, demonstrating a true difficulty in answering the questions in this benchmark. The authors report that most failures are associated with reading or interpreting information from scientific artifacts through error analysis. And they also conduct an evaluation with tool-assisted models which shows substantial performance improvements for several models at considerably higher computational costs.

Critical strengths

1. Expert-curated tasks of practically relevant scientific artifacts with sound examination and validation.

The benchmark is targeting questions and answers that are challenging and frequently-encountered in practical professional scenarios. This is essentially very different from existing work that has been mainly focusing sourcing from textbook materials. The curated questions are blinded-compared to existing benchmark that covers similar area and demonstrated clear practical motivation. The curation pipeline consists of independent expert attempts, additional review, and consensus procedures to establish reference answers, which all contribute to the quality of the examples.

2. Informative evaluation of frontier models and tool-assisted systems.

The authors systematically evaluate 10 models with repeated rollouts and reveal the substantial limitations of the models over the benchmark. The automatic judge for accuracy is also validated through an agreement test with human experts' judgement. The evaluation with tool-assisted systems demonstrate the trade-off between performance and computational costs.

Critical weaknesses

1. Error-type analysis is mainly done by a model and does not clearly receive the image, but 83–93% of the errors are attributed to categories associated with reading or interpreting the visual artifact.

The judge model for error types only receive the question, ground-truth answer, and the incorrect model response and generated reasoning. Without the image, attributing the mistake into error types like "Visual quantification error" which is described as "for errors in counting objects such as cells, colonies, or bands" might not be well-suited as the inputs do not reveal enough information to make judgements regarding the image. And it would be much more beneficial if the paper can include a comparison between model's error classification with expert's error classification.

Questions for authors
  1. Could the authors provide a domain-expert audit of a representative sample of the automatically assigned failure categories, including inter-annotator agreement and agreement with the LLM classifier?

  2. How should we interpret the difficulty of the benchmark? Can the author provide numbers like accuracies of human with different education backgrounds?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9kpB

We appreciate the reviewer’s feedback, which we have used to improve our paper. We respond to the individual comments below.

Error-type analysis is mainly done by a model and does not clearly receive the image, but 83–93% of the errors are attributed to categories associated with reading or interpreting the visual artifact.

The judge model for error types only receive the question, ground-truth answer, and the incorrect model response and generated reasoning. Without the image, attributing the mistake into error types like "Visual quantification error" which is described as "for errors in counting objects such as cells, colonies, or bands" might not be well-suited as the inputs do not reveal enough information to make judgements regarding the image. And it would be much more beneficial if the paper can include a comparison between model's error classification with expert's error classification.

Could the authors provide a domain-expert audit of a representative sample of the automatically assigned failure categories

Thank you for raising this concern. We agree that the automated labels alone cannot reliably separate difficulties in visual interpretation from gaps in scientific knowledge, and have clarified this in our revised paper. To assess their reliability, we asked domain experts to review 133 automatically labeled Gemini 3.7 Flash failure cases (all tasks where the model made an error in at least one rollout), where experts could also refer to the images during their review. The disagreement rate between human experts and our automated classifier was 12% (16 of 133 cases). For the final revision, we will ask a second expert to independently label the cases and report inter-rater disagreement, which we expect will be substantial as well.

How should we interpret the difficulty of the benchmark?

VIALS focuses on everyday interpretation tasks that are straightforward for PhD-level life sciences researchers who have previous experience handling these artifacts. The questions are designed to be answered directly from the provided images and text without external tools, but generally appear difficult for someone outside the particular domain. As reported in Section 3.5, independent expert attempts took 16 minutes to complete the average task, including writing detailed justifications of their answer/process. We did not measure the time needed to provide the answer alone, but estimate that most of the time was spent writing the justifications, and that scientists can answer many VIALS questions within a few minutes.

Reviewer_Nw4tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors introduces VIALS, which is a VQA benchchmark of 161 tasks. It can be used to evaluate how models interpret domain-specific visual artifacts in life science and biotechnology. With this benchmark, the authors found that the perfpramce of frontier models are limited. They also found that about 90% of failures stem from misreading, miscounting, or misinterpreting the visual artifacts rather than flawed downstream reasoning or misapplied scientific principles. If using models in an agentic fashion, the accuracy could be improved, but the token usage increases drastically.

Critical strengths
  1. The benchmark is well-developed. Compared with traditional benchmarks in the life science area using general academic knowledge, the VIALS using raw visual artifacts in real-world biotechnology and pharmaceutical R&D workflows. The blind expert survey provided evidence for VIALS' superior over those in HLE.
  2. The authors conducted good error analysis. The paper identifies some key issues make the models fail and why the accuracy could be improve with an agentic setup. They found most failures occur while models are reading or interpreting the artifact. When tool-assited agents were used, they can solve such perception problems.
Critical weaknesses
  1. The benchmark size is small. There are only 161 tasks, making each domain only has around 20 tasks. A single correct/incorrect would make the the accuracy in a domain change about 5%.
  2. The failure modes is developed in a very loose way. According to Appendix C.2, the failure modes were generated using Claude Sonnet 4.5 that does not see the task image. It classifies errors by comparing the question, the ground-truth answer, and the failing model's text reasoning trace. I think an expert-driven error analysis could be better or more reasonable.
  3. Some details were missed in the paper, for example, Section 3.6 reports that the authors used experts to judge the benchmark over HLE benchmarks, but does not explains who are the experts and how these expert assemment were carried out. Besides, Section 3.3 says that VIALS combines unpublished wet-lab data, CC-BY publication figures, and procedurally generated images. However, there is no information about the propotions, and if models perform differently on them.
Questions for authors
  1. Did models perform differently in procedurally generated vs. real experimental artifacts? did they achieve higher accuracy on clean procedurally generated artifacts compared to noisy, unpublished wet-lab artifacts within the same domains?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Nw4t

Thank you for reviewing our paper and for the thoughtful feedback! We address your questions and concerns below, and have accordingly improved our paper as well.

The benchmark size is small. There are only 161 tasks, making each domain only has around 20 tasks. A single correct/incorrect would make the the accuracy in a domain change about 5%.

We agree that the small domain subsets limit the precision of domain comparisons, and we will make this limitation clearer. VIALS focuses on carefully constructed tasks requiring substantial scientific expertise to construct, images encountered in real lab work, and a rigorous review process for every task involving several experts (including independent attempters). Its scale is similar to other focused scientific evaluations discussed in our related work: ScienceAgentBench [1] contains 102 tasks, while the Biology/Medicine and Chemistry VQA subset of HLE [2], contains 119 questions. We preferred to focus scientific experts’ valuable time on ensuring the realism and quality of these tasks rather than further expanding the number of tasks.

[1] Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. “ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery.” In Proceedings of the International Conference on Learning Representations (ICLR), 2025.

[2] Center for AI Safety, Scale AI, and HLE Contributors Consortium. “A benchmark of expert-level academic questions to assess AI capabilities.” Nature, 2026.

The failure modes is developed in a very loose way. According to Appendix C.2, the failure modes were generated using Claude Sonnet 4.5 that does not see the task image. It classifies errors by comparing the question, the ground-truth answer, and the failing model's text reasoning trace. I think an expert-driven error analysis could be better or more reasonable.

Thank you for raising this concern. To be clear, we did initially do an expert-driven error analysis to ensure the model errors are meaningful and to determine the recurring model failure modes. The automated failure mode classification was solely used to quantify the relative prevalence of failure modes across models and the entire benchmark. We agree that the automated labels alone cannot reliably separate difficulties in visual interpretation from gaps in scientific knowledge, and have clarified this in our revised paper. To assess their reliability, we asked domain experts to review 133 automatically labeled Gemini 3.7 Flash failure cases (all tasks where the model made an error in at least one rollout). The disagreement rate between human experts and our automated classifier was 12% (16 of 133 cases). For the final revision, we will ask a second expert to independently label the cases and report inter-rater disagreement, which we expect will be substantial as well.

Some details were missed in the paper, for example, Section 3.6 reports that the authors used experts to judge the benchmark over HLE benchmarks, but does not explains who are the experts and how these expert assemment were carried out.

Thanks for pointing this out. The survey involved 26 reviewers of which 69% had at least six years of relevant research or professional experience, and 88% had worked in industry life-sciences setting. As mentioned in Section 3.6, reviewers assessed the tasks within their areas of expertise without knowing whether they came from VIALS or HLE. From there, they rated each artifact’s professional appearance, how often they encountered similar artifacts at work, and the commercial or economic value of interpreting it. Appendix C.4 provides more details. We will add the survey participants’ background stats to Section 3.6.

Besides, Section 3.3 says that VIALS combines unpublished wet-lab data, CC-BY publication figures, and procedurally generated images. However, there is no information about the propotions, and if models perform differently on them.

Did models perform differently in procedurally generated vs. real experimental artifacts? did they achieve higher accuracy on clean procedurally generated artifacts compared to noisy, unpublished wet-lab artifacts within the same domains?

Of the 161 tasks, 121 use CC BY publication artifacts (75.2%), 23 use procedurally generated artifacts (14.3%), and 17 use unpublished internal lab artifacts (10.6%). Averaged across models, accuracy was 32.8% on procedural artifacts, 18.4% on internal lab artifacts, and 16.8% on publication artifacts. All ten models performed better on the procedural subset than on the internal lab subset. These differences partly reflect the domains represented in each subset. We will add these proportions and the performance comparison to the revised manuscript.

Reviewer_gwm2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VIALS is a benchmark of 161 open-ended visual questions covering scientific artifacts used in professional life sciences workflows. Tasks are created and reviewed by domain experts. The paper evaluates ten VLMs and several tool-assisted agents. Direct inference reaches at most 26.5% accuracy. Tool-assisted evaluation improves accuracy to 65.8%, with substantially higher token usage. The authors argue that reading and interpreting scientific visual evidence remains a major bottleneck.

Critical strengths
  1. The benchmark targets a useful evaluation gap. It covers experimental artifacts across several life sciences domains, rather than focusing on academic questions or a single imaging modality. The expert survey provides evidence of professional relevance.

  2. The reference-answer validation is a strong part of the work. Independent expert attempts and follow-up review help reduce ambiguous or incorrect answers. The reported 99.9% agreement between automated and expert grading also supports the scoring procedure.

Critical weaknesses
  1. The failure analysis does not establish the claimed causes of errors. The classifier never sees the images and relies on generated reasoning traces (Appendix C.2). This is insufficient to reliably distinguish visual errors from missing scientific knowledge. Likewise, tool-assisted evaluation increases both tool access and inference resources. Its gains do not isolate a perception gap.

  2. The small benchmark limits the strength of model and domain comparisons. Most domains contain only about 20–23 tasks. The confidence intervals in Figure 4 capture rollout variability on these fixed tasks, but not uncertainty from task sampling. Small score differences and domain-level patterns therefore need more cautious interpretation.

Questions for authors
  1. Can domain experts validate the failure labels on a representative sample while viewing the original images? What agreement do they achieve with the classifier?

  2. Does tool access still provide substantial gains against a direct-inference baseline with a comparable inference budget? This would help assess the proposed perception-gap explanation.

  3. Can you report task-level bootstrap confidence intervals and paired comparisons between models? Do the main ranking and domain-level conclusions remain supported?

1 comment
AuthorsOpenReview ↗

Response to Reviewer gwm2

We are grateful for the reviewer’s thoughtful assessment of the paper, which we have used to improve it. Below, we clarify the issues raised and provide additional results where relevant (also added to our revised paper).

The failure analysis does not establish the claimed causes of errors. The classifier never sees the images and relies on generated reasoning traces (Appendix C.2)...

Can domain experts validate the failure labels on a representative sample while viewing the original images? What agreement do they achieve with the classifier?

Thank you for raising this concern. We agree that the automated labels alone cannot reliably separate difficulties in visual interpretation from gaps in scientific knowledge, and have clarified this in our revised paper. To assess their reliability, we asked domain experts to review 133 automatically labeled Gemini 3.7 Flash failure cases (all tasks where the model made an error in at least one rollout), where experts could also refer to the images during their review. The disagreement rate between human experts and our automated classifier was 12% (16 of 133 cases). For the final revision, we will ask a second expert to independently label the cases and report inter-rater disagreement, which we expect will be substantial as well.

The confidence intervals in Figure 4 capture rollout variability on these fixed tasks, but not uncertainty from task sampling. Small score differences and domain-level patterns therefore need more cautious interpretation.

Can you report task-level bootstrap confidence intervals and paired comparisons between models? Do the main ranking and domain-level conclusions remain supported?

We averaged each model’s three attempts on each task, then sampled the 161 tasks with replacement 10,000 times to calculate 95% confidence intervals that also reflect uncertainty from task-sampling.

ModelAccuracyTask-level 95% CI
GPT-5.6 Sol26.5%[21.1%, 32.3%]
Gemini 3.7 Flash26.5%[20.7%, 32.7%]
Claude Opus 524.4%[18.8%, 30.2%]
Muse Spark 1.222.6%[17.0%, 28.4%]
Gemini 3.1 Pro21.5%[16.4%, 27.1%]
Kimi K320.9%[15.7%, 26.5%]
Grok 4.620.5%[15.5%, 25.7%]
GLM-4.6V10.8%[7.0%, 14.9%]
MiniMax-M310.8%[7.7%, 14.1%]
Mistral Medium 3.57.9%[4.6%, 11.4%]

We also performed paired comparisons by resampling the same tasks jointly for each model pair. Selected results are shown below, with differences calculated as the first model minus the second.

Model comparisonAccuracy difference (pp)Paired 95% CI (pp)
GPT-5.6 Sol − Gemini 3.7 Flash0.0[−7.0, 6.8]
GPT-5.6 Sol − Claude Opus 5+2.1[−4.1, 8.3]
Gemini 3.7 Flash − Claude Opus 5+2.1[−4.6, 8.7]
Grok 4.6 − GLM-4.6V+9.7[4.1, 15.5]

While GPT-5.6 Sol and Gemini 3.7 Flash have the highest observed accuracy, the paired comparisons do not show a clear difference between them or between either model and Claude Opus 5. We will update Figure 4 with the task-level confidence intervals and explain this in the text.

The task-level analysis supports our central finding that current models struggle with professional scientific artifact interpretation. We have added the following statement to the revised paper:

“VIALS covers several scientific artifact domains with a limited number of tasks in each. Domain results provide a descriptive view of model performance, while small differences between models or domains should be interpreted alongside their uncertainty estimates.”

Does tool access still provide substantial gains against a direct-inference baseline with a comparable inference budget?

We have updated our paper with more agentic tool-assisted evaluations, this time with low reasoning effort (still using OpenCode as the agent harness and reporting average results over three attempts per task). Below we report results for only those models (GPT-5.6 Sol, Grok 4.6) for which we are able to identify a low-enough reasoning setting under which the inference-costs of the tool-assisted run approximately match the costs of our original direct-VLM-inference. Even at the lowest reasoning settings, the agentic tool-assisted evaluations for Gemini 3.7 Flash and Claude Opus 5 remained much more expensive and hence could not be compute-matched against direct-VLM-inference.

ModelDirect VLM accuracyTool-Assisted (low reasoning) accuracyDirect VLM cost/attemptTool-Assisted (low reasoning) cost/attempt
GPT-5.6 Sol26.5%26.7%$0.037$0.038
Grok 4.620.5%21.1%$0.049$0.037

We find that both models still perform slightly better in the agentic tool-assisted runs even when compute-matched against direct VLM inference. But the difference is small; providing agentic tool access does not yield substantial accuracy gains when the inference-costs are matched.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.