acceptodds
Under review as a conference paper at ICLR 2027

AriadneBench: Evaluating Scientific Decisions by AI Agents in Cryo-EM Workflows

Abstract

Recent advances in AI agents are reshaping workflows for scientific discovery. However, reliable decision-making in cryo-electron microscopy (cryo-EM) remains challenging for agents, as they must align decisions with scientific objectives and autonomously adapt to multimodal evidence. We introduce AriadneBench, the first benchmark for evaluating these capabilities in real cryo-EM workflows. It comprises 180 tasks across eight task families and 13 workflows, targeting scientific goal alignment and adaptive autonomy. Workflow snapshots preserve the information available at each decision point, while expert annotations and auditable evidence tools enable joint evaluation of answer correctness, reference evidence coverage, and operational compliance. We evaluate seven state-of-the-art agent systems and find that the strongest achieves an overall Success Rate of only 63.21%. Agents perform better at configuring operations and recommending next steps than at particle selection and failure localization. Trajectory analysis shows that even complete reference evidence coverage does not guarantee a correct decision. To investigate whether distilled knowledge from our benchmark can improve these decisions, we induce one reusable skill per task family from training trajectories and expert annotations. On 37 held-out tasks, skill guidance improves Success Rate for five of seven agents, with an average gain of 4.25 percentage points across all seven agents.

Primary area:
datasets and benchmarks
Keywords:
scientific agents, agent benchmark, cryo-EM, automated workflow

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HzxpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates general-purpose AI agents on 180 tasks spanning eight task families derived from 13 curated cryo-EM workflows. After the initial evaluation, it uses 143 tasks for skill induction, extracting reusable guidance for each task family, and reevaluates agents on the remaining 37 test tasks. It also compares performance under expert supervision.

Critical strengths

This is an early benchmark effort for AI agents in cryo-EM. The data and experiments are presented clearly, the selection of agents is reasonably comprehensive, and the decomposition of cryo-EM workflows into task families is insightful.

Critical weaknesses

The main concern is the lack of publicly accessible benchmark data. Although the appendix lists raw cryo-EM dataset identifiers, I could not find the job-level data needed to inspect the inputs, outputs, and evaluation setup. This limits both scrutiny of the benchmark’s validity and its reuse for developing specialized agents. A benchmark should provide either an accessible, reusable resource or substantive insights that expose limitations in existing benchmarks and challenge prevailing assumptions. In its current form, the paper does not sufficiently deliver either contribution, despite its clear experimental presentation.

Questions for authors
  1. “Training tasks,” Section 4.2. This term may imply model-parameter updates, whereas these tasks are used for skill induction without such updates. Please use terminology that makes this distinction clear.
  2. Potentially misleading highlighting, Table 2. The pale-green highlighting of IS may suggest that it performs best among AE, IS, EL, and ES. However, ES achieves higher Score and Success on many tasks. Please revise the formatting to avoid this implication.
  3. Definitions of EL and ES, Section 4.3. EL (“evidence list”) and ES (“evidence + steps”) are insufficiently explained, particularly the meaning of “evidence.” Appendix D.5.1 does not fully resolve this ambiguity. Please clarify these settings in the main text or provide concrete examples in a clearly referenced appendix.
  4. Placement of skill induction, around line 363. Skill induction is a central methodological component, yet it is introduced only briefly in the experiments. Please describe it in greater detail in Section 3, together with IS, EL, and ES, before presenting the corresponding experiments.
  5. Inconsistent terminology, lines 121–123. “Workflows,” “candidate groups,” and “workflow groups” appear to refer to the same concept. Please clarify any distinctions or use consistent terminology throughout.
  6. Workflow selection, Section 2.1. The screening procedure is described too briefly. Please provide a detailed account of the selection process or refer readers to the relevant appendix.
  7. Definition of “Branches,” Table 3. I am not aware of “Branches” as a standard cryo-EM term. Please define it and explain its meaning in this benchmark.
  8. Distinguishing workflows 1 and 6, Table 3. Both are labeled “Filament” and “In-house.” Without access to the underlying data, readers cannot meaningfully distinguish them.
  9. Representativeness of the selected workflows. Please justify the coverage of the 13 workflows, including statistics on molecular weight, symmetry, signal-to-noise ratio, heterogeneity, and preferred orientation. Basic metadata can be checked for public EMPIAR datasets, but corresponding information for the in-house datasets is inaccessible to readers.
  10. Validity of single-step reference answers. A cryo-EM decision may have several valid answers depending on the downstream objective, and its benefits may emerge only over subsequent steps. Without access to the benchmark data, readers cannot assess whether the reference answers are appropriate.
Reviewer_tpQsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes AriadneBench, a benchmark for evaluating agentic capabilities in cryo-EM scientific workflows. It includes 180 tasks in 8 task families. It operates on workflow snapshots with expert annotations. The authors show an improvement across tasks when LLM agents are modified with a skill suite.

Critical strengths

The pipeline results in an auditable record of evidence and reasoning, which is useful for human expert-in-the-loop domains.

The task is not saturated, which is valuable for improvements to LLM agents. Additionally, some degree of dynamic range was shown by inclusion of the skill suite experiments.

Critical weaknesses

The benchmark appears to rely completely on precomputed workflows. Most tasks are read only and all expensive outputs are pregenerated; only Inspect Picks and Interaction allow configuration of parameters (but these configurations do not appear to be run). Agents are never able to see the outcomes of their choices, recover from failures, or truly iterate on workflows. This makes the benchmark cheaper, but means that the benchmark does not even test closed loop autonomous agentic workflow. How this benchmark transfers to real-world use cases is unclear.

There is quite limited evidence about action ordering or planning. It does not seem that inspecting evidence in causally relevant orders is rewarded. It seems that the benchmark measures one shot behavior rather than planning over workflows.

The paper does not sufficiently motivate the importance of each task in the overarching scientific questions relevant to cryo-EM.

Questions for authors

The scope of "local analysis" is unclear and needs clarification.

Give plain language explanations for each task family with a focus on what agent capability is being measured, as well as the scientifically relevant goal.

Why is the benchmark described as measuring adaptive autonomy if it is almost entirely open-loop?

What is a comparison to the performance of non-llm baselines and human experts?

Reviewer_e1PHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark of 180 tasks across eight task families and 13 cryo-EM workflows. Agents inspect workflow snapshots and make decisions, evaluated through answer correctness, reference-evidence coverage, and operational compliance. The strongest agent achieves a family-averaged success rate of 63.21%. The authors also derive one reusable skill per task family from training trajectories and expert annotations. On 37 held-out tasks, these skills improve success rates for five of seven agents, averaging a gain of 4.25 percentage points.

The tasks address meaningful scientific decisions. Preserving native workflow history, branches, and multimodal outputs is a strength, and the audit tools distinguish evidence retrieval from merely listing sources. The paper also reports failures and experimental limitations openly.

Critical strengths
  1. The benchmark targets meaningful scientific judgments in realistic workflow contexts.

Tasks require agents to interpret intermediate results, compare alternatives, and trace earlier processing decisions. Native dependencies, branches, and multimodal outputs make this a useful extension beyond scientific code-generation tasks. This supports acceptance because it provides a setting for evaluating decisions that matter in actual scientific work.

  1. Evidence auditing makes failures more observable.

The tools distinguish listing sources from retrieving their contents, and the evaluation reports answer, evidence, and compliance outcomes separately. Appendix gives a concrete example of incorrect scientific interpretation despite full reference coverage. This supports acceptance because the benchmark can help distinguish missing evidence from errors occurring after access, rather than reporting only final success.

Critical weaknesses
  1. The link between the scoring rules and the claimed capabilities needs stronger validation.

It is unclear how explicitly agents receive the scientific priorities behind reference decisions. For example, Inspect Picks involves a tradeoff between removing unwanted particles and retaining useful ones. Without representative original instructions, it is difficult to tell whether disagreement reflects poor judgment or a different interpretation of the objective.

The evidence check counts how many reference groups an agent accesses, but does not distinguish essential evidence from supporting context. An agent can therefore pass the 50% threshold while missing information needed for the decision. Coverage can also increase without a corresponding increase in scientific information Since the evidence check affects overall success rate, these cases raise a concrete question: does the metric reward acquisition of the information needed for a sound decision, or mainly access to the necessary sources chosen by the annotators?

  1. The analysis does not yet establish enough added insight into agent limitations.

ScienceAgentBench already separates execution success from scientific correctness, evaluates intermediate stages, and studies expert guidance. AriadneBench adds decisions over workflow snapshots, but its findings remain largely performance differences across task families. Poor particle selection, for example, could arise from misread images, missing knowledge, unclear goals, or poor evidence integration. Full reference coverage does not distinguish these causes.

Similarly, the skill results do not establish whether guidance improves evidence discovery, interpretation, or action selection. It is also unclear how often upstream history and branches are necessary to solve tasks; graph counts alone do not show this. Most tasks do not execute the chosen action and observe its consequences, limiting the claim of “adaptive autonomy.” This matters to my decision because the paper's distinctive design has not yet produced a clear account of what agents struggle with beyond the particular cryo-EM tasks.

Questions for authors
  1. How are scientific priorities communicated to agents, and how are reasonable alternative decisions scored? Please include an example task prompt.

  2. Can agents pass the 50% coverage threshold while missing critical evidence? How would accounting for such omissions affect the results?

  3. Which results show that workflow history is necessary, or identify the failures that skills correct, beyond the findings of ScienceAgentBench?

  4. Which workflow snapshots, annotations, scoring tools, and environment components will be released for independent evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.