- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper evaluates general-purpose AI agents on 180 tasks spanning eight task families derived from 13 curated cryo-EM workflows. After the initial evaluation, it uses 143 tasks for skill induction, extracting reusable guidance for each task family, and reevaluates agents on the remaining 37 test tasks. It also compares performance under expert supervision.
Critical strengths
This is an early benchmark effort for AI agents in cryo-EM. The data and experiments are presented clearly, the selection of agents is reasonably comprehensive, and the decomposition of cryo-EM workflows into task families is insightful.
Critical weaknesses
The main concern is the lack of publicly accessible benchmark data. Although the appendix lists raw cryo-EM dataset identifiers, I could not find the job-level data needed to inspect the inputs, outputs, and evaluation setup. This limits both scrutiny of the benchmark’s validity and its reuse for developing specialized agents. A benchmark should provide either an accessible, reusable resource or substantive insights that expose limitations in existing benchmarks and challenge prevailing assumptions. In its current form, the paper does not sufficiently deliver either contribution, despite its clear experimental presentation.
Questions for authors
- “Training tasks,” Section 4.2. This term may imply model-parameter updates, whereas these tasks are used for skill induction without such updates. Please use terminology that makes this distinction clear.
- Potentially misleading highlighting, Table 2. The pale-green highlighting of IS may suggest that it performs best among AE, IS, EL, and ES. However, ES achieves higher Score and Success on many tasks. Please revise the formatting to avoid this implication.
- Definitions of EL and ES, Section 4.3. EL (“evidence list”) and ES (“evidence + steps”) are insufficiently explained, particularly the meaning of “evidence.” Appendix D.5.1 does not fully resolve this ambiguity. Please clarify these settings in the main text or provide concrete examples in a clearly referenced appendix.
- Placement of skill induction, around line 363. Skill induction is a central methodological component, yet it is introduced only briefly in the experiments. Please describe it in greater detail in Section 3, together with IS, EL, and ES, before presenting the corresponding experiments.
- Inconsistent terminology, lines 121–123. “Workflows,” “candidate groups,” and “workflow groups” appear to refer to the same concept. Please clarify any distinctions or use consistent terminology throughout.
- Workflow selection, Section 2.1. The screening procedure is described too briefly. Please provide a detailed account of the selection process or refer readers to the relevant appendix.
- Definition of “Branches,” Table 3. I am not aware of “Branches” as a standard cryo-EM term. Please define it and explain its meaning in this benchmark.
- Distinguishing workflows 1 and 6, Table 3. Both are labeled “Filament” and “In-house.” Without access to the underlying data, readers cannot meaningfully distinguish them.
- Representativeness of the selected workflows. Please justify the coverage of the 13 workflows, including statistics on molecular weight, symmetry, signal-to-noise ratio, heterogeneity, and preferred orientation. Basic metadata can be checked for public EMPIAR datasets, but corresponding information for the in-house datasets is inaccessible to readers.
- Validity of single-step reference answers. A cryo-EM decision may have several valid answers depending on the downstream objective, and its benefits may emerge only over subsequent steps. Without access to the benchmark data, readers cannot assess whether the reference answers are appropriate.