- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces XRDBench, a benchmark for evaluating AI agents on experimental XRD analysis. It covers 14 tasks using experimental datasets and an agentic environment with scientific tools. The authors evaluate several frontier models and investigate the effects of visual inputs, reasoning effort, and joint analysis of related patterns.
Critical strengths
-
I appreciate the authors emphasize on real-world experimental data and executable scientific workflows, rather than individual knowledge questions. The task organization and incorporation of certified ground truths make the benchmark potentially useful to the community.
-
The inconsistent benefits of additional reasoning effort and the improved efficiency from jointly analyzing related diffraction patterns are interesting. The latter suggests that scientific agents may benefit from exploiting structure shared across experiments.
Critical weaknesses
-
AutoXRD already studies autonomous XRD analysis using executable workflows and experimental powder data. While XRDBench expands dataset coverage and evaluation dimensions, the paper does not sufficiently establish what fundamentally new capabilities or challenges it introduces. A more concrete comparison would strengthen the contribution.
-
A central motivation is evaluating whether general-purpose AI agents can perform meaningful scientific analysis. However, without direct comparisons against established XRD analysis pipelines (e.g., Dara), it is difficult to assess their practical competitiveness or added value. This is especially important given the substantial computational costs of agentic execution.
-
The claim that there is no significant "knowing-doing gap" is not fully convincing. Process scores are often above 0.9 even when phase identification and quantitative refinement remain substantially inaccurate. Correlations between process scores and performance do not necessarily establish that agents can reliably translate scientific understanding into correct execution. Moreover, the process judge is validated on only 10 trajectories, leaving some uncertainty about the reliability of these scores.
Questions for authors
-
What are the key differences between XRDBench and AutoXRD, beyond broader task and dataset coverage?
-
How do AI agents compare with established pipelines like Dara in accuracy, efficiency, and practical value?
-
Why are process scores consistently high even when outcome accuracy is relatively low? Does this contradict the claimed absence of a "knowing-doing gap"?
-
Does joint analysis genuinely exploit scientific relationships across patterns, or mainly reduce redundant computation?
3 comments
Thank you for your response. I appreciate the clarifications regarding the differences from AutoXRD, the comparison with Dara, and the additional plate-vs-spot analysis.
However, several of my main concerns are partially addressed. The practical benefits of agentic approaches against established pipelines is still not sufficiently addressed, and the distinction from AutoXRD is still largely qualitative. More importantly, the discrepancy between high process scores and relatively low outcome accuracy still raises questions about the claimed absence of a knowing-doing gap. The proposed additional judge calibration and revised interpretation would help address this concern.
Finally, while joint analysis improves efficiency, the evidence that agents genuinely exploit scientific relationships across patterns is limited. I encourage the authors to incorporate the promised analyses and appropriately qualify their conclusions. For now, my overall rating is unchanged.
Further Response
We thank the reviewer for their response. We have posted our revised version of the manuscript, and provide additional information here.
One of the main distinctions between this text and AutoXRD is the additional study of the harness we provide. We point the reviewer to newly added Figure 9, where we evaluate the performance of Opus 5 and GPT-5.6 Sol in different agentic harnesses. Further, Tables 8, 11, and 16, give the metrics and pass rates of all tasks for models in the neutral harness, Pi. This is compared with the equivalent tables of models in their native harness, and provides a detailed study of the impact of the harness on performance. Further, our vision ablation study of GPT-5.6 Sol, GPT-5.6 Luna, and Opus 5 identifies the importance of multimodality in these tasks, and sets us apart from AutoXRD.
We point the reviewer to Appendix D where we provide a further evaluation of the LLM judge.
On the practical benefit of agentic pipelines, we point the reviewer to the discussion section. We argue that agentic approaches currently do not beat established pipelines, as evidenced by the cost, time, and poor performance on some tasks that we find. Working with open source models and harnesses provides avenues for lower costs or private workflows, and our addition of Kimi K3 in the revised version shows that open weight models can start to approach the proprietary frontier. Further, agents can be useful for integrating different experimental modalities, literature text, or suggesting next steps in experiments, avenues that current workflows cannot attempt.
Preliminary Response
We thank the reviewer for their thorough review of the manuscript. In response to the critical weaknesses and corresponding questions,
There are key differences between our study and AutoXRD. In their framework, the agent’s behavior is constrained and pre-defined, while our harness allows the model to write any code, execute, observe and act again. Too many predefined actions make the framework hard to generalize to novel scenarios; more exploration space as in our harness allows for more room for training in the future. Further, our analysis of reasoning traces show that the models generally know how to solve these tasks, and a predefined action space is not necessary. In this work we exclusively evaluate multimodal models and study the significance of multimodality, while AutoXRD does not investigate the importance of vision for these XRD tasks. Finally, our expert-labeled datasets include closed source experimental sources that models have not been exposed to, while theirs are mostly synthetic or from older, open source datasets.
Figure 5c directly compares the performance of agents with Dara. The expert labels come from the precursor genome dataset, with phase labels and corresponding Rwp coming from Dara fits reviewed and/or further refined by an expert. This shows that frontier agents approach the performance of the expert + Dara labeling, while costing more. When comparing one of the frontier models (GPT-5.6-Sol at maximum effort) with the Rwps from Dara + human, we find that 68 times out of 100 the agent scores lower than Dara + human. This again shows that the agent is not significantly outperforming current open source, high-throughput methods quite yet. This finding will be included in figure 5 in the revised version.
We agree that low outcome accuracies co-occur with high process scores, for example with the exact-set rate for the Phase ID task only achieving around 30% while the process score approaches 1. There can be three causes of this: (1) a knowing-doing gap, where the agent knows what needs to be done but does not execute on it, (2) the process rubrics are too lenient and should require more components in the rubric to achieve a full score, (3) the metrics are too harsh and do not accurately catch adequate performance. We would therefore like to amend the statement to “These results provide evidence that the rubrics capture task-relevant reasoning. There is potentially a knowing-doing gap, but more on low-level execution instead of understanding of the general workflow to solve the problem.” Additionally, the revised version will include further calibration of the LLMs as judge, with 36 trajectories reviewed by human experts across model, process score, dataset, and task complexity.
With further analysis of the plate vs. spot results for the thin-film data, we find that across GPT-5.6-Sol efforts, the agent more often misses a phase that is there in the “plate” runs, while it more often predicts an extra phase in the “spot” runs. This could suggest that with more context and XRD patterns, the agent restricts the phase space that it searches for. We have added to the text, “Across low, medium, xhigh, and max efforts, the “plate” runs miss a correct phase 5, 4, 4, and 3 times versus 3, 3, 2, and 3 times for the “spot” runs. The “plate” runs predict an extra phase 4, 5, 6, and 9 times across runs versus 11, 15, 12, and 14 times for the “spot” runs.”