acceptodds
Under review as a conference paper at ICLR 2027

XRDBench: Benchmarking AI Agents on Experimental X-ray Diffraction Analysis

Abstract

Frontier multimodal large language models (MLLMs) increasingly demonstrate strong scientific-reasoning and coding capabilities, but whether these capabilities translate into reliable execution of domain-specific experimental workflows remains unclear. We investigate MLLM agents for X-ray diffraction (XRD) analysis, a widely used materials-characterization technique whose interpretation remains labor-intensive and expertise-dependent. We introduce XRDBench, a benchmark of 14 tasks organized into four tiers spanning peak-level perception, scientific unit analyses, multi-step characterization, and complex workflows such as phase identification, quantitative phase analysis, and Rietveld refinement. The tasks use predominantly expert-labeled experimental thin-film and powder data and are executed through a sandboxed agentic harness that provides multimodal inputs and access to scientific and crystallographic tools. We evaluate five proprietary frontier models and one open-weight model using task-specific outcome metrics and seven-axis process rubrics. We find that frontier models demonstrate strong XRD knowledge, although no single frontier model consistently outperforms the others, while the evaluated open-weight model remains behind. Removing visual access generally degrades performance, emphasizing the multimodal nature of XRD analysis. Agentic analysis may be particularly valuable for high-throughput and autonomous laboratories, where jointly analyzing related patterns across a compositional gradient reduces token use per pattern without accuracy loss. These findings highlight both the current limitations and the potential of agentic materials characterization.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Agentic, AI4Science, Scientific Reasoning, X-ray Diffraction, Materials Science

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yskrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces XRDBench, a benchmark for evaluating AI agents on experimental XRD analysis. It covers 14 tasks using experimental datasets and an agentic environment with scientific tools. The authors evaluate several frontier models and investigate the effects of visual inputs, reasoning effort, and joint analysis of related patterns.

Critical strengths
  • I appreciate the authors emphasize on real-world experimental data and executable scientific workflows, rather than individual knowledge questions. The task organization and incorporation of certified ground truths make the benchmark potentially useful to the community.

  • The inconsistent benefits of additional reasoning effort and the improved efficiency from jointly analyzing related diffraction patterns are interesting. The latter suggests that scientific agents may benefit from exploiting structure shared across experiments.

Critical weaknesses
  • AutoXRD already studies autonomous XRD analysis using executable workflows and experimental powder data. While XRDBench expands dataset coverage and evaluation dimensions, the paper does not sufficiently establish what fundamentally new capabilities or challenges it introduces. A more concrete comparison would strengthen the contribution.

  • A central motivation is evaluating whether general-purpose AI agents can perform meaningful scientific analysis. However, without direct comparisons against established XRD analysis pipelines (e.g., Dara), it is difficult to assess their practical competitiveness or added value. This is especially important given the substantial computational costs of agentic execution.

  • The claim that there is no significant "knowing-doing gap" is not fully convincing. Process scores are often above 0.9 even when phase identification and quantitative refinement remain substantially inaccurate. Correlations between process scores and performance do not necessarily establish that agents can reliably translate scientific understanding into correct execution. Moreover, the process judge is validated on only 10 trajectories, leaving some uncertainty about the reliability of these scores.

Questions for authors
  • What are the key differences between XRDBench and AutoXRD, beyond broader task and dataset coverage?

  • How do AI agents compare with established pipelines like Dara in accuracy, efficiency, and practical value?

  • Why are process scores consistently high even when outcome accuracy is relatively low? Does this contradict the claimed absence of a "knowing-doing gap"?

  • Does joint analysis genuinely exploit scientific relationships across patterns, or mainly reduce redundant computation?

3 comments
AuthorsOpenReview ↗

Preliminary Response

We thank the reviewer for their thorough review of the manuscript. In response to the critical weaknesses and corresponding questions,

  1. There are key differences between our study and AutoXRD. In their framework, the agent’s behavior is constrained and pre-defined, while our harness allows the model to write any code, execute, observe and act again. Too many predefined actions make the framework hard to generalize to novel scenarios; more exploration space as in our harness allows for more room for training in the future. Further, our analysis of reasoning traces show that the models generally know how to solve these tasks, and a predefined action space is not necessary. In this work we exclusively evaluate multimodal models and study the significance of multimodality, while AutoXRD does not investigate the importance of vision for these XRD tasks. Finally, our expert-labeled datasets include closed source experimental sources that models have not been exposed to, while theirs are mostly synthetic or from older, open source datasets.

  2. Figure 5c directly compares the performance of agents with Dara. The expert labels come from the precursor genome dataset, with phase labels and corresponding Rwp coming from Dara fits reviewed and/or further refined by an expert. This shows that frontier agents approach the performance of the expert + Dara labeling, while costing more. When comparing one of the frontier models (GPT-5.6-Sol at maximum effort) with the Rwps from Dara + human, we find that 68 times out of 100 the agent scores lower than Dara + human. This again shows that the agent is not significantly outperforming current open source, high-throughput methods quite yet. This finding will be included in figure 5 in the revised version.

  3. We agree that low outcome accuracies co-occur with high process scores, for example with the exact-set rate for the Phase ID task only achieving around 30% while the process score approaches 1. There can be three causes of this: (1) a knowing-doing gap, where the agent knows what needs to be done but does not execute on it, (2) the process rubrics are too lenient and should require more components in the rubric to achieve a full score, (3) the metrics are too harsh and do not accurately catch adequate performance. We would therefore like to amend the statement to “These results provide evidence that the rubrics capture task-relevant reasoning. There is potentially a knowing-doing gap, but more on low-level execution instead of understanding of the general workflow to solve the problem.” Additionally, the revised version will include further calibration of the LLMs as judge, with 36 trajectories reviewed by human experts across model, process score, dataset, and task complexity.

  4. With further analysis of the plate vs. spot results for the thin-film data, we find that across GPT-5.6-Sol efforts, the agent more often misses a phase that is there in the “plate” runs, while it more often predicts an extra phase in the “spot” runs. This could suggest that with more context and XRD patterns, the agent restricts the phase space that it searches for. We have added to the text, “Across low, medium, xhigh, and max efforts, the “plate” runs miss a correct phase 5, 4, 4, and 3 times versus 3, 3, 2, and 3 times for the “spot” runs. The “plate” runs predict an extra phase 4, 5, 6, and 9 times across runs versus 11, 15, 12, and 14 times for the “spot” runs.”

Thank you for your response. I appreciate the clarifications regarding the differences from AutoXRD, the comparison with Dara, and the additional plate-vs-spot analysis.

However, several of my main concerns are partially addressed. The practical benefits of agentic approaches against established pipelines is still not sufficiently addressed, and the distinction from AutoXRD is still largely qualitative. More importantly, the discrepancy between high process scores and relatively low outcome accuracy still raises questions about the claimed absence of a knowing-doing gap. The proposed additional judge calibration and revised interpretation would help address this concern.

Finally, while joint analysis improves efficiency, the evidence that agents genuinely exploit scientific relationships across patterns is limited. I encourage the authors to incorporate the promised analyses and appropriately qualify their conclusions. For now, my overall rating is unchanged.

AuthorsOpenReview ↗

Further Response

We thank the reviewer for their response. We have posted our revised version of the manuscript, and provide additional information here.

One of the main distinctions between this text and AutoXRD is the additional study of the harness we provide. We point the reviewer to newly added Figure 9, where we evaluate the performance of Opus 5 and GPT-5.6 Sol in different agentic harnesses. Further, Tables 8, 11, and 16, give the metrics and pass rates of all tasks for models in the neutral harness, Pi. This is compared with the equivalent tables of models in their native harness, and provides a detailed study of the impact of the harness on performance. Further, our vision ablation study of GPT-5.6 Sol, GPT-5.6 Luna, and Opus 5 identifies the importance of multimodality in these tasks, and sets us apart from AutoXRD.

We point the reviewer to Appendix D where we provide a further evaluation of the LLM judge.

On the practical benefit of agentic pipelines, we point the reviewer to the discussion section. We argue that agentic approaches currently do not beat established pipelines, as evidenced by the cost, time, and poor performance on some tasks that we find. Working with open source models and harnesses provides avenues for lower costs or private workflows, and our addition of Kimi K3 in the revised version shows that open weight models can start to approach the proprietary frontier. Further, agents can be useful for integrating different experimental modalities, literature text, or suggesting next steps in experiments, avenues that current workflows cannot attempt.

Reviewer_KGAbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces XRDBench, a 14-task benchmark for multimodal agents that do experimental X-ray diffraction analysis, ranging from peak detection up through phase identification, quantification, and Rietveld refinement. Tasks run in a sandboxed harness with the scientific and crystallographic tools an experimentalist would use, and the authors evaluate six models on mostly expert-labeled thin-film and powder data, scoring both final answers and the reasoning trace. There are also ablations on visual input and a test of whether analyzing related patterns along a composition gradient jointly helps.

Critical strengths

1.The benchmark targets real experimental XRD workflows rather than one proxy task. The 14 tasks in Figure 1 run from perception up to phase ID, QPA, and full refinement, and Sections 3.2 and Appendix A document real thin-film patterns, the certified QARR mixtures, and expert-reviewed powder refinements. Most prior work tests perception, reasoning, or tool use in isolation; testing them together on real patterns is what makes this valuable. 2.The evaluation is technically solid and shows where agents actually break. Table 1 scores final answers with task-specific metrics rather than trusting an LLM judge, and Tables 5, 8, and 9 make it clear that doing well on simple tasks does not mean a model can identify phases or refine structures reliably. The vision ablation in Figure 7 and the plate-versus-spot comparison in Figure 8 are the kind of concrete results I wish more benchmark papers included.

Critical weaknesses

1.The cross-model claims rest on thin statistical ground, and the harness is entangled with the model family. The central comparison in Section 5.1 covers six models across all 14 tasks and concludes that no frontier model consistently dominates and that Kimi K2.5 trails on most tasks. But most entries in Tables 5, 8, and 10 come from a single rollout per item, sometimes on very small sets (n=7 or n=8 on the thin-film unit tasks). The only repeated-rollout results are in Tables 13–17, which cover just Rietveld refinement and crystal system, and Claude Opus 5 is not in those variance tables at all. On top of that, Section 4 pairs GPT models with Codex CLI and Claude with Claude Code, so a cross-family gap could be the harness rather than the model; Kimi also runs through a chat-completions shim in Codex CLI, which makes its poor showing harder to interpret. To be clear, this does not invalidate the benchmark or the per-task numbers, but I would not lean on the fine-grained ranking claims as they stand. Bootstrap intervals over items, repeated rollouts on a representative subset of each tier, and a common-harness run for the leading GPT and Claude systems would address most of this. 2.The process scores are graded by one of the models under evaluation. Every process score in Tables 6, 7, and 12 was assigned by GPT-5.6 Sol, which is also being evaluated, and it ranks first or second in nearly every cell of Table 6. Self-preference bias is hard to rule out. The validation in Appendix D does not settle it: the ten trajectories all come from GPT-5.6 Luna and GPT-5.6 Sol, and the expert scores them high on average (12.9/14), so we never see whether the judge agrees with the expert on Claude or Kimi traces, or on genuinely bad ones. Since Figures 6 and 9 lean on these process scores, the issue matters. 3.The vision ablation is narrower than the claims built on it. Figure 7 ablates vision only for GPT-5.6 Luna and GPT-5.6 Sol, yet the abstract says removing visual access generally degrades performance. Section 5.2 also admits that the ablation removes the input image and blocks inspection of generated plots at the same time, so the two effects cannot be separated. And the effect varies a lot by model: Luna drops from 0.90 to 0.67 in mean process score, while Sol barely moves (0.93 to 0.92, worse on 5 of 11 tasks). Either scope the claim to what was tested, or add an ablation for at least Claude Opus 5. 4.The ground truth has soft spots, and contamination is never discussed. The thin-film labels come from a single scientist, with no inter-annotator agreement reported. Table 1 notes the powder microstrain labels carry 20–80% error bars and that no instrumental-broadening correction was applied to the powder crystallite-size and microstrain labels, so thresholds like δD < 0.50 and δε < 0.50 grade against noisy references. Separately, QARR has been public since 2001 and the powder data come from a public preprint, so some memorization by frontier models is plausible. A contamination check, or even a rough comparison of public-data versus closed thin-film performance, would make the results easier to trust.

Questions for authors

1.[Clarification] Were the GPT and Claude models ever run in the same harness? If not, could the authors say what Codex CLI and Claude Code actually do differently? Right now I cannot tell how much of the cross-family gap comes from the wrapper rather than the models, and evidence that the main findings survive a shared wrapper would raise my confidence. 2.[Clarification] For Tables 5, 8, and 10: which between-model differences would survive bootstrap intervals over items plus a few repeated rollouts? With one rollout per item, I cannot tell which of the gaps in the current tables are real. 3.[Clarification] The judge validation in Appendix D uses ten GPT-only trajectories that mostly score high. Was this set sampled independently of the evaluation runs, and can agreement be reported for lower-quality traces and for Claude or Kimi traces? This sets how much weight the process-score correlations in Figures 6 and 9 can carry. 4.[Data contamination] QARR has been public since 2001 and the powder data come from a public preprint. Did the authors check whether models simply recall the certified labels, for instance by reproducing published phase sets or weight fractions? Even a rough comparison of public-data versus closed thin-film performance would bound the concern. 5.[Vision ablation] Would it be feasible to run the ablation for Claude Opus 5 and Kimi K2.5, and to separate removing the input image from blocking inspection of generated plots? That would show where the degradation actually comes from and whether the multimodality claim holds outside the GPT family.

2 comments
AuthorsOpenReview ↗

Preliminary Response

We thank the reviewer for their thorough review of the manuscript. In response to the critical weaknesses and corresponding questions,

  1. In this revision, we re-run all evaluation runs in a neutral, open-source, coding agent CLI, Pi. The use of Pi allows us to evaluate all models in a neutral environment and make cross-model comparisons. We find that the backend does not impact the results outside of the variance for each model, and performance does not degrade for the OpenAI or Anthropic models when they are not driven by Codex or Claude Code, respectively, for this task. Further, we find that Kimi-K3 significantly improves on Kimi-K2.5 performance, and that Kimi-K2.5 performance improves when driven by Pi. The paper will include a close comparison of this harness ablation.

  2. We are re-running evaluations to get statistics on a more representative set of tasks in the benchmark, across task difficulty and including the variance of the Opus model. Preliminary bootstrapping on the current results show that some models outperform on a given task, but one model does not outperform across all tasks, as we see in the main text.

  3. A more in-depth evaluation of the LLM judge will be submitted upon submission of the paper at the end of the rebuttal period. This evaluation will include four expert reviewers used to score 36 reasoning traces. These 36 traces will be used to evaluate the LLM judges and choose the judge which scores that most closely correlate with the human scores. The 36 traces will span across tasks, models, and LLM judge process score. We will show how the judge scores correlate with expert scores for the GPT, Opus, and Kimi series.

  4. The thin-film labels come from standard XRD analyses derived from peak fitting and applying common equations. Details of and equations used in the labeling processes are in the appendix. In response to the grain size and microstrain labels, thank you for highlighting this limitation. The powder microstrain labels used in our evaluation are derived from Williamson–Hall fits, same as the thin film dataset. The <0.50 threshold is a chosen evaluation tolerance, not a confidence interval; it measures agreement with the reference estimates and does not establish physical accuracy within 50% or eliminate systematic bias. We will clarify this distinction and verify the provenance and dataset applicability of the stated 20–80% error bars. Finally, while we have sandboxed the agents to limit their web search capabilities, there is evidence in the trajectories that models recall the ground truth of the QARR dataset from memory and report that as the final answer. While this is not repeatedly the case, we are unable to discern if this recollection is used to influence the reasoning traces, in cases where the thinking output is hidden. Due to the small size of the QARR dataset used in this study – 12 samples – we remove this dataset from the study.

  5. Our claim on the vision ablation is indeed stronger than what we tested. The vision ablation study for Opus-5 will be in the revised paper. We believe that blocking inspection of generated plots is the appropriate way to conduct the vision ablation study. This completely removes the vision ability of the model and reduces the task to text-only. If generated images were allowed to be viewed, the model may simply generate the input data and our removal of the image has no impact. In this way we can completely remove vision and compare their performance with and without this modality. We restrict the vision ablation to Opus and GPT as these are the highest performing models at the time of evaluation.

AuthorsOpenReview ↗

Additional Comment

We have posted our revised version of the manuscript, and provide additional information to our previous comments.

We point the reviewer to Figure 9, which shows no significant difference between GPT-5.6 Sol and Claude Opus 5 when they are in their native harness or in a neutral harness. The appendix now includes Tables 8, 11, and 16, which give the metrics and pass rates for complex and non-complex tasks for models in the neutral harness. Still, we do not see one model outperform all the others. For runs in the neutral harness or in the native harness, we provide bootstrapping results in Figures 11-14 and 16-17 in the appendix to provide further statistical analysis on our results.

We point the reviewer to Appendix D where we provide a further evaluation of the LLM judge.

We point the reviewer to the updated Figure 7, where we provide the vision ablation study for Opus 5. We have revised our claims in the abstract to say, “Removing visual access degrades the performance of GPT-5.6 Luna, while GPT-5.6 Sol and Opus 5 remain relatively robust to vision removal.” The Opus 5 results follow with our previous results, that large models are capable at solving these tasks when the vision modality is removed.

Reviewer_xyoXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

XRDBench introduces a benchmark for evaluating multimodal large language model agents on experimental X-ray diffraction (XRD) analysis. The benchmark covers 14 tasks spanning peak detection and interpretation, crystallite-size and strain estimation, lattice characterization, phase identification, quantitative phase analysis, and Rietveld refinement. It combines experimental powder, thin-film, and quantitative phase analysis round-robin data with a sandboxed execution environment that provides access to scientific Python libraries and crystallographic software. The authors evaluate five proprietary model configurations and one open-weight model using task-specific outcome metrics, alongside a seven-axis rubric for assessing the quality of execution traces. They also investigate the effect of visual input, reasoning effort, and joint analysis of related diffraction patterns. The reported results indicate substantial but inconsistent capabilities across models, generally lower performance from the evaluated open-weight model, and potential efficiency gains from jointly analyzing patterns along a compositional gradient.

Critical strengths
  1. XRDBench addresses an important gap between evaluating scientific knowledge through question answering and evaluating the execution of practical characterization workflows. The inclusion of peak analysis, structural characterization, phase identification, quantitative phase analysis, and Rietveld refinement gives the benchmark a useful breadth. The use of experimental diffraction patterns, including certified phase-fraction data from the QARR round robin, is particularly valuable because it moves the evaluation beyond purely synthetic patterns and idealized inputs.
  2. The sandboxed agent environment provides access to established scientific tools, while the task-specific outcome metrics assess whether the resulting analyses are correct. The additional attempt to evaluate measurement, physical reasoning, tool selection, code execution, and answer integrity is potentially useful for diagnosing why scientific agents fail. The investigation of joint analysis of related patterns also offers a practical direction for improving the efficiency of high-throughput characterization.
Critical weaknesses
  1. The paper claims that per-item execution trajectories and the evaluation repository are provided, but the supplied archive contains only prompts and rubrics. Without the code, data, annotations, and trajectories, the results cannot be independently reproduced or rescored, undermining the verifiability of the benchmark's main findings.
  2. The reasoning-trace judge is validated against a single expert using only ten trajectories, which is insufficient to establish reliability across all tasks and rubric dimensions. Small task-level sample sizes, reuse of patterns or labels, and model-specific agent harnesses further limit the robustness of cross-model comparisons and claims about the relationship between reasoning quality and scientific correctness.
Questions for authors
  1. The paper states that the supplementary material includes the evaluation repository and per-item execution trajectories. However, these materials do not appear to be present in the supplementary archive available to me, which contains prompts and rubrics. Could the authors clarify where the evaluation code, data, labels, and trajectories can be accessed?
  2. Could the authors clarify how the single-run results in the main tables relate to the three-rollout results in the appendix? For example, the reported GPT-5.4 results for powder crystal-system identification differ between Table 8 (80.8%) and Table 17 (61.5 ± 16.8%). Are these estimates based on the same evaluation setting and aggregation procedure?
  3. The paper organizes tasks into tiers of increasing reasoning complexity and motivates this hierarchy partly by the role of simpler analyses in more complex workflows. Could the authors examine whether per-pattern failures on lower-level tasks predict failures in phase identification or refinement?
  4. Table 1 does not specify a pass criterion for the Vegard’s law task. What tolerance is used to determine whether a predicted composition is correct?
  5. Since the LLM judge receives both the execution trace and the final outcome, could the authors evaluate a subset of trajectories with the outcome hidden from the judge and report how the process scores change? Please also clarify the size and composition of the expert-validation set and how agreement with the expert was assessed.
2 comments
AuthorsOpenReview ↗

Preliminary Response

We thank the reviewer for their thorough review of the manuscript. In response to the critical weaknesses,

1a. The code will be made fully available upon submission of the paper at the end of the rebuttal period.

1b. A more in-depth evaluation of the LLM judge will be submitted upon submission of the paper at the end of the rebuttal period. This evaluation will include four expert reviewers used to score 36 reasoning traces. These 36 traces will be used to evaluate the LLM judges and choose the judge with scores that most closely correlate with the human scores. The 36 traces will span across tasks, models, and process score. We will show how the judge scores correlate with expert scores for the GPT, Opus, and Kimi series.

2a. While some tasks have small sample sizes compared to benchmarks that focus on traditional coding or language tasks, we highlight the exclusive use of experimental data in this benchmark. Many scientific benchmarks only evaluate on simulated or open-source experimental data due to the lack of labeled experimental data available to the community. Our experimental sample sizes are a representation of this limitation across the field. Additionally, the use of experimental data that has not been previously published allows for evaluation on data and crystal structures the model certainly has not seen yet. We’ve found evidence that models can recall the ground truth answer on the public QARR dataset, which has been used in other benchmarks and removed from this paper.

2b. Patterns are re-used in completely different tasks where the agent is separately sandboxed.

2c. In this revision, we re-run all evaluation runs in a neutral, open-source, coding agent CLI, Pi. The use of Pi allows us to evaluate all models in a neutral environment and more strongly make cross-model comparisons. We find that the coding CLI does not impact the results outside of the variance for each model, and performance does not degrade for the OpenAI or Anthropic models when they are not driven by Codex or Claude Code, respectively, for the tasks evaluated here. The revised paper will include these results and a close comparison of this harness ablation.

In response to the questions from the reviewer,

  1. The repository for the code will be publically available with a link shared in the final paper submission.

  2. Table 8 (80.8%) is one rollout. Table 17 (61.5 ± 16.8%) is the mean of three rollouts, including that single rollout. The two later runs scored much lower. The revised version of the paper includes more re-rolls to understand the variance across more tasks.

  3. It is difficult to examine whether per-pattern failures on lower-level tasks predict failures in phase identification or refinement because there is minimal, if any, overlap between the patterns used for these tasks. The patterns in the lower-level tasks consist of single-phase samples while those in the more difficult tasks consist of multi-phase samples, by design. Additionally, the data and context provided to the models vastly differ between these two tasks, making it difficult to compare if a unit task can predict success or failure on an exploratory task. This could be explored by prompting the agents to solve individual unit tasks, while they solve the exploratory tasks. This is out of the scope of this work.

  4. The Vegard’s law task is scored by the MAE between the predicted and ground truth for a given spot. A “pass” is if this MAE falls below 0.075 for a given spot. This has been added to Table 1.

  5. We run the phase ID, phase quantification, and rietveld refinement tasks without giving the judge the score for the task. We find that hiding the outcome barely changes the process score from Opus-5 or GPT-5.6-Sol. The average shift is 0.02 points out of 14 for Opus 5 and 0.05 for GPT-5.6-Sol. In our current framework, it is not possible to fully remove the answer from the trajectory for judge grading. The answers are often mentioned in turns before the final response turn, and this would consist of truncating the trajectories in arbitrary ways. Additionally, one of our rubric axes grades on the alignment between the final answer and the trajectory, so this would require changing the rubric itself.

AuthorsOpenReview ↗

Additional Comment

We have posted our revised version of the manuscript, and provide additional information to our previous comments.

With this version, we provide the code needed to run the evaluations and the datasets used in the paper.

We point the reviewer to Appendix D where we provide a further evaluation of the LLM judge.

We point the reviewer to Figures 11-14 and 16-17 in the Appendix which detail the bootstrapping results. We still do not see one model outperform all the others.

We point the reviewer to Figure 9, which shows no significant difference between GPT-5.6 Sol and Claude Opus 5 when they are in their native harness or in a neutral harness. The appendix now includes Tables 8, 11, and 16, which give the metrics and pass rates for complex and non-complex tasks for models in the neutral harness. Still, we do not see one model outperform all the others.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.