acceptodds
Under review as a conference paper at ICLR 2027

ParticleBench: Evaluating LLM Agents on Experimental Particle Physics Tasks

14 views

Abstract

We introduce ParticleBench, a benchmark for evaluating large language model (LLM) agents on computational tasks in experimental particle physics. The benchmark focuses on long-horizon workflows that combine domain-specific reasoning, code development, and iterative numerical experimentation. Agents receive task specifications, data, and access to scientific computing tools, and must develop executable analysis procedures rather than provide only textual answers. Tasks are defined by computational objectives together with explicit requirements for the validity of their results. Evaluation separates correctness from performance: task-specific verifiers check whether submitted solutions satisfy prescribed numerical and physics-based requirements, while scoring measures analysis quality or computational efficiency among solutions that meet these requirements. Final submissions are frozen and executed on held-out data, with evaluation samples excluded from solution development and optimization. This protocol assesses whether improvements remain valid beyond the data available to the agent during development. ParticleBench provides an executable evaluation setting for studying how LLM agents implement and refine multi-step scientific analyses, with an emphasis on verifiable task completion under experimentally meaningful constraints.

Primary area:
datasets and benchmarks
Keywords:
AI for science, LLM agents, experimental particle physics, long-horizon tasks, scientific computing, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_vpcBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ParticleBench assesses whether AI agents can create particle-physics analysis programs that transfer from a training study to a blind test involving a hidden truth. It offers ten task categories, expert oracles, verifier gates, and allows up to ten submissions before finalizing the program. Five model and harness systems are tested, with Astra earning the highest oracle-relative score. However, some reported tasks do not adhere to the proposed blind protocol, and the paper's claims regarding model quality depend on unblinded LLM ratings.

Critical strengths
  • The blind exam protocol is a valuable benchmark idea. Freezing the agent’s program before generating the exam instance tests whether the resulting procedure transfers to a new truth rather than merely fitting repeated verifier feedback. This addresses an important weakness of many agent benchmarks.

  • The benchmark aims to evaluate executable scientific procedures rather than final answers alone. Program submission, gate-before-score evaluation, execution isolation, expert oracles, and persistent verdict ledgers form a promising foundation for a more realistic scientific-agent benchmark.

Critical weaknesses
  • The blind-exam protocol is not consistently applied. In hhb and jhb, the official result comes from the best study submission, and jhb uses the same draw for study and evaluation. Hidden truth was also reachable in four task families, although the authors found no evidence that agents accessed it, and the reported runs used versions that differ from the released benchmark.

  • The cost and modeling analyses do not support the paper's broader conclusions. The benchmark limits agents to ten submissions and 24 hours, but it has no common token budget and reports no tool-use cost or cost per successful exam, despite presenting expensive feedback as a central motivation and stating that its ledger supports cost-of-pass analysis. The paper also overlooks relevant work on cost-aware evaluation of scientific agents [1]. The modeling-quality claim rests on correlations across only five model-level averages using retrospective, unblinded LLM ratings that were not systematically checked by humans.

[1] SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs. Cao et al., ICML 2026.

Questions for authors
  • How much do the results change when every task follows the same blind protocol, with the final program evaluated on a fresh draw and hidden truth fully inaccessible?

  • Since experimental cost is central to the paper's motivation, can the authors report token usage, tool runtime, and cost per successful exam, and explain how ParticleBench differs from cost-aware benchmarks [1] (see Weakness 2)?

  • Does the reported relationship between modeling quality and exam performance remain when the trajectories are evaluated by multiple blinded human experts rather than unblinded LLM raters?

Reviewer_ginbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ParticleBench, a scientific LLM benchmarking suite composed of data analysis tasks drawn from experimental particle physics. The authors highlight the "real-to-sim-to-real" loop: using high-fidelity simulations of an experiment to design robust analysis procedures that will generalize to real experimental data. The benchmark consists of ten tasks inspired by real particle physics experiments, but uses a simulation-only framework (no experimental data). Agents develop analysis code to perform each task, and are given limited access to a validator program that evaluates their solutions on a fixed simulation configuration. Their final submissions are scored against a simulation with freshly-drawn parameters. LLM performance is measured relative to an expert-written oracle solution, and their implementations are coarsely scored on six scientific procedural axes. The benchmark results indicate that frontier LLMs such as Astra and Fable perform well, achieving oracle-relative scores of ~75%.

Critical strengths
  • The paper discusses and attempts to address a relevant and important problem in LLM benchmarking: capabilities on long-running, open-ended scientific tasks where there may be domain shifts between simulation and reality
  • Generally speaking, the high-level structure of the benchmark is sound. It includes a nice diversity of tasks (calibration, parameter extraction, hypothesis testing, etc.) drawn from a wide array of subfields of particle physics
Critical weaknesses
  • The paper lacks any substantial detail about how the benchmark is implemented (simulation details, agent/judge prompts, detailed task descriptions, etc). Neither the main text nor the appendices contain enough information to understand what was done. There is no code linked or included in the supplementary material. This does not seem to be a space issue, as the authors only use 8 of the 9 available pages. There are no details about how numerical scores are computed, or even what the specific expected numerical outputs are for each task. The absence of any clear or detailed explanation of methodology is the paper's central flaw, and makes it very difficult to assess the benchmark's quality.
  • The text includes what appear to be AI hallucinations: there are isolated references to "Q1" and "Q2-Q4" that refer to nothing that exists in the text. These are either fully hallucinated, or they are slop that has been transferred over from conversations / agentic coding sessions the authors may have been using to implement their benchmark and write the paper.
  • The AI use statement notes that AI was used to create some of the oracle solutions (namely Astra, which is also one of the LLMs tested on the benchmark). This seems like an obvious methodological flaw
  • The introduction emphasizes a desire to address "real-to-sim-to-real" loop in LLM benchmarking, but the proposed benchmark doesn't quite line up. Agents are asked to perform data analysis tasks on data obtained from a simulated experiment. The authors seem to assume that they will choose to tackle these tasks by writing their own simulations of the experiments, but it is not clear from any of the results that they do this. They are, however, scored on whether or not they do this. The fact that the authors omit the prompts and source code makes it impossible to understand this inconsistency. The text does not sufficiently explain what is going on
Questions for authors
  • How are any of the scores computed? I understand that the score is measured relative to the oracle, but it is unclear how a numerical score for the oracle or for any of the LLM submissions is computed
  • The "scientific procedure" scoring (sec 5.1, table 3) is incoherent. What do these different categories refer to? What rubric is the LLM judge given for creating these scores? Do these different scoring categories (detector modeling, component analysis, etc.) correspond to specific things the agent is instructed to do or consider in developing its solution, or is the agent expected to do these things on its own?
  • It is fundamentally unclear what the agents are even being asked to do in this benchmark. For example, part of the "scientific procedure" scoring is "D: detector modeling". Are the agents instructed to create their own model of the simulated experiment? The text reads as if the agents are simply given data from a simulated experiment and asked to perform some data analysis/calibration task with it. There is no obvious suggestion that they are instructed to create their own model. Why, then, are they scored on a nebulous "detector modeling" requirement?
Reviewer_C1cSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that when experimental data are too scarce to learn from directly, a research agent must build a simulation of the experiment, calibrate it against measurements, and test hypotheses in it before using the instrument (the paper calls this the real-to-sim-to-real loop). It presents a particle-physics benchmark meant to measure this ability. Its argument that the benchmark measures it is that grader feedback is scarce, so an agent "should" build its own simulation to check its work.

The benchmark has ten task families modelled on seven experiments. The environments are Monte Carlo surrogates of those experiments built by the authors with AI coding assistants, not the experiments’ official simulations. Each family ships a generator with hidden truth, a verifier made of pass/fail gates followed by a score, and a reference solution. The agent submits an analysis program up to ten times on one draw and sees which gates passed and, when all pass, the score. The final program is graded once on a second draw from the same generator with a different seed. The agent is given the public configuration of the physical process and detector, while seeds, hidden parameters, and truth are withheld. Gates beyond contract checks on execution and output validity exist in three families. The verifier is checked by running the reference solution, a wrong solution, and a truth-reading solution through it. The text presents truth redrawn after the program is frozen as what sets the benchmark apart, while its own comparison table marks the FAIR Universe challenge as already having a held-out evaluation with redrawn truth and a re-run program.

Five model–harness systems were run three times per family. The best reaches 78.8% of the reference score, and no system matches the reference in aggregate. For two families, the reported number is the best study submission rather than the exam, and all runs used task versions that differ from the release. The experimental setup and the Table 2 caption do not say so. The reference solutions for three families contain agent-written code. A second finding is that LLM-rated modelling quality correlates with the exam score more strongly than LLM-rated procedure does, over five points. Note that the rating LLMs were not blind to the exam score, and the paper calls the comparison descriptive.

Critical strengths
  1. The problem is real, and the choice of domain is justified. The starting observation holds: agents have advanced in mathematics where every candidate is cheaply checkable, and in experimental science the checker is the experiment itself. Evaluating agents on scientific measurement, with a procedure rather than an answer as the graded object, is an important and active topic. Particle physics, with a community simulation stack validated against data over decades, is a defensible place to build such a benchmark.

  2. The proposed protocol and its metric details are designed against the ways a science benchmark can be passed without doing the measurement. In the proposed protocol, the submission is a program the grader re-runs. Feedback is gate outcomes with a score only when every gate holds, so an output that fails a gate earns nothing. What the benchmark grades are and why are introduced in detail; for example, three gate kinds (contract, performance, statistical) are named. Hidden parameters can be shifted from published values "to distinguish measurement from recall".

  3. The per-task results and the provenance and limitation disclosures are specific. Results are given per family with pass counts. The spread is wide enough to be informative: ghb runs from 0% to 98.08% and prb from 0% to 85.60% across systems. The paper describes a ledger that records every submission's verdict, gate vector, score, and timestamp, including failed evaluations. The AI use statement locates each point at which model output entered the artifact.

Critical weaknesses
  1. The paper does not test whether the exam score depends on an agent building, calibrating and using a simulation of its own. Without that test a high score cannot be read as closing the loop the title names. The main text links score and loop by conjecture: rationed feedback "should drive the agent to build its own simulation" (lines 069–070). The exam is a second draw of the same generator with a new seed. The agent is given the public configuration of the physical process and detector. How much of the forward model that supplies is not stated. The closest measurement, the Section 5.2 modelling rating, scores whether final programs "effectively represent the key relationships" of the task (lines 400–401), not whether a simulation was built, calibrated and used. The only planned check of whether skipped analysis steps change the score, the SOP-skip mutants, is "defined but not yet run".

  2. Table 2 was produced under protocols that differ from the one the paper presents as its method. Table 2 does not consistently implement the proposed protocol, and the exceptions are disclosed rather than removed, so the method contribution and its evidence do not match. Section 4 states that the runs follow "the study-exam protocol described in Section 3" (lines 306–307). Section 6 states that hhb and jhb report the best study submission and that jhb grades on the study draw (lines 424–425). The reproducibility statement adds that every run used an "owner version" differing from the release in "data, submit policy and wall clock" (lines 469–470). For jhb the reported grade is thus the best attempt on the feedback draw. No agent-performance result under the released protocol is reported. This is concerning because Table 2 shows that agents scored below the reference under mixed protocols, not the gap under the blind-exam protocol or the difference from Table 1's benchmarks.

  3. The modelling-quality ratings behind the second finding were produced with the exam score already known. The analysis therefore cannot separate a real association between modelling and score from the effect of the rater knowing the score. Section 6 lists the ratings as "retrospective single-LLM judgments, unblinded to score and not systematically validated by humans" (lines 430–431). While Section 5.1 describes the rater as an "LLM judge" working from trajectories, code, execution results, and validation evidence and does not say the score was among its inputs. Section 5.2 reports the association, and the contribution list presents it as a main finding.

Questions for authors
  1. Is the exam score conditional on building a simulation?
  2. Which rows of Table 2 were produced under the protocol stated at lines 306-307, and what are the exam-draw scores of all five systems on the released copies?
  3. How much of the association remains when the ratings are repeated without the exam score in view?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.