- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ParticleBench assesses whether AI agents can create particle-physics analysis programs that transfer from a training study to a blind test involving a hidden truth. It offers ten task categories, expert oracles, verifier gates, and allows up to ten submissions before finalizing the program. Five model and harness systems are tested, with Astra earning the highest oracle-relative score. However, some reported tasks do not adhere to the proposed blind protocol, and the paper's claims regarding model quality depend on unblinded LLM ratings.
Critical strengths
-
The blind exam protocol is a valuable benchmark idea. Freezing the agent’s program before generating the exam instance tests whether the resulting procedure transfers to a new truth rather than merely fitting repeated verifier feedback. This addresses an important weakness of many agent benchmarks.
-
The benchmark aims to evaluate executable scientific procedures rather than final answers alone. Program submission, gate-before-score evaluation, execution isolation, expert oracles, and persistent verdict ledgers form a promising foundation for a more realistic scientific-agent benchmark.
Critical weaknesses
-
The blind-exam protocol is not consistently applied. In hhb and jhb, the official result comes from the best study submission, and jhb uses the same draw for study and evaluation. Hidden truth was also reachable in four task families, although the authors found no evidence that agents accessed it, and the reported runs used versions that differ from the released benchmark.
-
The cost and modeling analyses do not support the paper's broader conclusions. The benchmark limits agents to ten submissions and 24 hours, but it has no common token budget and reports no tool-use cost or cost per successful exam, despite presenting expensive feedback as a central motivation and stating that its ledger supports cost-of-pass analysis. The paper also overlooks relevant work on cost-aware evaluation of scientific agents [1]. The modeling-quality claim rests on correlations across only five model-level averages using retrospective, unblinded LLM ratings that were not systematically checked by humans.
[1] SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs. Cao et al., ICML 2026.
Questions for authors
-
How much do the results change when every task follows the same blind protocol, with the final program evaluated on a fresh draw and hidden truth fully inaccessible?
-
Since experimental cost is central to the paper's motivation, can the authors report token usage, tool runtime, and cost per successful exam, and explain how ParticleBench differs from cost-aware benchmarks [1] (see Weakness 2)?
-
Does the reported relationship between modeling quality and exam performance remain when the trajectories are evaluated by multiple blinded human experts rather than unblinded LLM raters?