- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the performance of different frontier LLM systems on 12 scientific tasks. They carefully curate a set of tasks for which both a modest and expert baseline solution exist, against which they can assess model performance.
They find significant heterogeneity in performance across tasks.
Critical strengths
- Interesting approach, relying on the existence of both modest and expert solutions per task to gauge model performance is a valuable framing
- These are real world scientific tasks, not contrived
- Evaluations span a range of important models
Critical weaknesses
- The methodology is insufficiently clarified. It is unclear to what extent soft-optimisations are employed (even if unintentionally by the authors) to improve model performance on the tasks. For example, it is widely known that model performance is highly sensitive to harness design & prompt phrasing. It is unclear when exactly the tasks became well-defined, and how divorced was the process of creating those tasks and running the evaluation procedure. For high quality model evaluation work, I would require authors pre-register their tasks before any evaluations are performed. Without a clear pre-registration, I am concerned that there have been many adjustments to the evaluation prompts and harness that are correlated with the test sets within each task.
- while I very much appreciate the focus on real world scientific tasks, 12 tasks is insufficient for a main paper, especially without additional analysis to elucidate why models differ in performance across different tasks.
Questions for authors
What do you see as the main bottleneck(s) in scaling up your work to more than 12 tasks?
We thank the reviewer for careful reading of the paper, please find our comments below.
The paper represents a methodology proposal that can later be used to update the benchmark with new problems. We decided to fix the amount of problems to 12 in order to test the approach in different domains with different amounts of data and information description, thus developing a unified setting for problem verification. Additional problems will be added once methodology is fixed. The increase of the number of problems at this stage is not needed, since the problems are carefully positioned and test different problem-specific peculiarities, for example, abundance of data, or the amount of description available in the internet. Tasks were not formally pre-registered, but tasks, prompts, harness and graders were frozen before all reported runs. Every model then ran under identical conditions: the same agent loop and system prompt, the same harness-side tools (code execution, data exploration, and a format-only validation check that never returns a score), the same isolated sandbox, and the same token and time limits, with no per-model tuning. Ground truth stayed on the host, so nothing could be adapted to the test labels.
The approach is two stage:
We hope that this answer clarifies methodology, We will add this description into the Appendix F.