- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a new benchmark for evaluating AI coding agents on 40 machine-learning tasks from the neuroscience domain. The authors state that the main difference from standard machine-learning benchmarks is that the split between training and test data is more structured. Instead of holding out a random subset of examples, the test set is made up of whole units such as a new patient or subject.
During a run, the agent can submit its predictions to a hidden evaluator as often as it likes and get a score back. It never sees the test labels or the evaluation code, so it can use the feedback to improve its solution without direct access to the answers. The authors use an LLM acting as a judge to audit the agents’ runs for various kinds of cheating, including hill climbing on the hidden evaluator score or searching for the answers. Runs found to have cheated are disqualified.
Agents generally have no trouble producing a working solution that the evaluator accepts. However, in most cases they do not reach or surpass the state-of-the-art solutions published for the same tasks. The authors report, anecdotally, that agents perform worse on tasks that require “heavy domain pipelines.”
Critical strengths
The authors benchmark on a large set of real neuroscience problems of significant size rather than toy, synthetic, or small tasks. This makes the results more relevant to how these agents would actually be used. 40 problems is large compared to many other works, particularly since the tasks are large. The evaluation scheme is aware of a key and often mishandled part of supervised machine learning for science: splitting the data so the test set measures the generalization the scientific question requires. An agent counts as successful only if it matches or beats the best published result on a task. Surpassing the state of the art is unambiguously a success, so when an agent clears it, the result means something.
Critical weaknesses
As far as I could tell, the same data is used both for the agent’s intermediate feedback (through the hidden evaluator) and for its final score. Standard ML practice is to run algorithm and hyperparameter search on a holdout (validation) set and then report performance once on an untouched test set. Without this separation, repeated evaluation lets the agent tune toward the test data, as the paper noted and considered cheating. One might expect an agent to know on its own not to overfit by evaluating repeatedly, and instead constructing its own holdout set. However, the prompt doesn’t appear to say this; the agent was instead told to use the evaluation service’s feedback to guide its optimization. I think the most rigorous fix would be to redo the evaluation with a three-way split into training, holdout and test data. The hidden evaluator would score on the holdout set, and the agent would be told its final score comes from a separate test set. I recognize this requires additional runs, but it would considerably strengthen confidence in the reported numbers.
I’d encourage the authors to say more about how neuroscience ML tasks differ from other ML tasks, whether general ones like those in MLE-bench or those in other scientific domains. This bears on whether it makes sense to organize ML-task agent benchmarks by scientific domain at all. One useful analysis would be whether success on this benchmark correlates with success on MLE-bench or similar benchmarks, or whether it produces a different ranking of agents.
The paper attributes some failures to “heavy domain pipelines,” but I wasn’t sure what this means. It seems to combine task size (for example, 3D image segmentation is more computationally demanding than standard 2D) with the need for domain-specific processing steps. Disentangling these strikes me as the most valuable thing this benchmark could teach us, with more rigor in its evaluation. Could the authors score each SOTA solution based on criteria related to "heavy domain pipelines", e.g. how long training/inference take for SOTA, how much bespoke processing there is in the SOTA solution beyond just training a network with a standard architecture? As written, the main result is largely a count showing that most tasks remain unsolved. Many agentic AI benchmarks, including benchmarks for agentic AI in science, are appearing now. It would help readers to understand what this one reveals about scientific tasks, or about what agents succeed and fail at, beyond that number. I would genuinely like to see this paper succeed, and I think a sharper answer to “what did we learn?” would make its contribution much clearer.
It would help to justify the 12-hour budget relative to what these challenges typically require. If some reference solutions need substantially more compute or time, falling short on those tasks may partly reflect the budget rather than the agents’ abilities. Detail of GPU specs is missing (VRAM, cores).
Minor: With a single run per combination, it’s hard to tell how much of the differences between agents, or between tasks, reflects run-to-run variability. Multiple runs, even on a subset of tasks, would help establish how stable the results are.
Questions for authors
- Is there a reason not to use a holdout set for the hidden evaluator that the agent has access to?
- Could you include GPU specs, and discuss whether the computational and time resources are appropriate for the problem scale?
- Why use only pass/fail for beating the SOTA. Another option would be normalize agent score by the reference score to provide some notion of how close to SOTA each task is.
13 comments
Response to W1 and Q1 (2/4): use of the score over the budget, and a no-score control protocol
(continued)
How the agents use the score over the budget. To characterise the use of the score in time, the LLM judge of Section 4.4 read every trajectory a second time and segmented it into steps. Each step corresponds to one change of the pipeline or one tuning decision and is assigned to one of fourteen types, which the revised Appendix I defines together with the reading instructions. Seven types modify the method on the agent's own development data, which Figure 7 of the revised paper groups into four classes: representation and domain-specific processing, model family, training and scaling, and ensembling and post-processing. Three types keep the method fixed and adjust its parameters according to the score that the evaluator returned, namely hyper-parameters chosen by comparing submission scores, ensemble weights or thresholds swept against the score, and output values adjusted against the score. We refer to these three types as score-driven tuning, in contrast to the seven types in which the choice is made on the agent's own development data. The remaining four types cover the baseline, changes of the agent's own validation scheme, bug fixes and other steps. Figure 7 of the revised paper reports the resulting proportions by quarter of the budget and by configuration. Score-driven tuning constitutes 20% of all recorded steps. Their proportion increases from 9% of the steps taken in the first three hours to 35% of those taken in the last three, whereas the proportion of changes to representation and model family decreases from 35% to 10% over the same period, and 86% of runs contain at least one score-driven tuning step.
The configurations differ in degree rather than in nature. Claude Opus 5 exhibits the smallest proportion of score-driven tuning steps (12%) and the largest proportion of training and ensembling on its own data, GPT-5.6-sol the largest proportion of score-driven tuning steps (38%) and the smallest proportion of training, and DeepSeek-V4-Pro, most of whose reaches belong to category B, the second largest proportion of score-driven tuning steps (22%). Table 10 lists these proportions per configuration. Figure 8 presents two reference-reaching runs in full. The run of category A reaches the reference at its fifteenth submission, in the tenth hour, after replacing ridge regression with a three-dimensional convolutional network, introducing age-balanced training and per-study calibration, and enlarging the ensemble of seeds. The run of category B reaches it within a greedy forward selection over cached models in which every candidate is scored on the evaluator. Both behaviours are what the protocol is designed to record, and the audit distinguishes between them.
A second protocol in which no score is returned. The analyses above establish how the score was used but not what it contributed to the reported results. We therefore performed a complete second set of 200 runs under a protocol, referred to as the no-score protocol, that differs from the paper's in a single respect: the evaluation service never returns a score. The agent may validate its output as often as it wishes, which checks the format and returns nothing else, and may submit it at any time, which registers the current output as the final answer. The most recent submission is scored once after the budget ends, and a run that never submits is scored on its last validated output. For the purpose of analysis, every output that a run validated or submitted (its candidates) was scored after the run had ended, and the agent saw none of these scores. Tasks, data, splits, references, budget, hardware and all other rules are unchanged. Because no holdout is split on the agent's behalf, the protocol additionally records how agents construct one. Every run was audited with the judge and policy of Section 4.4. The clauses that concern returned scores cannot apply, and no run was judged invalid.
(continued below)
Response to W1 and Q1 (3/4): results of the no-score control
(continued)
Table 11 of the revised paper. Results under the no-score protocol per configuration (200 runs, audited). Under this protocol the evaluation service returns no score during the run, and the final submission is scored once after the run has ended. Reached: tasks on which the submitted answer meets the reference. Calls: median number of validations and submissions per run. Candidates: every output that a run validated or submitted. All candidates were scored after the run had ended, and the agent saw none of these scores. Best candidate: the candidate of a run with the highest evaluation score. Within 0.02 P of the best candidate: proportion of runs whose submitted answer scores within 0.02 P of the best candidate. Best candidate submitted: proportion of runs whose submitted answer is the best candidate. P denotes floor-anchored progress, , where is the score on the task's own metric, the score of the trivial predictor and the reference, so that corresponds to the trivial predictor and to the reference.
| Configuration | Reached | Calls | Within 0.02 P of the best candidate | Best candidate submitted |
|---|---|---|---|---|
| Claude Opus 5 | 14 | 16 | 90% | 40% |
| DeepSeek-V4-Pro | 2 | 15 | 82% | 60% |
| GLM-5.3 | 5 | 12 | 82% | 48% |
| Qwen3.8-Max | 3 | 16 | 78% | 48% |
| GPT-5.6-sol | 9 | 74 | 72% | 15% |
| Any configuration / all runs | 15 | 17 | 81% | 42% |
The set of reachable tasks is unchanged. Of the 17 tasks reached under score feedback, 13 are reached without it. Three of the four tasks that are no longer reached end within 5% of the reference, and the two tasks that are newly reached lie at the reference. Of the 23 tasks that no configuration reached under score feedback, 21 remain unreached (Figure 9a of the revised paper). For 31 of the 40 tasks, the best value of P across configurations differs by less than 0.1 between the two protocols. Table 12 of the revised paper lists the per-task scores under both protocols.
Where the per-configuration counts move, the trajectories explain the movement. The counts move in both directions. The GPT-5.6-sol configuration gains most. Under score feedback it submitted approximately one thousand times per run, and its runs that met a reference were mostly invalidated for reconstructing labels from the score. Without a score it still calls the service about five times more often than the other configurations, but every call now carries a new candidate, and it reaches nine tasks. Claude Opus 5, the configuration that submitted least under score feedback, reaches four additional tasks. On each of them its no-score run ends with a neural ensemble where its score-feedback run had ended with a linear model, and three of the four lie at the reference line. DeepSeek-V4-Pro loses most. Six of its nine reaches under score feedback belong to category B, having been obtained by scripted hill-climbing on the score, and without a score it reaches two tasks. Across the three configurations that lose tasks, 12 of the 21 configuration–task pairs that are no longer reached still end within 10% of the reference.
The course of a run without a score. The step reader described above was applied to the 200 no-score trajectories with the same fourteen types, so that Figure 10 of the revised paper can be read against Figure 7. The first quarter of the budget is alike under both protocols: a baseline is established, after which representation and model family are changed. These two classes account for 38% of the steps taken in the first three hours without a score and for 35% with it. From the second quarter onward, the score-driven tuning steps observed under score feedback are replaced by training choices and by ensembling and post-processing on the agent's own data. Together, these account for 54% to 58% of the steps in the middle quarters, compared with 36% to 38% under score feedback. The last quarter is devoted to enlarging the ensemble, re-running the final pipeline and writing the submission note.
(continued below)
Response to W1 and Q1 (1/4): what the score-feedback protocol measures
W1 and Q1. The evaluation set provides both the intermediate feedback and the final score.
We thank the reviewer for this careful reading of the protocol and for the suggestion of additional runs, which we have taken up. We agree that scoring every submission on the evaluation set does not provide the separation between a validation set and an untouched test set that standard practice requires. The protocol was chosen to measure how an agent optimises a method when it receives the exact score of each submission, and the audit of Section 4.4 records how each run used that score. Inspired by this review, we extended the analysis in two directions. First, we examined the recorded trajectories of all 200 runs step by step, in order to characterise what the agents did with the score. Furthermore, we performed a complete second set of runs, 40 tasks × 5 configurations, under a different protocol, referred to below as the no-score protocol, in which the evaluation service returns no score and the final submission is scored once after the run. This protocol provides the separation the reviewer asks for. The results show that the set of tasks on which the published reference is reached is the same under both protocols, whereas the manner in which the agents reach it differs. The revision reports both analyses in two new appendix sections.
What the score-feedback protocol measures. The benchmark measures iteration under exact feedback. The prompt therefore leaves the use of the score to the agent, and the constraint on that use is placed in the audit, which, as defined in Section 4.4 and Appendix E, assigns every run a behaviour category according to how its best score was obtained. In category A, method iteration, the best score results from changes of features, model family, training or preprocessing. In category B, scripted hill-climbing, it results from a scripted loop over ensemble weights, hyper-parameters, thresholds or post-processing that is scored on the evaluator. Runs of category B are valid and are counted in Table 2. Category C denotes a shortcut or degenerate output, and adjusting output values directly against the score renders a run invalid.
Over all 200 runs, the audit assigns 157 runs to category A, 35 to category B and 8 to category C (Table 10 of the revised paper). It judges 24 runs invalid: 8 whose best score had met the reference and 16 whose best score had not. Of the 38 runs that reach a published reference, 28 belong to category A and 10 to category B. We use configuration to denote a backbone model together with its coding harness. When only runs of category A are counted, Claude Opus 5, GLM-5.3, Qwen3.8-Max, DeepSeek-V4-Pro and GPT-5.6-sol reach 8, 8, 9, 3 and 0 tasks respectively, compared with 10, 9, 9, 9 and 1 in Table 2, and 15 of the 17 reached tasks remain reached. A second check, which does not rely on the judge, leads to the same conclusion. We removed every submission belonging to a scripted loop, defined as ten or more consecutive submissions less than five seconds apart, and recomputed the best valid score of each run. All 38 runs and all 17 tasks still reach the reference, and the gain contributed by such loops over the best score outside them has a median of 0% of the reference and a maximum of 0.59%.
(continued below)