acceptodds
Under review as a conference paper at ICLR 2027

EvoNeuroBench: Evaluating Coding Agents on Real Research Tasks in Neuroscience

Abstract

Whether large language model (LLM) agents can autonomously develop and iteratively improve neuroscience algorithms is an important question for their effective use in brain research. Successful code execution alone is insufficient because agents must also choose suitable methods, meet experimental requirements, and improve their solutions through feedback. However, these capabilities remain insufficiently evaluated across the diverse tasks encountered in neuroscience. To address this gap, we introduce EvoNeuroBench, a benchmark of 40 executable machine-learning tasks spanning multiple modalities, experimental settings, and biological scales. Each task is grounded in a research dataset, a published study or a public challenge and preserves the scientific objective and evaluation design of its source. Agents work for 12 hours in isolated environments with hidden deterministic evaluators, revising their solutions from quantitative feedback while every submission is retained, so that both final performance and progress through experimentation are observable. Using this framework, we compare five coding-agent configurations under matched conditions and contextualise their performance with published references. Even the strongest configuration reaches the published state of the art on only 25% of the tasks, and 58% of the tasks are reached by none. Success concentrates on problems that translate into standard supervised pipelines, whereas dense clinical segmentation, speech neuroprostheses and cross-site generalisation remain far from the reference. We find that some agents use thousands of submissions to infer evaluation labels from exact score feedback. Post-hoc auditing is therefore needed to distinguish feedback exploitation from genuine algorithmic progress. Together, EvoNeuroBench provides a unified, auditable testbed for how coding agents develop and improve neuroscience algorithms.

Primary area:
datasets and benchmarks
Keywords:
AI for Science, LLM Agents, Benchmark, Neuroscience, Scientific Task, Evaluation, Coding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dGtiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a new benchmark for evaluating AI coding agents on 40 machine-learning tasks from the neuroscience domain. The authors state that the main difference from standard machine-learning benchmarks is that the split between training and test data is more structured. Instead of holding out a random subset of examples, the test set is made up of whole units such as a new patient or subject.

During a run, the agent can submit its predictions to a hidden evaluator as often as it likes and get a score back. It never sees the test labels or the evaluation code, so it can use the feedback to improve its solution without direct access to the answers. The authors use an LLM acting as a judge to audit the agents’ runs for various kinds of cheating, including hill climbing on the hidden evaluator score or searching for the answers. Runs found to have cheated are disqualified.

Agents generally have no trouble producing a working solution that the evaluator accepts. However, in most cases they do not reach or surpass the state-of-the-art solutions published for the same tasks. The authors report, anecdotally, that agents perform worse on tasks that require “heavy domain pipelines.”

Critical strengths

The authors benchmark on a large set of real neuroscience problems of significant size rather than toy, synthetic, or small tasks. This makes the results more relevant to how these agents would actually be used. 40 problems is large compared to many other works, particularly since the tasks are large. The evaluation scheme is aware of a key and often mishandled part of supervised machine learning for science: splitting the data so the test set measures the generalization the scientific question requires. An agent counts as successful only if it matches or beats the best published result on a task. Surpassing the state of the art is unambiguously a success, so when an agent clears it, the result means something.

Critical weaknesses

As far as I could tell, the same data is used both for the agent’s intermediate feedback (through the hidden evaluator) and for its final score. Standard ML practice is to run algorithm and hyperparameter search on a holdout (validation) set and then report performance once on an untouched test set. Without this separation, repeated evaluation lets the agent tune toward the test data, as the paper noted and considered cheating. One might expect an agent to know on its own not to overfit by evaluating repeatedly, and instead constructing its own holdout set. However, the prompt doesn’t appear to say this; the agent was instead told to use the evaluation service’s feedback to guide its optimization. I think the most rigorous fix would be to redo the evaluation with a three-way split into training, holdout and test data. The hidden evaluator would score on the holdout set, and the agent would be told its final score comes from a separate test set. I recognize this requires additional runs, but it would considerably strengthen confidence in the reported numbers.

I’d encourage the authors to say more about how neuroscience ML tasks differ from other ML tasks, whether general ones like those in MLE-bench or those in other scientific domains. This bears on whether it makes sense to organize ML-task agent benchmarks by scientific domain at all. One useful analysis would be whether success on this benchmark correlates with success on MLE-bench or similar benchmarks, or whether it produces a different ranking of agents.

The paper attributes some failures to “heavy domain pipelines,” but I wasn’t sure what this means. It seems to combine task size (for example, 3D image segmentation is more computationally demanding than standard 2D) with the need for domain-specific processing steps. Disentangling these strikes me as the most valuable thing this benchmark could teach us, with more rigor in its evaluation. Could the authors score each SOTA solution based on criteria related to "heavy domain pipelines", e.g. how long training/inference take for SOTA, how much bespoke processing there is in the SOTA solution beyond just training a network with a standard architecture? As written, the main result is largely a count showing that most tasks remain unsolved. Many agentic AI benchmarks, including benchmarks for agentic AI in science, are appearing now. It would help readers to understand what this one reveals about scientific tasks, or about what agents succeed and fail at, beyond that number. I would genuinely like to see this paper succeed, and I think a sharper answer to “what did we learn?” would make its contribution much clearer.

It would help to justify the 12-hour budget relative to what these challenges typically require. If some reference solutions need substantially more compute or time, falling short on those tasks may partly reflect the budget rather than the agents’ abilities. Detail of GPU specs is missing (VRAM, cores).

Minor: With a single run per combination, it’s hard to tell how much of the differences between agents, or between tasks, reflects run-to-run variability. Multiple runs, even on a subset of tasks, would help establish how stable the results are.

Questions for authors
  • Is there a reason not to use a holdout set for the hidden evaluator that the agent has access to?
  • Could you include GPU specs, and discuss whether the computational and time resources are appropriate for the problem scale?
  • Why use only pass/fail for beating the SOTA. Another option would be normalize agent score by the reference score to provide some notion of how close to SOTA each task is.
13 comments
AuthorsOpenReview ↗

Response to W1 and Q1 (1/4): what the score-feedback protocol measures

W1 and Q1. The evaluation set provides both the intermediate feedback and the final score.

We thank the reviewer for this careful reading of the protocol and for the suggestion of additional runs, which we have taken up. We agree that scoring every submission on the evaluation set does not provide the separation between a validation set and an untouched test set that standard practice requires. The protocol was chosen to measure how an agent optimises a method when it receives the exact score of each submission, and the audit of Section 4.4 records how each run used that score. Inspired by this review, we extended the analysis in two directions. First, we examined the recorded trajectories of all 200 runs step by step, in order to characterise what the agents did with the score. Furthermore, we performed a complete second set of runs, 40 tasks × 5 configurations, under a different protocol, referred to below as the no-score protocol, in which the evaluation service returns no score and the final submission is scored once after the run. This protocol provides the separation the reviewer asks for. The results show that the set of tasks on which the published reference is reached is the same under both protocols, whereas the manner in which the agents reach it differs. The revision reports both analyses in two new appendix sections.

What the score-feedback protocol measures. The benchmark measures iteration under exact feedback. The prompt therefore leaves the use of the score to the agent, and the constraint on that use is placed in the audit, which, as defined in Section 4.4 and Appendix E, assigns every run a behaviour category according to how its best score was obtained. In category A, method iteration, the best score results from changes of features, model family, training or preprocessing. In category B, scripted hill-climbing, it results from a scripted loop over ensemble weights, hyper-parameters, thresholds or post-processing that is scored on the evaluator. Runs of category B are valid and are counted in Table 2. Category C denotes a shortcut or degenerate output, and adjusting output values directly against the score renders a run invalid.

Over all 200 runs, the audit assigns 157 runs to category A, 35 to category B and 8 to category C (Table 10 of the revised paper). It judges 24 runs invalid: 8 whose best score had met the reference and 16 whose best score had not. Of the 38 runs that reach a published reference, 28 belong to category A and 10 to category B. We use configuration to denote a backbone model together with its coding harness. When only runs of category A are counted, Claude Opus 5, GLM-5.3, Qwen3.8-Max, DeepSeek-V4-Pro and GPT-5.6-sol reach 8, 8, 9, 3 and 0 tasks respectively, compared with 10, 9, 9, 9 and 1 in Table 2, and 15 of the 17 reached tasks remain reached. A second check, which does not rely on the judge, leads to the same conclusion. We removed every submission belonging to a scripted loop, defined as ten or more consecutive submissions less than five seconds apart, and recomputed the best valid score of each run. All 38 runs and all 17 tasks still reach the reference, and the gain contributed by such loops over the best score outside them has a median of 0% of the reference and a maximum of 0.59%.

(continued below)

AuthorsOpenReview ↗

Response to W1 and Q1 (2/4): use of the score over the budget, and a no-score control protocol

(continued)

How the agents use the score over the budget. To characterise the use of the score in time, the LLM judge of Section 4.4 read every trajectory a second time and segmented it into steps. Each step corresponds to one change of the pipeline or one tuning decision and is assigned to one of fourteen types, which the revised Appendix I defines together with the reading instructions. Seven types modify the method on the agent's own development data, which Figure 7 of the revised paper groups into four classes: representation and domain-specific processing, model family, training and scaling, and ensembling and post-processing. Three types keep the method fixed and adjust its parameters according to the score that the evaluator returned, namely hyper-parameters chosen by comparing submission scores, ensemble weights or thresholds swept against the score, and output values adjusted against the score. We refer to these three types as score-driven tuning, in contrast to the seven types in which the choice is made on the agent's own development data. The remaining four types cover the baseline, changes of the agent's own validation scheme, bug fixes and other steps. Figure 7 of the revised paper reports the resulting proportions by quarter of the budget and by configuration. Score-driven tuning constitutes 20% of all recorded steps. Their proportion increases from 9% of the steps taken in the first three hours to 35% of those taken in the last three, whereas the proportion of changes to representation and model family decreases from 35% to 10% over the same period, and 86% of runs contain at least one score-driven tuning step.

The configurations differ in degree rather than in nature. Claude Opus 5 exhibits the smallest proportion of score-driven tuning steps (12%) and the largest proportion of training and ensembling on its own data, GPT-5.6-sol the largest proportion of score-driven tuning steps (38%) and the smallest proportion of training, and DeepSeek-V4-Pro, most of whose reaches belong to category B, the second largest proportion of score-driven tuning steps (22%). Table 10 lists these proportions per configuration. Figure 8 presents two reference-reaching runs in full. The run of category A reaches the reference at its fifteenth submission, in the tenth hour, after replacing ridge regression with a three-dimensional convolutional network, introducing age-balanced training and per-study calibration, and enlarging the ensemble of seeds. The run of category B reaches it within a greedy forward selection over cached models in which every candidate is scored on the evaluator. Both behaviours are what the protocol is designed to record, and the audit distinguishes between them.

A second protocol in which no score is returned. The analyses above establish how the score was used but not what it contributed to the reported results. We therefore performed a complete second set of 200 runs under a protocol, referred to as the no-score protocol, that differs from the paper's in a single respect: the evaluation service never returns a score. The agent may validate its output as often as it wishes, which checks the format and returns nothing else, and may submit it at any time, which registers the current output as the final answer. The most recent submission is scored once after the budget ends, and a run that never submits is scored on its last validated output. For the purpose of analysis, every output that a run validated or submitted (its candidates) was scored after the run had ended, and the agent saw none of these scores. Tasks, data, splits, references, budget, hardware and all other rules are unchanged. Because no holdout is split on the agent's behalf, the protocol additionally records how agents construct one. Every run was audited with the judge and policy of Section 4.4. The clauses that concern returned scores cannot apply, and no run was judged invalid.

(continued below)

AuthorsOpenReview ↗

Response to W1 and Q1 (3/4): results of the no-score control

(continued)

Table 11 of the revised paper. Results under the no-score protocol per configuration (200 runs, audited). Under this protocol the evaluation service returns no score during the run, and the final submission is scored once after the run has ended. Reached: tasks on which the submitted answer meets the reference. Calls: median number of validations and submissions per run. Candidates: every output that a run validated or submitted. All candidates were scored after the run had ended, and the agent saw none of these scores. Best candidate: the candidate of a run with the highest evaluation score. Within 0.02 P of the best candidate: proportion of runs whose submitted answer scores within 0.02 P of the best candidate. Best candidate submitted: proportion of runs whose submitted answer is the best candidate. P denotes floor-anchored progress, P=(s−s0)/(R−s0)P=(s-s_0)/(R-s_0), where ss is the score on the task's own metric, s0s_0 the score of the trivial predictor and RR the reference, so that P=0P=0 corresponds to the trivial predictor and P=1P=1 to the reference.

ConfigurationReachedCallsWithin 0.02 P of the best candidateBest candidate submitted
Claude Opus 5141690%40%
DeepSeek-V4-Pro21582%60%
GLM-5.351282%48%
Qwen3.8-Max31678%48%
GPT-5.6-sol97472%15%
Any configuration / all runs151781%42%

The set of reachable tasks is unchanged. Of the 17 tasks reached under score feedback, 13 are reached without it. Three of the four tasks that are no longer reached end within 5% of the reference, and the two tasks that are newly reached lie at the reference. Of the 23 tasks that no configuration reached under score feedback, 21 remain unreached (Figure 9a of the revised paper). For 31 of the 40 tasks, the best value of P across configurations differs by less than 0.1 between the two protocols. Table 12 of the revised paper lists the per-task scores under both protocols.

Where the per-configuration counts move, the trajectories explain the movement. The counts move in both directions. The GPT-5.6-sol configuration gains most. Under score feedback it submitted approximately one thousand times per run, and its runs that met a reference were mostly invalidated for reconstructing labels from the score. Without a score it still calls the service about five times more often than the other configurations, but every call now carries a new candidate, and it reaches nine tasks. Claude Opus 5, the configuration that submitted least under score feedback, reaches four additional tasks. On each of them its no-score run ends with a neural ensemble where its score-feedback run had ended with a linear model, and three of the four lie at the reference line. DeepSeek-V4-Pro loses most. Six of its nine reaches under score feedback belong to category B, having been obtained by scripted hill-climbing on the score, and without a score it reaches two tasks. Across the three configurations that lose tasks, 12 of the 21 configuration–task pairs that are no longer reached still end within 10% of the reference.

The course of a run without a score. The step reader described above was applied to the 200 no-score trajectories with the same fourteen types, so that Figure 10 of the revised paper can be read against Figure 7. The first quarter of the budget is alike under both protocols: a baseline is established, after which representation and model family are changed. These two classes account for 38% of the steps taken in the first three hours without a score and for 35% with it. From the second quarter onward, the score-driven tuning steps observed under score feedback are replaced by training choices and by ensembling and post-processing on the agent's own data. Together, these account for 54% to 58% of the steps in the middle quarters, compared with 36% to 38% under score feedback. The last quarter is devoted to enlarging the ensemble, re-running the final pipeline and writing the submission note.

(continued below)

Reviewer_eJnLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces EvoNeuroBench which has 40 neuroscience ML tasks sampled from public datasets and community challenges. Each has an experiment-specific held-out unit (from participant, site, session, stimulus, volume), a deterministic hidden evaluator, and a verified published reference. Five coding-agent configurations are run under a matched 12-hour protocol with score feedback and unlimited submissions, which necessitates each run be audited.

Critical strengths

I found the details about reward exploitation and its mitigation interesting, if not expected.

Results are not saturated and separate cleanly by output form.

Critical weaknesses

Some tasks in the pool simply require multi-session modeling to improve beyond certain levels. While producing a baseline on these tasks may be a challenging task, without longer runs or sparser agent calls spread over time some benchmarks don’t truly measure model capability.

Several EEG/MEG tasks are not just reached but exceeded by wide margins (Lee et al. 2019, Brain Treebank, Grasp-and-Lift, THINGS-EEG2), which usually indicates a weak reference, an easier local split, or leakage. This might be taken as signal to revise the included benchmarks based on a validation run of models.

Questions for authors

Does the THINGS-EEG2 evaluation average over image repetitions? Current number for Opus seems a tad high.

Can the trivial and cheap-baseline scores from the screening step be added to Table 3 or 7?

4 comments
AuthorsOpenReview ↗

Response to W1: the 12-hour budget

W1. Whether a longer budget, or calls spread over a longer period, would raise performance on some tasks.

We thank the reviewer for this comment, which we read as asking whether the 12-hour budget limits what the agents attain on some tasks, so that longer runs, or calls spread over a longer period, would raise their performance. We agree that a fixed budget caps what any agent can attain and that a longer budget might alter the outcome on individual tasks. The budget of 12 hours is nonetheless a deliberate trade-off between comparability and cost. It lies within the range used by comparable benchmarks, and it keeps the evaluation of five configurations on 40 tasks at 2,400 GPU-hours. The recorded trajectories furthermore show that on most tasks progress had levelled off before the budget ended. No further task is reached after hour 10, and on 83% of the tasks that no configuration reaches, the best run had attained at least 90% of its final progress by hour 8.

The choice of the budget. Three considerations determined the budget. First, benchmarks that evaluate coding agents on machine-learning tasks assign budgets of hours (Table 1): 8 hours in RE-Bench (Wijk et al. 2024), 12 in PaperBench (Starace et al. 2025) and 24 in MLE-bench (Chan et al. 2024). These benchmarks share a common design, which we adopt: a budget declared in the protocol, under which every agent is compared on equal terms against an external landmark, Kaggle medal thresholds in MLE-bench and the published reference in our case. A budget of 12 hours lies within this range.

Second, the scale of the evaluation constrains the budget. The 200 runs of 12 hours on one GPU each amount to 2,400 GPU-hours in addition to the agents' model calls, and any extension multiplies this cost for every configuration. Third, 12 hours suffices to observe what the benchmark is designed to measure, namely the result attained under the declared budget and the iterative development of the method by which it is attained. Within 12 hours a run completes the full cycle of development. It starts from an initial end-to-end pipeline, moves through changes of representation, model family and training, including deep networks trained on the raw recordings, and ends with ensembling and post-processing, with the use of the evaluator recorded at every stage.

Progress within the budget. Because every submission is recorded with its time and score, the trajectories show whether the budget ended while the agents were still improving. For the tasks that are reached, taking the best configuration per task, the reference is reached on 9 tasks by hour 6, on 14 by hour 8 and on 17 by hour 10, and no further task is reached in the final two hours (Figure 4a). On the 23 tasks that no configuration reaches, we follow the best valid run of each task over the budget, with its progress measured as the fraction of the distance from the trivial predictor to the reference that its score covers (Figure 6, Appendix H). These trajectories consist of large gains in the early hours followed by a plateau. On 83% of the tasks the best run has attained at least 90% of its final progress by hour 8, and on 96% by hour 10. Over the final two hours its progress changes by less than 0.02 on 70% of the tasks, and the late improvements that do occur arise from adjustments of ensemble weights, thresholds and post-processing rather than from changes of the method. On the tasks that are not reached, the budget therefore ends after progress has levelled off rather than while it is still being made.

Changes in the revision. The new Appendix H defines the progress measure and presents Table 9 and Figure 6, and Sections 5.3 and 6 each gain one sentence on the budget.

AuthorsOpenReview ↗

Response to W2: the margins over the reference on four tasks

W2. Margins over the reference on Lee et al. 2019, Brain Treebank, Grasp-and-Lift and THINGS-EEG2.

We thank the reviewer for raising these four tasks, and we agree that a wide margin over a published number calls for an explanation. The benchmark records what such an explanation needs: the setting of every reference (Appendix C) and an audit record of every run (Appendix E). From these records, no margin comes from a local split easier than the reference setting, no reaching run fetched labels, solutions or weights, and each margin has a different source. On Lee et al. 2019 the highest score, 0.974, is an invalid run, and the margin of the best valid run is 0.06 (Table 7). On Brain Treebank the local window set is stricter than the published one. On Grasp-and-Lift the agents exceed the winning team's own score on the same series by at most 0.015. On THINGS-EEG2 the margin comes from joint decoding on the structure of the evaluation set.

Lee et al. 2019. The reference, 0.842, is a subject-independent deep ConvNet trained on this dataset (Zhang et al. 2021). It was evaluated leave-one-subject-out over 54 participants on their second-day trials, with a per-participant standard deviation of about 10 points. Our evaluation holds out 14 participants and scores both days. The best valid score is 0.902. Restricted to the reference's own condition, the second-day trials of the held-out participants, the four valid runs score 0.88 to 0.91 with EEGNet and ConvNet ensembles under Euclidean alignment.

Brain Treebank. The reference, 0.662, is the cross-participant result for volume-change detection that the Neuroprobe benchmark (Zahorodnii et al. 2025) reports on this corpus. It is measured on 17,429 windows in which the windows of consecutive words overlap. During construction we found that a probe which only checks whether windows overlap, and uses no neural signal, scores well above chance on that set. Our evaluation therefore keeps the 4,940 windows whose onsets are at least 2 s apart, so the local window set is stricter than the reference setting. The four valid runs score 0.69 to 0.79 with spectral features and logistic-regression ensembles.

Grasp-and-Lift. The reference, 0.981, is the 2015 competition winner scored on the private series 9 and 10. Our evaluation uses series 7 and 8 of the same participants. The winning team's published table also scores its final ensemble on these two series, at 0.978, which gives a like-for-like comparison. The agents' ensembles of temporal U-Nets and dilated convolutions score 0.983 to 0.993, above the winner on the same series by 0.005 to 0.015.

THINGS-EEG2. The reference, 0.682, is the 2026 result on the official split with evaluation responses averaged over 80 repetitions, and our split and averaging are the same. The margin comes from joint decoding on the one-to-one correspondence between evaluation responses and candidate images, which the task brief states and the audit records under the tag struct. prior. Ranked response by response, the submitted decoders score 0.561. Our response to Q1 gives the details.

Screening before admission already serves as the validation run. Before admitting a task we measured a trivial predictor and a cheap solution built from standard features and a simple model. Candidates in which the cheap solution came within noise of the published reference were excluded. On each of these four tasks the cheap solution lies well below the reference (Table 9 of the revised paper). We did not go further and remove tasks on which agents later did well. The margin and the behaviour that produced it are part of what the benchmark records.

AuthorsOpenReview ↗

Response to Q1: repetition averaging and the Opus score on THINGS-EEG2

Q1. Repetition averaging and the Opus score on THINGS-EEG2.

We thank the reviewer for this question. The evaluation does average over image repetitions, as the published reference does, on the providers' official partition of images into training and test concepts. The Opus score is high for a different reason, which the audit recorded. Published methods rank each test response on its own. The run instead decodes all 200 responses of a participant jointly, using the one-to-one correspondence between responses and candidate images that the task brief states. This joint decoding uses no label and no information outside the task, so the audit counts the run as valid.

Averaging and split follow the reference. Each evaluation unit is the mean of a participant's 80 presentations of one test image. The development side ships the four single presentations of every training image, unaveraged. The work that defined the 200-way zero-shot protocol, NICE (Song et al. 2024), averages the test repetitions, and the reference, HyFI (Jo et al. 2026), follows it. The split is the official one: 16,540 training images from 1,654 object concepts and 200 test images from 200 further concepts, with no concept shared, which is what makes the protocol zero-shot. The reference of 0.682 was obtained under the same averaging and the same split.

The margin comes from joint decoding, not from a stronger decoder. The reference, like every published method on this task, trains a decoder and ranks each test response against the 200 candidate images independently. The evaluation set has a structure that this leaves unused. Within each participant the 200 responses correspond one-to-one to the 200 images, and all ten participants viewed the same 200 images. Both facts are stated in the task brief and are properties of the official test design. The agent trained per-participant contrastive decoders on the development set and then treated decoding as an assignment problem. Each participant's responses were matched one-to-one to the images, and the matchings were made consistent across participants through a shared codebook (an EM loop with Hungarian assignment). To separate the decoders from the decoding rule, we re-scored the submitted blend of decoders from its saved per-model predictions under each rule, with the task's primary metric, top-1 accuracy over the 2,000 evaluation responses.

Decoding rule applied to the submitted decodersTop-1 accuracy
Each response ranked on its own, as in the published protocol0.561
Responses of each participant matched one-to-one to the 200 images0.864
Matching made consistent across the ten participants (the submitted run)0.973

Ranked response by response, the decoders score 0.561 against a reference of 0.682. The margin over the reference is what joint decoding adds on this test design, not a stronger decoder.

The run is judged valid by the audit. The audit rule of Section 4.4, set before the audit and applied to every configuration, separates information the task provides from information it does not. A run is invalid when it uses the latter: labels recovered from inputs or public sources, weights trained on the dataset, a public solution, or output constants fitted against the evaluator score. The one-to-one structure is information the task provides. It is stated in the brief and is a property of the official test design. Using it constrains the 200 predictions of a participant jointly but reveals nothing about which image any individual response belongs to.

The audit records such use under the tag struct. prior (Appendix E, Table 5). The tag marks a score that relies on a prior over the structure of the answer set, such as fixed class quotas or a one-to-one correspondence, rather than on information about individual labels. The three other configurations that reach the reference on this task use the same within-participant assignment. The Qwen run went further and probed the evaluator to fix its assignment, which recovers label information from the score. That run was judged invalid and is marked as such in Table 7.

AuthorsOpenReview ↗

Response to Q2: trivial-predictor and cheap-solution scores, now in Table 9

Q2. Trivial and cheap-solution scores from the screening step.

We thank the reviewer for this suggestion, which we have adopted in the revision. Both scores now appear for all 40 tasks in Table 9 of the new Appendix H, beside the published reference and the best valid score of every configuration. Table 3 has no room for further columns at its current size, and Appendix H is where each task's scores are read against its reference, so the two columns are placed there.

What Table 9 lists. The trivial predictor is a prediction that ignores the input (constant, majority-class or empty), scored on the evaluation set before any agent run. Its score s0s_0 is also the floor of the progress measure reported in the same table, P=(s−s0)/(R−s0)P = (s - s_0)/(R - s_0), where RR is the published reference score for the task. Under this measure P=0P = 0 is the trivial predictor and P=1P = 1 the reference, independent of the scale and direction of the metric. The cheap solution is a standard feature representation for the modality fitted with a simple model, with hyper-parameters chosen on the development side only. Both were scored once by the task's own evaluator before any agent run. Neither score was visible to the agents.

Changes in the revision. The new Appendix H adds Table 9, which lists s0s_0, the cheap-solution score, RR and PP of the best valid submission of every configuration for every task.

Reviewer_4w39OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoNeuroBench assembles 40 executable neuroscience machine learning tasks from research datasets and community challenges. Five model agent configurations receive 12-hour budgets, quantitative evaluator feedback, and opportunities to revise their solutions. The benchmark retains submission histories and applies a post-hoc validity audit. The main results report that the strongest configuration reaches the published reference on 10 of 40 tasks, while no configuration reaches it on 23 tasks. An important finding is that some agents reconstruct evaluation labels through repeated exact-score queries, motivating analysis of both final performance and the process used to obtain it.

Critical strengths

The task collection spans multiple recording modalities, output types, and experimental generalization units. Preserving task-specific input/output contracts and retaining complete attempts provides substantially more information about scientific algorithm development than a binary code-execution outcome. The documented environment and reconstruction procedures further increase the potential value of the resource. The paper identifies eight invalid runs involving evaluator-driven label reconstruction, including examples where apparently dramatic progress follows a genuine model plateau.

Critical weaknesses

Agents receive unlimited exact feedback, evaluation time is excluded from the active-time budget, and the best submitted score determines performance. The audit explicitly permits using this feedback to choose hyperparameters, ensemble weights, and uniform thresholds. These choices may satisfy the benchmark’s rules, but the resulting evaluation set is also an adaptive model-selection resource. Excluding direct label reconstruction does not establish that the remaining gains transfer to unseen experimental units. This affects the interpretation of reference attainment and late-run progress as improved algorithmic performance.A feedback set and a separate untouched final test set would preserve the iterative process while distinguishing improvement against feedback from generalization. The current protocol does not provide that distinction.

Questions for authors

Do the best valid programs retain their gains on experimental units whose scores were never exposed during the run? A separate final holdout would materially affect my assessment of both reference attainment and long-horizon improvement.

What are the results when restricted to tasks with genuinely aligned reference and local evaluations?

4 comments
AuthorsOpenReview ↗

Response to W1 and Q1 (1/3): how the score feedback is used

W1 and Q1. The evaluation set as a model-selection resource, and transfer to units whose scores were never exposed.

We thank the reviewer for this comment, which concerns a limitation we weighed when designing the protocol, and we are glad to discuss it. The protocol returns the score of every submission because the benchmark is designed to measure how an agent optimises a method when it receives such feedback, and the audit of Section 4.4 records how each run used that score.

Inspired by this review, we conducted two additional analyses. First, we deepened the audit of the recorded trajectories of all 200 runs from a verdict per run to a step-by-step reading, in order to characterise how the agents performed the selection. Second, we performed a complete second set of runs, 40 tasks × 5 configurations, under a protocol, referred to below as the no-score protocol, in which the evaluation service returns no score during the run and the final submission is scored once, on the untouched evaluation set.

What the score-feedback protocol measures. The prompt leaves the use of the score to the agent, and the constraint on that use is placed in the audit, which, as defined in Section 4.4 and Appendix E, assigns every run a behaviour category according to how its best score was obtained. In category A, method iteration, the best score results from changes of features, model family, training or preprocessing. In category B, scripted hill-climbing, it results from a scripted loop over ensemble weights, hyper-parameters, thresholds or post-processing that is scored on the evaluator. Runs of category B are valid and are counted in Table 2. Category C denotes a shortcut or degenerate output, and adjusting output values directly against the score renders a run invalid.

Over all 200 runs, the audit assigns 157 runs to category A, 35 to category B and 8 to category C (Table 10 of the revised paper). It judges 24 runs invalid: 8 whose best score had met the reference and 16 whose best score had not. Of the 38 runs that reach a published reference, 28 belong to category A and 10 to category B. We use configuration to denote a backbone model together with its coding harness. When only runs of category A are counted, Claude Opus 5, GLM-5.3, Qwen3.8-Max, DeepSeek-V4-Pro and GPT-5.6-sol reach 8, 8, 9, 3 and 0 tasks respectively, compared with 10, 9, 9, 9 and 1 in Table 2, and 15 of the 17 reached tasks remain reached. A second check, which does not rely on the judge, leads to the same conclusion. We removed every submission belonging to a scripted loop, defined as ten or more consecutive submissions less than five seconds apart, and recomputed the best valid score of each run. All 38 runs and all 17 tasks still reach the reference, and the gain contributed by such loops over the best score outside them has a median of 0% of the reference and a maximum of 0.59%.

How the agents use the score over the budget. To characterise the use of the score in time, the LLM judge of Section 4.4 read every trajectory a second time and segmented it into steps, each assigned to one of fourteen types defined in the revised Appendix I. Three types keep the method fixed and adjust its parameters according to the score that the evaluator returned, namely hyper-parameters chosen by comparing submission scores, ensemble weights or thresholds swept against the score, and output values adjusted against the score. We refer to these three types as score-driven tuning. The other types modify the method on the agent's own development data or concern the baseline, the agent's own validation scheme, and fixes or re-runs. Score-driven tuning constitutes 20% of all recorded steps, and their proportion increases from 9% of the steps taken in the first three hours to 35% of those taken in the last three (Figure 7 of the revised paper).

(continued below)

AuthorsOpenReview ↗

Response to W1 and Q1 (2/3): a no-score control protocol

(continued)

A protocol in which no score is exposed. The analyses above establish how the score was used but not whether the gains transfer to units whose scores were never exposed. We therefore conducted a complete second set of 200 runs under the no-score protocol, which differs from the paper's in a single respect: the evaluation service never returns a score. The agent may validate its output as often as it wishes, which checks the format and returns nothing else, and may submit it at any time, which registers the current output as the final answer. The most recent submission is scored once after the budget ends, and a run that never submits is scored on its last validated output. For the purpose of analysis, every output that a run validated or submitted (its candidates) was scored after the run had ended, and the agent saw none of these scores. Tasks, data, splits, references, budget, hardware and all other rules are unchanged. Because no holdout is split on the agent's behalf, the protocol additionally records how agents construct one, and the comparison with the published reference is unchanged. Every run was audited with the judge and policy of Section 4.4.

Table 11 of the revised paper. Results under the no-score protocol per configuration (200 runs, audited). Reached: tasks on which the submitted answer meets the reference. Calls: median number of validations and submissions per run. Candidates: every output that a run validated or submitted, as defined above. Best candidate: the candidate of a run with the highest evaluation score. Within 0.02 P of the best candidate: proportion of runs whose submitted answer scores within 0.02 P of the best candidate. Best candidate submitted: proportion of runs whose submitted answer is the best candidate. P denotes floor-anchored progress, P=(s−s0)/(R−s0)P=(s-s_0)/(R-s_0), where ss is the score on the task's own metric, s0s_0 the score of the trivial predictor and RR the reference, so that P=0P=0 corresponds to the trivial predictor and P=1P=1 to the reference.

ConfigurationReachedCallsWithin 0.02 P of the best candidateBest candidate submitted
Claude Opus 5141690%40%
DeepSeek-V4-Pro21582%60%
GLM-5.351282%48%
Qwen3.8-Max31678%48%
GPT-5.6-sol97472%15%
Any configuration / all runs151781%42%

Where the per-configuration counts move, the trajectories explain the movement. The counts move in both directions. The GPT-5.6-sol configuration gains most. Under score feedback it submitted approximately one thousand times per run, and its runs that met a reference were mostly invalidated for reconstructing labels from the score. Without a score it still calls the service about five times more often than the other configurations, but every call now carries a new candidate, and it reaches nine tasks. Claude Opus 5, the configuration that submitted least under score feedback, reaches four additional tasks. On each of them its no-score run ends with a neural ensemble where its score-feedback run had ended with a linear model, and three of the four lie at the reference line. DeepSeek-V4-Pro loses most. Six of its nine reaches under score feedback belong to category B, having been obtained by scripted hill-climbing on the score, and without a score it reaches two tasks. Across the three configurations that lose tasks, 12 of the 21 configuration–task pairs that are no longer reached still end within 10% of the reference.

(continued below)

AuthorsOpenReview ↗

Response to W1 and Q1 (3/3): transfer to unexposed units

(continued)

Reference attainment is unchanged at the task level. Of the 17 tasks reached under score feedback, 13 are reached without it. Three of the four tasks that are no longer reached end within 5% of the reference, and the two tasks that are newly reached lie at the reference. Of the 23 tasks that no configuration reached under score feedback, 21 remain unreached (Figure 9a of the revised paper). For 31 of the 40 tasks, the best value of P across configurations differs by less than 0.1 between the two protocols. Table 12 of the revised paper lists the per-task scores under both protocols. At the level of the benchmark's conclusions, therefore, the gains are retained when no score of any evaluation unit is exposed: the same agents, on the same tasks and against the same references, reach the same set of tasks within the differences stated above.

The course of a run without a score. Long-horizon improvement is likewise retained. Under the no-score protocol the answer that is finally scored was first produced at a median of 11.4 hours into the 12-hour budget, and 98% of runs improved from their first to their last candidate by the agents' own validation, by 0.42 P at the median. The step reader of Appendix I, applied with the same fourteen types, shows what fills the budget when no score is returned (Figure 10 of the revised paper). The first quarter is alike under both protocols, with a baseline followed by changes of representation and model family. From the second quarter onward, the score-driven tuning steps observed under score feedback are replaced by training choices and by ensembling and post-processing on the agent's own data. Together, these account for 54% to 58% of the steps in the middle quarters, compared with 36% to 38% under score feedback. The last quarter is devoted to enlarging the ensemble, re-running the final pipeline and writing the submission note.

Selection on the agent's own validation is effective. The rule that the run's score is its best valid submission does not act under the no-score protocol, because the agent's own choice of a final answer is what is scored. This choice costs little. In 81% of runs the submitted answer scored within 0.02 P of the best candidate, the candidate of the run with the highest evaluation score, and in 42% of runs it was that candidate (Table 11). Every audited run constructed a validation of its own, in the form of a participant-, session- or site-wise holdout, grouped or k-fold cross-validation, or leave-one-subject-out. The validation scores that the agents reported with their final answers correlate with the evaluation score at a Spearman ρ of 0.72 (Figure 9b of the revised paper).

What the two protocols establish together. The score-feedback protocol measures what an agent reaches when it may select against the evaluator, and it records every such selection for the audit. The no-score protocol measures what the same agents reach, and how they proceed, when no score is exposed and the selection is entirely their own. The two protocols agree on which tasks are within reach and differ in how the agents reach them, and the revision states this finding. We retain score feedback as the benchmark's protocol and report the no-score protocol alongside it as a control.

Changes in the revision. Two appendix sections are added: Appendix I, "Use of the evaluator feedback" (Table 10, Figures 7 and 8), and Appendix J, "No-score control protocol" (Tables 11 and 12, Figures 9 and 10). Section 4.2 refers to Appendix J, and the sentence in Section 6 on withholding scores is revised accordingly.

We hope that this addresses the concern and would welcome further questions.

AuthorsOpenReview ↗

Response to Q2: results on the nine tasks with aligned reference and local evaluation

Q2. Results restricted to tasks whose reference and local evaluation are aligned.

We thank the reviewer for this question and give the requested results below. Appendix C records, for every task, the relation between the evaluation behind the published reference and the local evaluator (Table 4). The closest relation, same evaluation set, denotes that the local evaluator scores the same evaluation units with the same labels as the published number. Nine of the 40 tasks carry this relation: ADHD-200, NLB'21 MC-Maze, WMH 2017, HDR NeuroForecast, SparrKULee, BEETL 2021, MMBCI 2026, BCI Comp. 2020 and THINGS-EEG2. On these nine tasks, at least one configuration reaches the published reference on five, and three or more configurations reach it on one. The table below restricts the counts of Table 2 to the nine tasks.

Claude Opus 5DeepSeek-V4-ProGLM-5.3Qwen3.8-MaxGPT-5.6-solReached by ≥ 1Reached by ≥ 3
Nine aligned tasks2430151

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded a revised version of the paper. The revision uploaded on 19 November 2026 after 11:30 UTC is the final one and supersedes the earlier uploads. All changes relative to the submitted version are marked in blue, and the main text remains within the page limit. Each of our responses ends with a paragraph, "Changes in the revision", that lists the changes made in response to that comment, by position in the paper.

AuthorsOpenReview ↗

General response and thanks to all reviewers

We thank all reviewers for the time and care they have devoted to our paper. We appreciate the positive feedback on our benchmark, as well as the insightful questions and suggestions. They have improved the paper and shaped the additional analyses reported in the individual responses.

We apologise for posting our responses late in the discussion period. Several of the questions called for new experiments and trajectory-level analyses, and completing them took longer than we had anticipated. We recognise that this leaves the reviewers little time to react before the discussion closes, and we regret the inconvenience. We hope that the responses and the revised manuscript nevertheless address the concerns raised.

We thank the reviewers once more for their engagement with our work, and we wish them every success in their research and all the best in life.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.