- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for evaluating coding agents on neuroscience data reuse. It includes eight datasets from published studies of large-scale neural population recordings in behaving mice, spanning different recording modalities, experimental paradigms, and storage formats. Agents receive the data, published code, paper, and structured instructions, and must convert the datasets into a common format for downstream neural decoding. The evaluation combines automated checks of dataset properties and decoder performance with a process-based rubric. Two human evaluators independently rate 1,236 decisions each across the initial agent runs. The paper identifies scientifically consequential errors that executable code and aggregate decoding metrics can miss. It also evaluates reference-assisted and reference-free agent judges, and uses the former to compare newer models, prompts, and harnesses. The study provides useful evidence about current agent limitations, but important questions remain about the interpretation of benchmark failures and the reliability of automated comparisons.
Critical strengths
- The benchmark targets a meaningful scientific bottleneck and evaluates semantic correctness beyond executability. Understanding heterogeneous datasets requires interpreting variable meanings, filtering rules, and temporal alignment. The examples show why successfully running conversion code or achieving good decoder accuracy does not establish faithful data reuse. This is a useful contribution to evaluating scientific agents.
- The initial evaluation involves substantial manual inspection and transparent reporting. Two evaluators independently assess the decisions with agent identities hidden, and each dataset–agent pair is run three times. The rubric permits justified alternatives and solutions that improve on the reference. Detailed failure cases, agreement measurements, and the reported release of reference solutions and trajectories make the study more informative than an evaluation based only on aggregate success rates.
Critical weaknesses
- The benchmark does not sufficiently separate scientifically incorrect conversions from reference disagreement and potentially concerning choices. Section 4.1 acknowledges that outcome checks can reject valid alternatives, including unspecified units. Table S1 also reports failures arising from different missing-data handling decisions rather than technical violations of the specification. The process rubric partly addresses this, but Appendix H defines “concerning” as not strictly wrong, while Section 4.4 groups it with “incorrect” for error detection. Consequently, the reported failure measures combine demonstrated mistakes with choices requiring further inspection. This matters especially for end-to-end assessment, where one disputed decision can determine whether an otherwise acceptable conversion passes. The paper needs a clearer, adjudicated distinction between scientific invalidity, justified alternatives, and unresolved concerns.
- The automated evaluator is insufficiently validated for the subsequent model and prompt comparisons. Reference assistance improves judging, but Table 2 reports supervised-judge precision of only 0.342 and 0.292, and the combined judge reaches 0.359. Figure S3 shows a per-trial score correlation of only 0.47 with the first human evaluator. These results establish partial usefulness, but do not establish that automated scores reliably preserve comparisons between configurations. Figure 4 relies on these judges, without a corresponding human validation of the newer configurations. Moreover, the outcome-based evidence does not consistently corroborate the process-score trends: under the revised prompt, GPT-5.6 increases the process score from 0.85 to 0.88 relative to GPT-5.4, while its outcome score decreases from 0.76 to 0.68 in Figure S7. The latter difference is not statistically significant, but its opposite direction warrants a more qualified interpretation.
Questions for authors
- Evaluation clarification: How do end-to-end success rates change when only adjudicated “incorrect” decisions count as scientific failures, while “concerning” decisions are reported separately? How were disagreements about scientifically acceptable alternatives resolved?
- Evaluator validation: Do you have a blinded human audit of the newer-model and revised-prompt outputs that tests whether the automated judges preserve configuration rankings and estimated improvements? Are judging errors systematic across datasets or agent families?
- Result interpretation: How do you explain the opposite process-score and outcome-score trends for GPT-5.4 versus GPT-5.6 under the revised prompt? Which evidence supports interpreting the higher process score as improved scientific conversion quality?
3 comments
Evaluation validation
Weakness 2 and Question 2
We do agree with the reviewer that human validation is needed to confirm the Figure 4 trends. We are therefore running a blinded human evaluation of the Claude Code configuration with Opus 5 and the revised prompt. Because each configuration involves rating several hundred individual decisions against the manual solutions, this evaluation requires additional time and will not be complete within the discussion period. We will report the results in the revision.
With that said, our goal in Section 4.5 was to build a reasonable automated pipeline that would make it practical to evaluate new models, prompts, and harnesses without a full round of human rating each time. We see the limits of this pipeline as part of our contribution. Our results show that agent-as-judge evaluation must be used with caution in this setting. The unsupervised judges are close to chance, and even the supervised judges have low precision (Table 2). Our human ratings of 1,236 decisions from two independent evaluators can therefore serve as a benchmark asset for developing and validating more reliable agent judges.
At the same time, we believe the supervised judge is a reasonable basis for an overall score, even if individual ratings are noisy. At the level of per-trial scores, the combined supervised judge correlates with the first human evaluator at r = 0.47, compared with r = 0.63 between the two human evaluators (Figure S3). The comparisons in Figure 4 are further averaged over 24 trials per configuration, which reduces the impact of noise in individual judgments. We used this pipeline for these reasons. We agree, however, that this does not rule out systematic judge biases that could affect particular comparisons. In the revision, we will state explicitly that the Section 4.5 results are estimates from the supervised judges and qualify our conclusions accordingly. We will also highlight improving this pipeline as an important direction for future work. That includes making agent judges more reliable, so that a full human evaluation is not needed each time a new model is released.
Regarding systematic judging errors, for agent families, Table S5 in the paper breaks down judge performance by which agent’s trials are being scored, including cross agent family judging. We find no evidence that either judge favors its own model family. Each judge’s false-positive rate is nearly identical on Claude Code and Codex trials (Claude judge 0.20 vs. 0.20; Codex judge 0.32 vs. 0.30), and the difference in precision in Table S5 is largely explained by the higher error rate in Claude Code trials (15.2% vs. 10.5% of decisions).
Across datasets, we also did not find statistically significant differences. The judges’ precision does not differ reliably across datasets. We did confirm that both judges are consistently harsher than the human evaluators on subtasks where several approaches can be defensible. For trial filtering, the proportion of decisions rated at least ok was 0.85 for Evaluator #1, but only 0.50 and 0.29 for the Claude and Codex judges, which rated it lower in 6 and 7 of the 8 datasets, respectively. For variable processing, the proportions were 0.78 for Evaluator #1, compared with 0.57 and 0.47 for the judges, which were lower in 7 and 8 of the 8 datasets. Lastly, we found that all raters (human and agent) agree that Hasnain2024 is the most difficult dataset.
More detailed analysis of the evaluation metric
Question 3
We thank the reviewer for this careful observation. We looked into the discrepancy between the outcome-based and process-based score in this case. On closer inspection, we found that this does not reflect a general disagreement between the two evaluations. The drop in outcome score is driven by two datasets, Hasnain2024 and Zhang2025, and in each case by a single choice about which data to include (i.e., data filtering choice). Excluding these two datasets, both scores increase (outcome 0.711 → 0.730, process 0.888 → 0.901)
Hasnain2024 (outcome 0.79 → 0.33, process 0.75 → 0.74): All three GPT-5.6 trials kept only 12 of the 44 reference sessions, namely the two-context sessions. The agent’s notes stated that it did this deliberately, since the context variable was only meaningful in those sessions. This single decision caused nearly every outcome metric to fail. The numbers of sessions, trials, subjects, and neurons all fell to 23–50% of the reference, and with less data, decoder accuracy also dropped below the threshold for several variables. The process-based judges did catch this error, rating both the data-loading and session-split subtasks as incorrect in all three trials. Because these were only 2 of 29 rubric questions, this did not have a big effect on the process score.
Zhang2025 (outcome 1.00 → 0.74, process 0.74 → 0.91): Two of the three GPT-5.6 trials kept every Kilosort cluster without quality filtering, resulting in 8.3 times the reference number of neurons. This caused the neuron-count statistic to fail and lowered the accuracy of two decoders by about 10%.
More broadly, this example highlights that the two evaluations measure different aspects of a solution. Outcome-based metrics are highly sensitive to upstream choices about which data to include. A single decision at the data loading stage propagates to nearly every dataset statistic and decoder metric, so it can dominate the outcome score. The process-based rubric instead localizes such an error to the one or two subtasks where it occurs, while also crediting improvements in the many downstream decisions for the newer agents that outcome metrics do not directly measure. Neither score alone fully captures solution quality, which is why we report both. We will add this analysis to the revision.
Rating scale
We thank the reviewer for their careful reading of our paper and supplementary materials. Their questions pointed to specific places where our analysis could be clearer, and to a few additional analyses. Addressing them has strengthened the paper.
Weakness 1 and Question 1
We would like to clarify our rating scale. The rubric already separates the three cases the reviewer describes. A justified alternative that differs from the reference is rated ok, or better if it improves on it. Both count as correct throughout the paper, including in the end-to-end assessment.
We group concerning with incorrect deliberately. Together they define decisions that fall below the scientific standard expected of a data-reuse pipeline. A concerning decision is not technically wrong, in the sense that the code runs and the output satisfies the specification, but it is a choice that would not be accepted in actual data conversion code.
For example, two Codex trials on Zhong2025 filtered neurons by stimulus or reward selectivity (d′) before exporting the dataset (Appendix B, Example 1). Nothing in the task forbids this, and each filter has a plausible rationale. However, a dataset prepared for general decoding analyses that has already been filtered for decodable neurons would bias every downstream analysis toward high decoding performance. No experienced researcher would accept it as a faithful conversion. Similarly, one Codex trial on Chen2024 used raw DeepLabCut tongue positions without handling occluded frames (Example 4). The output is well-formed, but it mixes tracking noise into the decoded variable.
We agree that Appendix H’s wording (“not strictly wrong”) understates this, and we will revise the definition to make the standard explicit. That said, we agree it is informative to report the two categories separately. The end-to-end row of Figure 2 already distinguishes trials containing at least one incorrect decision (x) from trials whose only issues are concerning (!). Counting only incorrect decisions as failures, 23 of 48 trials pass end-to-end, compared with 6 when concerning decisions are included. We will report both numbers in the revision.
Lastly, we would like to note that each evaluator independently judged whether the agent’s solution was a justified alternative (ok) or fell below the standard (concerning or incorrect). The ratings were not adjudicated after the fact. Instead, we report agreement between the two evaluators (Figure S1), who agree on 90% of decisions at the binary level, indicating that the boundary between acceptable and below-standard decisions is reproducible across two independent sets of human evaluators.