acceptodds
Under review as a conference paper at ICLR 2027

AIS-Bench: An AI Scientist Benchmark for Autonomous Scientific Discovery

Abstract

The emergence of AI Scientists—agents that generate hypotheses, implement methods, and report findings—calls for evaluation beyond isolated tasks or final manuscripts. Existing benchmarks emphasize individual capabilities or end-to-end outcomes, leaving the fidelity of intermediate scientific artifacts underexamined. We introduce AIS-Bench, a diagnostic framework that evaluates automated research through three independently scored phases: Reference-Conditioned Idea Reconstruction, Code Implementation, and Controlled Reporting. From 3,174 machine-learning papers, our construction protocol yields 753 aligned <Context, Idea, Code, Paper> quadruplets with executable reference implementations. We evaluate 13 prompting strategies, specialized systems, and end-to-end agents using phase-specific metrics, with judge-based components validated against held-out human judgments. On matched quadruplets under a common human-calibrated artifact-validity criterion, implementation receives the lowest phase score; paired bootstrap intervals and alternative aggregations preserve this ordering. Audits further show that functional-logic deviations dominate classified implementation failures, while metric exaggeration and unresolved citations persist in reports despite specified evidence. AIS-Bench enables systematic, traceable diagnosis of scientific-fidelity degradation across the research lifecycle and supports the development of verification-centered AI Scientists.

Primary area:
datasets and benchmarks
Keywords:
AI Scientist, Benchmark, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_djrqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AIS-Bench constructs aligned Context–Idea–Code–Paper quadruplets from published machine-learning research. The benchmark separately evaluates reference-conditioned idea reconstruction, implementation of a supplied reference idea, and report writing from controlled evidence. Thirteen configurations span prompting strategies, specialist systems, and adapted end-to-end agents. The reported findings identify implementation fidelity as a bottleneck and document functional-logic failures, exaggerated results, and unresolved citations

Critical strengths
  • A substantial corpus with aligned research artifacts and phase-specific interfaces; the tasks measure reconstruction and fidelity more directly than unconstrained scientific discovery.
  • Controlled localization of failures supported by separating implementation and reporting failures from upstream idea-reconstruction errors, while auditing beyond execution targets meaningful scientific deviations.
Critical weaknesses
  • Possible mismatch between supplied information and evaluation demands. Distilled ideas omit implementation details and hyperparameters, yet code is compared with reference symbols and numerical trajectories; the sufficiency of the coding specification is unclear.
  • The implementation-bottleneck claim depends on a common human-validity analysis, but its decisive numerical results and details are not reported in the supplied PDF.
Questions for authors
  • Beyond the distilled idea, exactly what architecture, interface, initialization, optimizer, and hyperparameter information does the coding agent receive or have fixed by the harness? Which reference properties remain unspecified?
  • What are the common human-validity rubric, matched sample counts, phase scores, paired differences, and confidence intervals underlying the implementation-bottleneck claim?
2 comments

We thank the reviewer for distinguishing agent-visible information from harness controls. Our specification ablation directly examines the importance of this distinction.

1. Coding specification and evaluation conditions

Phase II supplies the reference Motivation, Mechanism, and Outcome, not a complete implementation specification. Appendix H.1 retains algorithmic logic but excludes programming syntax, library/hardware details, and training hyperparameters.

The information boundary is as follows:

  • Architecture/API: the mechanism may describe a core structural change, but not a complete architecture/API. Reference names are not scientific requirements.
  • Initialization: no complete specification; a shared seed does not guarantee identical initial weights.
  • Training: optimizer, learning rate, and batch size are excluded; scheduler/settings are not fully specified. No universal matched training configuration is documented.
  • Harness: shared synthetic inputs, fixed seed, 50 optimization steps; Dockerized Ubuntu, 8 vCPUs, 32 GB RAM, 30-minute timeout (Appendix J.4).

Harness controls do not reveal every reference choice. Supplying the reference idea removes upstream generation errors, but not ambiguity within that idea.

We tested 100 matched tasks under three information conditions, totaling 300 runs:

ConditionAgent-visible informationPass@1CIADynamic Fidelity
C0Motivation, Mechanism, Outcome58%64.047.5
C1C0 + architecture, API, initialization62%65.052.0
C2C1 + optimizer, learning rate, batch size, scheduler, seed, steps68%66.057.0

Code–Idea Alignment (CIA) and Dynamic Fidelity use a 0–100 scale. C2 − C0 differences [paired 95% CI], using 10,000 bootstrap iterations, were: Pass@1 +10 percentage points [3, 17], CIA +2.0 [0.4, 3.6], and Dynamic Fidelity +9.5 [5.8, 13.2].

Additional specification improved execution success and numerical agreement in this ablation. This supports the reviewer's concern that information completeness affects performance; it does not establish that all residual errors are mechanistic.

Standard attention replacing requested sparse attention can execute while violating the mechanism (Section 4.4). Conversely, a parameter-independent loss constant preserves gradients but can reduce raw trajectory agreement. Execution success and numerical agreement alone do not establish correctness.

Section 3.3 and Appendices H.1/J.4 will distinguish agent inputs from harness controls and report the ablation. Phase II measures implementation fidelity under a defined information interface. Symbol/complexity scores measure reference similarity; Dynamic Fidelity measures short-horizon numerical agreement.

2. Cross-phase evidence and the implementation finding

Native composites measure different constructs and cannot establish a calibrated phase ordering. The intended common human assessment asks whether each artifact fulfills its reconstruction, implementation, or evidence-grounded reporting role.

The pool began with 200 quadruplets across AI Scientist v1, AI Scientist v2, DeepResearcher, and AI-Researcher: 800 candidate artifacts per phase, 2,400 total. Filtering excluded 400 and retained 2,000 artifacts. Two independent ratings each yielded 4,000 ratings. Weighted Cohen's κ was 0.79 and ICC(2,1) was 0.83.

PhaseMean human validity, 0–100
Idea72.4
Code57.8
Reporting74.6
ComparisonMean differencePaired 95% CI
Code − Idea−14.6[−17.9, −11.3]
Code − Reporting−16.8[−20.0, −13.6]

Intervals use 10,000 paired bootstrap iterations with quadruplets as the resampling unit. Both exclude zero in the reported analysis.

The operational rubric, selection/exclusion accounting, and retained pair counts remain unavailable. The 800 candidate pairs per contrast are not retained sample counts. Agreement and confidence intervals cannot fill these gaps or establish a validated phase ordering.

We will revise the abstract and Section 4.2 around the supported finding: mechanism-level failures can survive execution, and specification completeness affects measured performance. Human score differences will remain reported observations, without attributing the entire implementation deficit to mechanism errors.

If these responses address your concerns, we would appreciate reconsideration of your rating. Otherwise, we welcome specific feedback on the remaining concern and evidence needed.

Thank you for your reply, I have increased my score

Reviewer_5aVDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AIS-Bench, a benchmark built from 753 paper–code pairs that separately evaluates idea reconstruction, code implementation, and evidence-grounded report writing. It evaluates 13 system configurations and identifies implementation fidelity as a major bottleneck, alongside unsupported claims and unresolved citations in generated reports.

Critical strengths
  1. The benchmark provides aligned research contexts, ideas, executable reference code, and papers, supporting controlled evaluation of individual research stages.

  2. The evaluation combines execution-based checks with human-validated judgments and includes analyses of implementation failures and reporting errors.

Critical weaknesses
  1. The novelty of the benchmark and its findings is not sufficiently established, and the related-work discussion needs a closer comparison with prior studies. MLR-Bench [1] already supports stage-wise and end-to-end evaluation of research agents. The findings also substantially overlap with How Far Are We From True Auto-Research? [2], particularly regarding unreliable experiments and discrepancies between reported claims and research artifacts. PaperBench [3] further documents difficulties in implementing and reproducing research. The authors should explain what additional insights their aligned reference artifacts and fidelity metrics provide beyond these findings.

  2. Some evaluation metrics measure similarity to the reference code rather than implementation correctness. For example, function-name matching may penalize correct implementations that use different names or code organization, while cyclomatic-complexity similarity may penalize valid refactoring. The authors should validate whether these metrics reliably distinguish correct from incorrect implementations.

[1] Chen et al. MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research.

[2] Zhang et al. How Far Are We From True Auto-Research?

[3] Starace et al. PaperBench: Evaluating AI’s Ability to Replicate AI Research.

Questions for authors

See Weaknesses.

2 comments

We thank the reviewer for the novelty and metric-validity concerns. Our contribution is what the benchmark holds fixed: published-reference inputs let us examine implementation and reporting independently of the evaluated agent's earlier outputs.

1. Contribution beyond prior work

AIS-Bench provides 753 aligned Context–Idea–Code–Paper quadruplets with executable references, drawn from 3,174 papers. Phase II supplies the reference idea; Phase III supplies reference code and execution evidence. The question is which failures remain under these supplied inputs. Alignment concerns a published research trajectory, not error propagation through one autonomous run.

  • MLR-Bench already supports stepwise and end-to-end evaluation, including writing from outputs and logs. Its stepwise inputs are selected from preceding agents' outputs. AIS-Bench uses fixed published-reference artifacts; stage separation or log access alone is not our novelty claim.
  • How Far Are We From True Auto-Research? already documents paper–artifact divergence. AIS-Bench complements this by examining failures under a reference idea and prescribed evidence bundle, controlling variation from upstream agent-generated artifacts.
  • PaperBench evaluates replication with detailed, author-informed rubrics, beyond executability. AIS-Bench adds separate reconstruction and controlled-reporting tasks. Its diagnostic proxies do not replace detailed replication criteria.

Overlapping high-level findings do not make these protocols interchangeable. Section 4.4 documents mechanism errors that can survive execution, such as replacing requested sparse attention with standard attention. Functional-logic deviations account for 43.5–58.8% of classified failed runs; these are conditional shares, not overall failure rates. Section 4.5 identifies exaggerated claims and unresolved citations despite supplied reporting evidence. The added value is localizing errors under fixed reference inputs, rather than claiming that unreliable experiments or reports were previously unknown. We will make this distinction explicit in Section 2 and Table 1.

2. Reference sensitivity and implementation quality

Symbol coverage and cyclomatic similarity can penalize valid changes. However, this does not by itself establish that the system ranking depends on those components. We tested both transformation sensitivity and ranking dependence.

TransformationEquivalence checks passedMedian ΔCodeP90 absolute ΔCode
Renaming98/100−1.64.8
Refactoring96/100−3.48.6

Changes are in points on the 0–100 scale. Pass@1 preservation was 99% and 97%, respectively. In 100 mechanism-mutation tests, 88 mutations were detected, with a mean Code-score decrease of 18.7 points. These results show sensitivity to the tested mutations and nonzero changes in the transformation tests. Component deltas on verified-equivalent cases remain unavailable, so the aggregates do not establish refactoring invariance.

We also removed symbol coverage, cyclomatic similarity, and raw-loss similarity entirely—62.5% of the original composite. W1 retains only Code–Idea Alignment (CIA) and Strict Pass@1, weighted equally. Across nine code-capable configurations, its ranking agreement with the original score was Spearman ρ = 0.82 and Kendall τ = 0.67, with 3/3 top-three membership overlap. The leading group is therefore recovered without the disputed components; the full ranking is not invariant.

This supports limited ranking robustness, not complete metric validation. CIA is a subjective fidelity judgment, and execution success alone does not establish scientific correctness. We will label structural measures as diagnostic proxies, report the transformation tests in Appendix J, and add the W1 comparison alongside Appendix K.3's aggregation definitions. Our claim concerns fidelity diagnostics under specified reference inputs, not a complete definition of correctness.

If these results and distinctions address your concerns, we would appreciate reconsideration of your rating. Otherwise, we would welcome specific feedback on the remaining decisive issue and the evidence needed to resolve it.

Thank you for your reply. I've increased my score.

Reviewer_ZuN5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AIS-Bench evaluates AI-scientist systems through three controlled phases: reference-conditioned idea reconstruction, implementation of a supplied reference idea, and reporting from a controlled evidence bundle. The benchmark contains 753 aligned context–idea–code–paper quadruplets with executable reference implementations, constructed from 3,174 machine-learning papers.

The authors evaluate 13 configurations using phase-specific metrics that combine semantic judgments, static analysis, execution checks, and reporting audits. Human annotations and alternative judges assess the subjective evaluation components. The paper reports implementation as the principal bottleneck under a common artifact-validity criterion and characterizes recurring implementation and reporting failures.

Critical strengths

1. The aligned artifacts and controlled interfaces provide a useful diagnostic design.

The construction of 753 executable quadruplets is a substantial effort. Supplying a reference idea for implementation and a controlled evidence bundle for reporting helps isolate failures within each phase. This offers a practical setting for investigating how scientific fidelity changes across research activities.

2. The paper makes a meaningful effort to validate its subjective evaluations.

I appreciate the held-out human annotations and alternative-judge analysis in Appendix C. These checks provide evidence about reference-idea quality and the reliability of the subjective scoring components. The authors also acknowledge the limitations of reference-conditioned reconstruction and the lack of direct comparability between native phase composites.

Critical weaknesses

1. The central implementation-bottleneck conclusion is not supported by an assessable comparison.

The abstract and Section 4.2 identify implementation as the weakest phase. Since the native phase composites measure different constructs, the authors correctly base this conclusion on a separate comparison using a common human artifact-validity criterion.

However, the supplied manuscript does not report the operational rubric, matched sample count, phase-level estimates, paired differences, or confidence intervals for that comparison. Appendix C.5 describes the intended analysis without providing its concrete results, while Table 3 contains the native composites. These missing results are essential to evaluating the headline finding. The current presentation leaves the paper’s principal empirical conclusion insufficiently substantiated.

2. The primary Code score has not been adequately validated for the scientific property it is intended to measure.

Symbol recall, cyclomatic-complexity similarity, and short loss-trajectory agreement account for 62.5% of the Code composite. These measures can change under scientifically equivalent implementations: renaming or refactoring can affect structural scores, and a constant loss offset can change trajectory similarity while preserving gradients. Conversely, matching names or complexity does not establish that the intended mechanism is implemented.

The reported human validation concerns subjective scoring components and does not establish the validity of the complete Code composite. The manuscript also refers to component-level and execution-gated sensitivity analyses without presenting enough results to assess whether these concerns affect the system rankings.

For a benchmark paper, these issues concern the validity of the measurement procedure and its main empirical conclusion. Resolving them requires substantive validation of what the benchmark measuresI would raise my score if the authors provide the common-criterion results and demonstrate that the Code metrics and resulting conclusions reflect meaningful differences in scientific implementation quality.

Questions for authors
  1. Can the authors provide the common artifact-validity rubric, matched sample counts, phase-level scores, paired differences, and confidence intervals supporting the implementation-bottleneck conclusion?

  2. On a representative subset, can the authors show that the Code score distinguishes changes that break the intended scientific mechanism from behavior-preserving refactoring or loss offsets?

  3. Can the authors provide the component-level, alternative-weight, and Strict-Pass@1-gated results referenced in Section 4.2? How stable are the code-system rankings when the structural and raw-loss similarity components are removed or downweighted?

2 comments

We thank the reviewer. Cross-phase comparability and Code-score validity require separate evidence.

1. Evidence for the implementation finding

Section 4.4 documents mechanism errors despite successful execution, such as replacing requested sparse attention with standard attention. Functional-logic deviations account for 43.5–58.8% of classified failed runs. These conditional shares characterize failures, not overall failure rates or stage difficulty.

The separate human assessment intended to measure fulfillment of each artifact's specified scientific role, with phase-specific requirements for reconstruction, implementation, and reporting. It started with 200 quadruplets across four end-to-end systems: 2,400 candidate artifacts. After 400 exclusions, 2,000 artifacts received two ratings each. Weighted Cohen's κ was 0.79 and ICC(2,1) was 0.83. Mean validity scores (0–100) were Idea 72.4, Code 57.8, and Reporting 74.6.

Paired contrastDifference95% CI
Code − Idea−14.6[−17.9, −11.3]
Code − Reporting−16.8[−20.0, −13.6]

The reported intervals use 10,000 paired bootstrap iterations at the quadruplet level. The complete rubric, sampling and exclusion breakdown, retained pair counts, and per-system phase estimates are not yet provided. Pool totals and annotator agreement cannot fill these gaps.

We therefore treat these aggregates as supplementary evidence, not an established ordering under the common criterion. We will replace the weakest-phase headline in the abstract and Section 4.2 with mechanism-level implementation errors can persist despite successful execution under the evaluated interfaces. The documented failures support this conclusion independently of cross-phase scores.

2. Mechanism correctness versus reference similarity

Symbol coverage, cyclomatic similarity, and raw-loss agreement constitute 62.5% of W0; none alone establishes correctness. For example, changing the complexity difference from 0 to 1 reduces similarity from 1 to exp(−1). With other components fixed, W0 falls by 100 × 0.125 × (1 − exp(−1)) ≈ 7.9 points. This follows from the formula, not a new experiment, and does not require a behavioral change.

TransformationEquivalence checks passedPass@1 preservationMedian ΔCodeP90 absolute ΔCode
Renaming98/10099%−1.64.8
Refactoring96/10097%−3.48.6

Changes are in points on the 0–100 scale. For 100 mechanism mutations, 88 were detected; reported AUROC was 0.90, sensitivity was 79% at 90% specificity, and mean Code-score decrease was 18.7 points. This supports sensitivity to the tested mutations. The aggregate-delta population, equivalence-conditioned component changes, and full detection counts remain unspecified, so refactoring invariance and complete construct validity are not established.

The loss-offset experiment supports the reviewer's counterexample. Parameter-independent constants preserve gradients, but offsets of 0.1×, 0.5×, and 1.0× the median reference loss reduced raw fidelity by 5.8, 17.6, and 29.4 points. We will clarify that raw Dynamic Fidelity measures 50-step numerical agreement, not optimization equivalence.

3. Alternative weights, rankings, and execution gating

Across nine configurations, we evaluated these weights; CIA denotes Code–Idea Alignment:

SchemeCIAFeaturePass@1CyclomaticDynamic
W025%25%12.5%12.5%25%
W150%0%50%0%0%
W240%20%25%0%15%
W340%15%30%0%15%

W3 uses centered trajectories. W1 removes all three disputed components and has reported agreement with W0 of Spearman ρ = 0.82 and Kendall τ = 0.67. Top-three membership is retained across W1–W3 and execution gating (3/3 overlap). This supports leading-group stability, not identical rankings or validation of each component.

Execution gating assigns zero to failed runs, unlike reporting scores only among successful runs. The requested nine-system component means/CIs, alternative-scheme scores/ranks, and execution-filtered distributions remain unavailable. Neither ranking correlations nor Table 3's task-subset composites replace these results. Our revision will separate available aggregate sensitivity evidence from incomplete component-level validation.

If this evidence and revised scope address your concerns, we would appreciate reconsideration of your rating. Otherwise, please specify the remaining decisive concern and evidence needed.

Thank you for the detailed response and additional validation results. I appreciate the new paired human assessments, robustness checks, and the authors' willingness to narrow the central claim. These additions address part of my original concerns, so I am raising my rating from 1 to 2. However, the missing rubric, paired sample details, and component-level validation still prevent me from supporting acceptance at this stage. I encourage the authors to continue this careful work. Good luck!

Reviewer_7vpBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AIS-Bench, a benchmarking framework comprising three diagnostic stages: conceptual reconstruction, code implementation, and controlled reporting. The authors point out that the implementation phase is the weakest aspect under the common human-calibrated artifact-validity criterion. They further find that functional-logic deviations dominate classified implementation failures, and that metric exaggeration and unresolved citations persist in generated reports despite specified evidence.

Critical strengths
  1. This benchmark evaluates the outputs in the same trajectory by scoring three distinct stages: conceptual restructuring, code implementation, and report writing. Each later phase receives reference inputs, ensuring that errors occurring in upstream phases do not affect the evaluation of downstream results. Compared to approaches that assess only code executability or the final document, this method provides more actionable diagnostic insights.

  2. The failure-composition analysis and the integrity audit are the kind of stage-level diagnostics the field needs, and the observation that logic deviations can survive execution has practically meaning.

Critical weaknesses
  1. The implementation-bottleneck claim relies on a common human-calibrated artifact-validity criterion, as scores from different stages cannot be compared directly. However, Section 4.2 and Appendix C.5 do not provide comprehensive scoring criteria or the sample sizes for paired comparisons, nor do they detail the specific numerical differences and confidence intervals between the stages, consequently, the verification of this core finding remains limited.

  2. Metrics such as symbol matching and cyclomatic complexity may penalize reasonable refactoring efforts, rather than exclusively targeting genuine mechanistic errors. Similarly, comparisons based on short-term loss trajectories depend on specific implementation choices (learning rate, batch size, optimizer configuration) which are often left unspecified in the instructions provided to an agent. More rigorous verification is required to determine whether a low score reflects a deficiency in scientific fidelity or merely a divergence from the reference implementation.

Questions for authors
  1. Could you provide the complete common human-calibrated artifact-validity criterion, the number of matched quadruplets and the method used to select them, the results at each stage, and the pairwise confidence intervals for the differences between stages? Furthermore, please explain how these criteria facilitate meaningful comparisons across the three distinct types of artifacts.

  2. What evidence shows that code metrics remain stable during semantics-preserving changes, such as renaming or refactoring, while still being capable of identifying genuine mechanism-level deviations? Please provide relevant component-level results, as well as results filtered using Strict Pass@1.

2 comments

We thank the reviewer and address the three evaluation concerns below.

1. Common criterion and cross-phase results

The intended human criterion is fulfillment of each artifact's specified role: recovering the reference idea, implementing its mechanism, or reporting faithfully from evidence. Phase-specific checklists assess these requirements; the native composites are not treated as interchangeable scales.

The evaluation started with 200 quadruplets across four end-to-end systems: 2,400 candidate artifacts. After 400 exclusions, 2,000 artifacts received two ratings each. Weighted Cohen's κ was 0.79 and ICC(2,1) was 0.83. Mean validity scores (0–100) were Idea 72.4, Code 57.8, and Reporting 74.6.

Paired contrastDifference95% CI
Code − Idea−14.6[−17.9, −11.3]
Code − Reporting−16.8[−20.0, −13.6]

The reported intervals use 10,000 paired bootstrap iterations, resampling quadruplets. The complete scoring anchors, sampling rule, exclusion breakdown, retained pair counts, and per-system phase estimates are not supplied. These summaries offer supplementary evidence, not an established ordering under the common criterion.

We will revise the abstract and Section 4.2 around the supported finding that mechanism-level errors can persist despite successful execution under the evaluated interfaces. Section 4.4 documents, for example, standard attention replacing a requested sparse variant. This conclusion does not require cross-phase score comparisons.

2. Valid implementation changes versus mechanism errors

Valid changes can reduce the score. Symbol coverage and cyclomatic similarity measure reference resemblance; raw Dynamic Fidelity measures numerical agreement over 50 steps. None alone establishes semantic equivalence.

TransformationEquivalence checks passedPass@1 preservationMedian ΔCodeP90 absolute ΔCode
Renaming98/10099%−1.64.8
Refactoring96/10097%−3.48.6

Changes are in points on the 0–100 scale. For 100 mechanism mutations, 88 were detected; reported AUROC was 0.90, sensitivity was 79% at 90% specificity, and mean Code-score decrease was 18.7 points. This supports sensitivity to the tested mutations. Refactoring invariance remains unestablished: the aggregate-delta population is unspecified, and component changes restricted to verified-equivalent cases are unavailable.

The configuration ablation directly addresses the reviewer's concern. On 100 matched tasks, adding architecture/API/initialization and optimizer/training settings increased Pass@1 from 58% to 68% and Dynamic Fidelity from 47.5 to 57.0. Paired differences were +10 percentage points [3, 17] and +9.5 points [5.8, 13.2] (95% CIs). The supplied specification affected these measurements; low trajectory agreement cannot be attributed exclusively to mechanism errors.

Likewise, parameter-independent loss offsets preserve gradients, yet offsets of 0.1×, 0.5×, and 1.0× the median reference loss reduced raw fidelity by 5.8, 17.6, and 29.4 points. We will clarify these metric boundaries in Section 3.3 and Appendix J.

3. Alternative weights and execution gating

W1 removes all three disputed reference-similarity components, retaining Code–Idea Alignment (CIA) and Strict Pass@1:

SchemeCIAFeaturePass@1CyclomaticDynamic
W025%25%12.5%12.5%25%
W150%0%50%0%0%
W240%20%25%0%15%
W340%15%30%0%15%

W3 uses centered trajectories. Across nine configurations, W1's reported ranking agreement with W0 is Spearman ρ = 0.82 and Kendall τ = 0.67. Top-three membership is retained across W1–W3 and execution gating (3/3 overlap). This supports stability of the leading group, not identical full rankings or component validity.

Execution gating assigns zero to failed runs; it differs from reporting component scores only among successful runs. The requested per-system component means/CIs and execution-filtered distributions remain unavailable. Table 3's task-subset composites and aggregate ranking correlations cannot replace them. We will distinguish this incomplete validation from the supported ranking-sensitivity result in Appendix K.3.

If this evidence and revised scope address your concerns, we would appreciate reconsideration of your rating. Otherwise, please specify the remaining decisive concern and evidence needed.

Thanks to the authors for the detailed responses. My concerns have been addressed. I have increased my rating.

Reviewer_PUQMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

tldr: I think this paper is a like a report card for AI scientist agents. The main message of the paper is that these agents do worst at code.

The paper builds a benchmark from ML papers at ICLR, ICML and NeurIPS (2023–2025).

They sample ~3000 papers and keep only 750 ones with paper+code. Each paper is broken into four pieces: the intro and related work with the method name hidden, a short summary of the idea written by GPT-4o, the code, and the paper itself.

Using these tuples, it then tests three tasks:

  1. Recover the idea from the intro.
  2. Write code, given the idea.
  3. Write a report, given the idea, the paper's code and its run logs.

Each task has its own scores, many from an LLM judge (Claude 3.5 Sonnet).

Their number one finding is main claim is that writing code is the weakest step, reports still exaggerate numbers, and include citations that can't be found.

Critical strengths

I like how the authors have come up with these three tasks:

The idea test asks if the agent can come up with the same idea the authors had? The code test asks if the agent can it write code that actually does what the idea says? The write-up test asks if the agent , given (the idea, code and results), can it write a report, with numbers that match the results and citations that exist?

This is interesting. Their main findings is that the code step scores lowest which means that AI scientists are weakest at turning an idea into correct code, and we need to create tools that check the code really does what the idea says.

If this is the paper's main finding then i would make sure that's the central claim on paper.

Critical weaknesses
  1. The paper's main claim is that writing code is the hardest step for AI scientists. My main concern is that the newest model the paper names is Claude 3.5 Sonnet from mid-2024. The only other named model is GPT-4o, from May 2024, and it’s used only to build the benchmark and as a second judge.

Given this, it's hard to judge this paper cause I feel that the models have gotten so much better at coding since 2024, so I’m not sure “given the idea, code implementation is the hardest step” still holds today.

In the rebuttal phase, it would be great if the authors also ran the whole setup on the latest models (even under a small budget).

  1. I'm concerned about how the agent's written code is checked and might mean that the correct code written a different way can still score low. The agent is rewarded for things like matching function names (line 224-228) or how complex the code is (line 230-236) and one compares the first 50 loss values at each step (line 238-245). The low code scores might come from this, not from the agents directly (although sonnet 3.5 isn't a great coding model).

In the rebuttal phase, can you please give me examples of if this holds true?

  1. The paper is really talking about research

Some missing references (not critical):

  1. Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Questions for authors

Please see my questions in the weakness section.

1 comment

We thank the reviewer for identifying two central issues: evaluation with newer generators and whether Code scores distinguish scientific errors from valid implementation choices.

1. Evaluation with newer models

We extended all three phases to GPT-5.6 and Claude-4.6, using 100 matched quadruplets and 300 phase-level tasks per model. These human-validity scores use a 0–100 scale, separate from Table 3's native composites.

MeasureGPT-5.6Claude-4.6
Idea validity73.576.2
Code validity62.866.1
Reporting validity75.678.0
Code − Idea: difference [paired 95% CI]−10.7 [−15.2, −6.2]−10.1 [−14.5, −5.7]
Code − Reporting: difference [paired 95% CI]−12.8 [−17.0, −8.6]−11.9 [−16.4, −7.4]

Code has the lowest reported score for both models; all four intervals exclude zero. This extends the pattern beyond Claude 3.5 Sonnet, the original prompting backbone and primary subjective evaluator.

The summaries lack the complete human rubric and exact generation/judging configurations. Their ordering is therefore an observation, not a calibrated comparison of phase difficulty. Our central claim will emphasize mechanism-level errors that persist despite successful execution, supported by Section 4.4 independently of cross-phase comparisons.

2. Reference similarity versus scientific correctness

These are distinct properties. Execution checks whether code runs; Code–Idea Alignment (CIA) checks inspected logic against the supplied mechanism; symbol, complexity, and trajectory comparisons measure reference similarity. None alone proves correctness.

A concrete example in Section 4.4 is replacing requested sparse attention with standard attention: execution can succeed while the intended operation is absent. Conversely, renaming a helper without changing its computation is not a scientific error even if symbol recall falls. CIA assesses operations from an AST-derived summary, but remains a static semantic check, not runtime verification.

We tested intended behavior-preserving transformations:

TransformationEquivalence verifiedPass@1 preservationMedian ΔCodeP90 absolute ΔCode
Renaming98/10099%−1.64.8
Refactoring96/10097%−3.48.6

Deltas are points on the 0–100 scale. Residual changes are evident, particularly for refactoring. The summaries do not specify whether deltas were computed on verified-equivalent cases; component-level changes for that subset are unavailable. These tests therefore do not establish semantic invariance.

Of 100 mechanism mutations, 88 were detected; AUROC was 0.90, sensitivity 79% at 90% specificity, and mean Code-score decrease 18.7 points. The detection count and fixed-specificity sensitivity are separate summaries. They support sensitivity to tested errors, not perfect discrimination; the mutation mean is also distinct from the transformation medians.

We additionally tested loss offsets, with 100 tests at each level:

Offset / median reference lossRaw Dynamic Fidelity decreaseCentered-score change
0.1×5.80.7
0.5×17.60.9
1.0×29.41.2

Replacing L(θ) by L(θ)+c, for parameter-independent c, preserves gradients. When trajectories otherwise coincide, Equation (5)'s absolute discrepancy becomes |c|, lowering the raw score. Centering showed smaller empirical changes, but the nonzero changes do not establish exact offset invariance.

Thus, low Code scores do not invariably identify mechanism errors. We will clarify in Section 3.3 and Appendix J that raw Dynamic Fidelity measures short-horizon numerical agreement, report these sensitivity tests, and distinguish them from full validation of the composite.

3. Research scope and Paper2Code

AIS-Bench evaluates machine-learning research artifacts; Phase I is reference-conditioned reconstruction, not unrestricted discovery. We will add Paper2Code. Its paper-to-code task is directly relevant; AIS-Bench complements it through aligned Context–Idea–Code–Paper artifacts and separate interfaces for reconstruction, implementation from a supplied idea, and reporting from execution evidence. The abstract and Section 4.2 will foreground the scoped implementation finding.

If these responses address your concerns, we would appreciate reconsideration of your rating. Otherwise, we welcome specific feedback on the remaining concern and evidence needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.