acceptodds
Under review as a conference paper at ICLR 2027

Training AI Scientists to Replicate Research

Abstract

The replicability of papers is a cornerstone of scientific knowledge, ensuring the reliability of existing results and providing a base for further experiments. The act of replication typically illuminates details that were previously underspecified, and thus requires similar hypothesis-driven exploration to open-ended research. In this work, we develop Replica, a scalable task space for paper replication. To provide reward signal, we introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality. We post-train Faraday, a 27B-parameter "AI Scientist" agent that leverages coding agents as tools, surpassing the performance of Claude Opus 4.8 and GPT-5.5 on held-out replication tasks. Qualitative analysis of individual rollouts reveals that Faraday adopts a more scientifically-principled approach. We believe that our results provide a stepping stone towards AI agents capable of long-horizon scientific innovation without requiring complex harnesses.

Primary area:
reinforcement learning
Keywords:
AI Scientists, Paper Replication, Open-Endedness, Reinforcement Learning, AI for Science, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_3fxkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Faraday, a 27B AI Scientist trained to perform scientific research replication. The authors introduce Replica, a benchmark consisting of 310 tasks derived from ML and AI-for-science papers, where agents are asked to reproduce experimental results under limited time and computational resources. To evaluate these tasks, the authors develop a rubric-based LLM judge and use it as the reward for GRPO-based post-training. Faraday is trained to use a frontier coding agent as a tool for implementing and running experiments. Experiments show that Faraday outperforms strong coding agents such as Claude and Codex on both in-distribution and held-out tasks. Human studies and qualitative analyses further suggest that the rubric judge aligns reasonably well with human preferences and that Faraday exhibits more rigorous scientific behaviors, such as implementing the underlying experimental mechanisms rather than simply matching expected results.

Critical strengths
  1. The research problem has strong practical significance.
    This paper focuses on training an AI Scientist capable of conducting scientific research replication. This problem is important for improving the scientific capabilities of large language models, addressing the rapid growth in the number of research paper submissions, and improving the reproducibility of scientific findings.

  2. The experimental design is comprehensive and convincing.
    The authors compare Faraday with strong coding agents such as Claude and Codex, and further conduct a human study to validate the agreement between the rubric judge and human evaluations. In addition, the paper provides several qualitative analyses to further examine Faraday's improved capabilities in scientific research replication.

Critical weaknesses
  1. The experimental results rely heavily on the LLM-based rubric judge, leaving some uncertainty about whether the results reflect true scientific capabilities.
    Although the authors conduct a human study showing that the rubric judge has good agreement with human evaluations, the judge is not perfectly reliable. Moreover, both the training reward and the final evaluation of Faraday rely on the same judge. Therefore, it remains difficult to completely rule out the possibility that part of Faraday's performance gain comes from adapting to the characteristics of the benchmark and the judge, rather than from genuinely improved scientific capabilities.

  2. The coverage and generalization of the test tasks remain limited.
    Replica mainly focuses on a specific form of scientific task: reproducing a results figure from a given paper. Although these tasks involve a certain degree of openness, they still differ substantially from real scientific research in aspects such as problem formulation, hypothesis generation, experimental design, result interpretation, and the discovery of new knowledge. Therefore, the current experimental results are not yet sufficient to demonstrate that Faraday can generalize to a broader range of open-ended scientific research tasks.

Questions for authors
  1. How do the authors rule out the possibility that Faraday's performance improvement is partly due to overfitting to the rubric judge or the specific characteristics of the Replica benchmark?
    In particular, have the authors considered evaluating Faraday with an independent judge, human evaluators who are unaware of the model identity, or alternative evaluation criteria that were not used during training?

  2. To what extent can the capabilities learned from Replica generalize beyond paper replication tasks?
    Have the authors evaluated Faraday on other types of open-ended scientific tasks, such as hypothesis generation, experimental design, or scientific discovery, that are not directly based on reproducing a figure from an existing paper?

2 comments

Thank you for your careful review and useful feedback.

W1/Q1 – Potential overfitting to the judge

We accept that using the same rubric judge for training and evaluation was a weakness of the paper. Our human studies were a step towards validating this approach; our evaluators were unaware of the model identity. Additionally, we have run a suite of additional experiments to demonstrate that the improvement in capabilities that arises from our training regime generalises to independent judges. We invite you to take a look at the shared response above and let us know if these suitably address your concerns or if you would like to discuss further.

W2/Q2 – Task coverage and generality

Displaying generalisation to broad, open-ended scientific discovery is out of scope of this paper. We felt that constraining to a more open-ended variant of paper replication than in previous work would be more than sufficient as a first step, and that the quest for fully creative discovery of new knowledge should inevitably continue in future work. We discuss this in the opening paragraph of Section 5 in the paper.

However, we did include some preliminary experiments exploring early methods towards innovation in Figure 5. While we would not necessarily expect generalisation beyond our replication tasks, we do see some signs of it, including on the perturbed tasks (Figure 5, right), on the much larger compute budget experiments (Appendix A.1), and on our new PaperBench results (see the shared rebuttal response).

The contribution of this paper, though, centres on showing that it is possible to train with RL on highly underspecified, open-ended tasks, and identifying the ingredients we needed to make this work (e.g. judge access to the agent's whole trajectory, multiple judge samples for sufficient SNR, judge-generated turn-level credit weights). We view replication as an initial step towards training for creativity and innovation.

Thank you for your reply. My concerns have been addressed, and I will update my rating to a higher score.

Reviewer_PSmiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, a Qwen base model is post-trained as an orchestrator to a coding agent to replicate figures from research papers. This model/agent, called "Faraday", is trained on the training split of "Replica", which consists of 242 tasks from ML papers, and evaluated on the test split of "Replica", which consists of 68 tasks from AI for Science papers. The tasks are created from 100 research papers with the figured redacted by a Gemini model. Given a meta rubric, Opus 4.7 is used to generate a task-specific rubric. Codex is used to judge outputs against the rubric, and provides a reward and per turn credit assignment weights. Results are compared to ClaudeCode (Opus 4.8) and Codex (GPT-5.5).

Critical strengths

The individual training recipe techniques aren't novel but making it work is not trivial, so the fact that it clearly works in improving the base Qwen model (at least for the given evaluation setup) suggests that the recipe may be a useful resource for others.

Critical weaknesses

My concerns are mostly related to the evaluation design and setup.

Problem 1: WHAT is evaluated. The paper claims to evaluate/test paper replication capabilities, however the input are full papers with only the figure redacted, i.e. the agents likely have access to the results from e.g. the abstract, figure captions, so they can adjust/tweak their solution to fit the expected result - this is not a clean evaluation of research replication. It would be cleaner to redact any results from the paper, and provide only the input/dataset and methodology. Another evaluation design issue is that introducing the 60 min time limit and GPU limit makes any conclusions drawn from the scaled down figure versions less credible.

Problem 2: HOW it is evaluated. You're using the same LLM-rubric grader for training and for evaluation, so Faraday may have simply learned to game or better satisfy the rubric. Also, the comparison to Opus 4.8 (ClaudeCode) and GPT-5.5 (Codex) is not fair or consistent, and is set up in a way that makes it difficult or impossible to understand whether Faraday's better performance compared to these two models is due to the harness setup or due to training. On the one hand you evaluate how well Faraday can orchestrate the coding agent, while you're evaluating ClaudeCode and Codex on themselves, not in the same role of managing/orchestrating another coding agent. So the claim of having trained Faraday resulting to it outperforming these models is untrustworthy, because the difference might be just due to the setup, not training.

There are several other evaluation design issues, but the ones above are the two most critical ones. Without a strong evaluation setup, many of the claims and conclusions drawn from the results in the paper are hard to trust.

Questions for authors
  1. Have you evaluated the model on an independent benchmark? There are benchmarks that test agents on how well they can replicate research paper results, and claims of performance improvements on research replication would need to translate to be credible.

  2. Have you run the comparison to Claude Opus 4.8 and GPT-5.5 as orchestrators of a coding agent (see problem 2)?

2 comments

Thank you for your review and helpful feedback.

Novelty of training recipe

We’re glad you agree that the training recipe is a key contribution. We would like to further emphasize that our “coding agent as a tool” (CAT) method of training is itself a novel and important contribution, building on prior works we cited [6, 7], that unlocks the development of model capabilities complementary to those of frontier coding agents. Moreover, our advantage reweighting using judge-based turn-level credit assignment is a novel credit attribution technique, beyond that of [8], that enabled post-training in this long-horizon noisy-environment setting without collapse.

Problem 1.1 – Figure redaction

We agree that faithfulness to the methodology is extremely important, and the model tweaking the method to get the desired result was a key concern of ours too. Therefore, we specifically set up our judge so that it sees each agent's entire trajectory, rather than just its final submission. This allowed us to include judging rubrics that penalise any reward-hacking behaviour, including attempts to artificially aim for specific results. Appendix C.2 reports the implementation fidelity rubric score, which judges whether the underlying experiment actually implements and tests what the paper describes. We see that Faraday performs similarly to Claude and is Pareto better than Codex on this dimension.

However, we agree that it is valuable to shift perspectives and verify whether Faraday’s lead persists if we redact all results from the paper. We therefore ran the experiment you suggested. We picked the five least-cited papers from train and test and redacted any numerical results as well as any sentences discussing results. We ran 4 seeds per task per agent. We found Faraday maintained its lead, scoring 0.768 compared to 0.708 for GPT-5.5 and 0.690 for Opus-4.8.

Problem 1.2 – Resource constraints

We have replied to your points regarding resource constraints in the shared response.

Problem 2.1 and Question 1 – Judge overfitting and independent benchmarks

We have replied to your points regarding judge overfitting and independent benchmarks in the shared response.

Problem 2.2 and Question 2 – Comparing against Claude and Codex as orchestrators

Thank you for suggesting we evaluate Claude and Codex as orchestrators. Initially we had not done this under the assumption that, because both harnesses can already spawn subagents, there would be no gain in expressivity by doing so. However, prompted by your concern, we decided to indeed check this explicitly.

Evaluating Codex (GPT-5.5) in the same orchestrator role as our system on all 68 held-out test tasks, Codex orchestrating Codex scored 0.710, whereas our system orchestrating Codex scored 0.802. This shows that our advantage does not come purely from the orchestration setup.

[6] - ToolOrchestra: Elevating intelligence via efficient model and tool orchestration. 2025. https://arxiv.org/abs/2511.21689

[7] - Learning to orchestrate agents in natural language with the conductor. ICLR 2026. https://arxiv.org/abs/2512.04388

[8] - TIPS: Turn-level information-potential reward shaping for search-augmented LLMs. ICLR 2026. https://arxiv.org/abs/2603.22293

Over the last week we have also run another human study and posted the results in our shared response above. In the original paper submission, we looked just at human ratings on tasks where the judge preferred Faraday’s response. In this new study, we looked at human ratings on randomly sampled tasks. These new results provide additional evidence that our system has not learned to game the rubric judge as the performance gains transfer to expert human preference ratings. Thank you for your critical feedback, it prompted us to significantly improve our evaluation methodology across multiple independent benchmarks and alternative test-time judges.

Reviewer_BQFaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Replica, a benchmark of 310 figure-replication tasks constructed from 100 research papers. It then trains Faraday, a 27B model that plans and supervises experiments using a coding agent as a tool, through reinforcement learning on 242 tasks. The training combines task-specific rubric rewards, multiple judge evaluations, and turn-level credit assignment. Faraday achieves higher judge scores than frontier coding-agent baselines on 68 held-out AI-for-science tasks, with further experiments testing transfer to stronger coding tools and larger compute budgets.

Critical strengths
  1. The paper introduces Replica, a dataset of 310 figure-replication tasks with automatically generated task-specific rubrics and human checks of task quality.

  2. The paper goes beyond benchmarking by training an agent to plan and supervise replication experiments, with improvements in held-out research domains.

Critical weaknesses
  1. Agents (e.g., Codex) have internet access, so prompt instructions alone may not prevent them from using the original implementation. Did the authors check the trajectories to verify that agents followed this restriction?

  2. Replica automates task generation, but every task still requires human inspection and filtering. How much manual effort does this require, and can this process scale to a substantially larger dataset?

  3. The rubric judge’s advantage over the baseline judge in matching human judgments is not statistically significant (p = 0.109). Since the same judge is used for training and evaluation, human evaluation on randomly sampled test outputs would help confirm that the score gains reflect better replication quality.

Questions for authors

See Weaknesses.

3 comments

Thank you for your review, and for your pertinent questions.

Checking whether agents looked up the paper

We have examined this further in response to your question. Across all 2,480 evaluation rollouts of the final checkpoint of our agent, we found that in only 16 (0.6%) the model looked up the paper on the internet, and in 15 of these 16 cases the judge correctly penalized it so that it scored lower than in other rollouts from the same task without cheating. Therefore, as intended, the judge's “anti-cheating” penalty outweighs any gain, and this explains why the model did not learn to reward hack in this way. We have included this result in Section 4.3 of the updated paper.

Automated task generation

Thank you for pointing out that we presented task generation as automated but also said that we inspected every task, and prompting us to clarify. The task generation process itself is completely automated and scalable to much larger datasets. When we came close to publishing the paper, we felt it was our responsibility to ensure quality, and so built an efficient pipeline to check the redaction by eye (taking ~5 seconds per paper, so about 30 minutes of work across all tasks). This resulted in us fixing the captions in 17% of tasks, as reported in Table D.1.

If we wanted to scale the creation of high-quality tasks by 1000x, this would be possible with sufficiently many human raters. Moreover, we expect that the error rate would be much lower with the base models of today compared to those of 6 months ago.

Rubric judge and human evaluation of replication quality

Yes, we agree that the rubric judge’s advantage over the baseline judge in matching human judgement is not significant, and have updated the caption of Figure 3 to reflect this more clearly. Improvement over baseline aside, the absolute agreement of the rubric judge with humans is enough to inspire some confidence that the gains in replication quality track human judgement, but the human study you suggest would indeed be the best way to show this cleanly.

We ran the human evaluation you suggested on randomly sampled test outputs as a stricter evaluation of whether our judge score gains reflect better replication quality. Please see our shared response above for further details.

Thanks for your reply. I think my concerns are addressed. I will raise my score.

Reviewer_jPDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Replica, a dataset of 310 figure-replication tasks drawn from 100 papers, aimed at the downstream task of replicating scientific papers. The authors then present Faraday, a Qwen3.6-27B model post-trained with a GRPO variant. Faraday handles the planning and delegates coding tasks to the Codex CLI. The authors report that their method outperforms Claude Code and Codex used on their own.

Critical strengths
  • Well-motivated problem. Automating paper replication is an important and timely goal that could help address reproducibility concerns in machine learning and related fields.

  • Transparency. The paper documents its system in considerable detail, including the agent, task, and baseline prompts, example judge rubrics, the multi-stage training recipe, the infrastructure, and the human-study protocols.

Critical weaknesses
  • Task selection and resource constraints.

    • Papers were hand-picked for being “well-known” and easy for humans to rate, and every task was then manually filtered. This can introduce biases in the task set toward canonical, highly cited results that are likely to replicate. Such papers are also likely to be in the pretraining data, and redacting the figure does not stop a model from recalling the original results. The authors discuss contamination (Appendix D.1) but do not measure its effect.

    • Each task allows only “a 60-minute time limit, and a single one-seventh MIG slice of an H200 GPU.” I understand that compute is an issue for tasks like this, but this budget is far below many modern training-heavy papers. The only larger-scale evidence (Appendix A.1) is 8 tasks with a single rollout each.

  • Novelty. Faraday is a Qwen3.6-27B planner that delegates all coding to the Codex CLI, which uses GPT-5.5, the same coding agent used as the “Codex” baseline. The comparison is therefore “Codex + trained planner” versus “Codex alone”. The abstract’s framing, that a 27B model surpasses GPT-5.5, is kind of an overstatement.

  • Lack of baselines and external benchmarks

    • The paper proposes a system for replicating scientific experiments but does not compare it with existing replication systems. This is despite the authors naming the closest prior work themselves (Zhao et al., 2026; Seo et al., 2026). A non-exhaustive list: AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage (Zhao et al., ACL 2026), Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning (Seo et al., ICLR 2026)

    • All results come from the authors’ own task space, scored by the same judge Faraday was trained against. I would be more interested in seeing whether the same improvement holds against well-established datasets such as PaperBench or CoreBench.

Questions for authors

NA

3 comments

Thank you for your review and for acknowledging the strengths of our work.

Selection of well-known papers

Our selection of papers intentionally introduces a bias in the training set toward canonical, highly-cited results. We made this design choice to facilitate efficient human review of trajectories and to avoid validity of the papers complicating RL training. If we had chosen lesser known papers, there is a risk the paper could not be replicated and the agent would be penalised in RL for failing to complete an impossible task. However, generalising to the attempted replication of results that may or may not be valid – i.e. falsification – is an excellent next step in future work towards innovation. We have now expanded our discussion of this in the paper (see Section 5).

To your second concern, about pre-training contamination: yes, some of our training and evaluation papers will have been in the pretraining data of the models in question. This was something we chose to accept to enable us to use papers spanning a range of years, and is a property shared with prior benchmarks such as PaperBench and Paper2Code. However, we believe that contamination is relatively unimportant in our setting. The process of replication requires filling in many details the paper does not contain, including designing experiments, iterating on intermediate results, analysing data, and scaling down claims to fit reduced resources. To test this hypothesis, we generated 20 additional tasks, from 10 machine learning papers released after January 2026 (the latest cutoff among the models we used). The relative ranking of agents remained the same (specifically, Faraday scored 0.849, Claude 0.800 and Codex 0.793). Thus our agent’s replication capability, as measured by the rubric judge, generalises to papers not in the pre-training datamix.

Resource constraints

We have replied to your points regarding resource constraints in the shared response.

Abstract framing

We agree that the abstract framing could be seen as an overstatement without the context from the introduction, and have edited the sentence to:

“We post-train Faraday, a 27B-parameter “AI Scientist” agent that leverages coding agents as tools. Faraday uses GPT-5.5 as a tool to achieve stronger performance than Claude Opus 4.8 and GPT-5.5 on held-out replication tasks.”

Baselines and external benchmarks

We appreciate the nudge here. We agree that testing generalisation to existing benchmarks is interesting and would strengthen the work. We have therefore run several new experiments, testing our agent on PaperBench and Paper2Code, which we discuss in the shared response and in the updated paper (Appendix A.5).

Thank you for your response. The additional experiments addressed part of my concerns (weakness 3), but the major ones remained. Therefore, I will maintain my assessment.

Thank you for considering our response. We would be grateful if you could explain which of our responses did not address your concerns and why, as we are keen to improve the paper. We have attempted to further address those we think you may be referring to:

W1.1 Regarding Task Selection

We explicitly tested your concern of pretraining data contamination by evaluating 20 new tasks published after the cutoff of these models and found Faraday still outperformed both Claude and Codex. This is described in the response above, see “To test this hypothesis, we…”. Could you clarify whether this addressed your concern, and if not then please help us understand why.

W1.2 Regarding Resource Constraints

As described in the shared response (sorry if this was hard to find), “the GPU budget of just the inner-loop experiments was ~5,300 H200-hours”. This is already extremely high and hence increasing the training budget significantly beyond this was not feasible. Moreover, in Appendix A.1 we show that training on this budget is sufficient to teach skills that generalise to much higher budgets during evaluation.

Since our initial response we have now run another experiment further testing the ability of all agents to generalise to larger resource budgets. We evaluated our system against Codex on 61 tasks from 28 papers, each given 2 to 12 hours and 1 to 8 B300 GPUs. Our system scored 0.80 against 0.68 for Codex, and scored higher on 54 of 61 tasks (mean per-task gain +0.115 ± 0.015 s.e.). We have added these new results to Appendix A.1 in the updated paper. We would be more than happy to discuss if you have any remaining reservations.

W2 Regarding Novelty

We ablated your concern regarding “Codex + trained planner” versus “Codex alone” by evaluating Codex (GPT-5.5) in the same role as our model on all 68 held-out test tasks. Codex orchestrating Codex scored 0.710, whereas our model orchestrating Codex scored 0.802. This shows that our contribution cannot be reduced to just adding an orchestrator role.

Moreover, we made several algorithmic contributions including the coding agent as tool (CAT) paradigm, novel judge-generated turn-level credit assignment weightings, judge access to the entire trajectory, and several others as explained in Section 3 of the paper.

We have also rephrased the abstract based on your feedback and would be more than happy to make further edits if you have remaining reservations.

Thank you for engaging in the discussion. We would really appreciate any more specific details of your concerns.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Shared Response

We thank all reviewers for your thoughtful and constructive feedback. We have updated the paper in response to your observations and have run various additional experiments which we hope address your concerns.

We gently ask that you consider updating your score in light of our revisions, and would be delighted to discuss any remaining concerns with you.

Judge overfitting and independent benchmarks

Reviewers wondered whether our performance improvements might come from overfitting to specific biases in the judge rather than learning meaningfully useful behaviours. To investigate whether this was the case, we have run several additional experiments. The results of these experiments indicate that our system does indeed learn behaviors that generalise across judges and to other published benchmarks, and we have incorporated them into the appendices in the revised paper.

Firstly, to test for overfitting to the judge, we evaluate our models at different points in training using a panel of independent LLM judges. The new judges are based on a framework of pair-wise comparisons: given two trajectories the judge is asked which is better [1]. This is deliberately a different setting to our rubric-based judge from the paper, which assigns each trajectory a numerical score. Specifically, we used GPT-6 Astra, Claude Opus 5.5 and Gemini 3.1 Pro to create three new judge prompts, within thinking mode max, max, and high respectively. In this “judge-generator” capacity, the models received the replication task instructions, six example captions drawn at random, and the instruction to create a prompt for a judge. The judge-generators were not given access to the rubric judge prompt.

The panel of judges comprises these three judges run using each of GPT-6 Astra, Claude Opus 5.5 and Gemini 3.1, thus creating 9 judge-model combinations. We re-judged 136 trajectories (2 seeds of each of the 68 test tasks) from our system on checkpoints from the beginning (i.e., untrained model), middle, and end of training. We found that, according to this new panel of comparative judges, our system consistently improves over training with the win rates of the final vs. untrained system at 88%, middle of training vs. untrained at 75%, and final vs. middle of training at 64%. This suggests that the model learns generally useful behaviour over training rather than just overfitting to one specific judge. We have added this Appendix A.4 in the updated paper.

Secondly, we ran new experiments on two independent benchmarks: PaperBench [2] and Paper2CodeBench [3]. On PaperBench our system scored 40.3%, while GPT-5.5 alone scored 29.4%. For context, another recent paper DisCo+AREXSkill [4] also reports that GPT-5.5 alone scored 29.5% on PaperBench while their agent (GPT-5.5 with a harness containing skills distilled from github that are dynamically loaded) scored 39.6%. On Paper2CodeBench, our agent scores 2.99±0.07 out of 5 compared to PaperCoder [3] score of 2.71±0.06. For more details on both new experiments please see Appendix A.5 in the updated paper.

[1] Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. NAACL 2024. https://arxiv.org/pdf/2306.17563

[2] PaperBench: Evaluating AI’s Ability to Replicate AI Research. ICML 2025. https://arxiv.org/abs/2504.01848

[3] Paper2Code: Automating code generation from scientific papers in machine learning. ICLR 2026. https://arxiv.org/abs/2504.17192

[4] DisCo+AREXSkill, https://arxiv.org/abs/2609.02749

Resource constraints

Some reviewers observed that the per-replication experiment budget constraints are tight. This is a necessity with a meta-learning training setup. Indeed, it is unusual to have any GPU budget for experiments in the inner loop of RL; similar large-scale published work gives agents access to 1 GPU for 90 seconds on each RL step [5]. By contrast, in our experiments every rollout had access to 1 MIG slice of an H200 GPU for 1 hour (30 minutes initially), and with 80 of them needed for a single RL step it meant that the total GPU budget of just the inner-loop experiments was ~5,300 H200-hours.

We see the budget constraints as a feature, not a bug, in our task design. Learning to faithfully reproduce an experiment while scaling down to the resources at hand forces an agent (human or AI) to extract and rigorously test the “core research insight”, an important scientific skill. We discuss this in Section 3.2.

We do, however, agree that generalisation to replication with more time and compute than during training is an interesting and important property to check for. We conducted an initial study on evaluating agents with more resources (8 hours on 8 B300s) in Appendix A.1, showing that our agent outperforms Claude on 5 of the 8 tasks, providing weak but positive evidence of generalization.

Running this scaled-up evaluation across all train and test tasks would have incurred at least 800x evaluation cost relative to the original setting, which is infeasible. We do expect that training across a distribution of resource budgets would be a very interesting direction for future work, requiring some infrastructure changes, and have added this to the discussion in Section 5.

[5] Synthetic Sandbox for Training Machine Learning Engineering Agents, https://arxiv.org/pdf/2604.04872

Correction on reasoning effort

While preparing this response, we found that in Section 4 we stated that we use xhigh reasoning effort, whereas we actually ran at high for Claude Opus 4.8, medium for GPT-5.5 and medium for Faraday's GPT-5.5 coding tool. We have corrected this in the paper, and, for completeness, have rerun all three at xhigh. At xhigh the ordering still holds: Faraday 0.814, Claude 0.779 and Codex 0.771, against 0.791, 0.748 and 0.729 at the previous default efforts. We have corrected the text and added the xhigh results to Appendix A.3.

Human Expert Evaluation

A concern raised by reviewers was the significance of our initial human study results that focussed on rollouts in which the rubric judge deemed that Faraday held a strong advantage. We are pleased to announce that we have now run a new more general human study with 22 expert raters and 117 sets of ratings. Each rater was given 4 samples from a randomly sampled task from the test set: 2 from our system, and 2 from codex alone. We ran a binomial mixed-effects model with task and rater random effects on these randomized blind rankings and we found that the human raters preferred our system in 61% of cross-system pairs (one-sided p = 0.033). This is a significantly stronger result than we had in the original submission, so we have replaced the previous human study in Appendix F.2 and updated the paper introduction and Section 4.4 to reflect this. Thank you for encouraging us to run this!

MeasureOursChancep
Preferred in cross-system pairs*61%50%0.033
Ranked first70/117 (60%)50%0.021
Codex ranked last72/117 (62%)50%0.008

* Binomial mixed-effects model with task and rater random effects; the other rows use one-sided exact binomial tests over rankings.

Area_Chair_BV2zOpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.