acceptodds
Under review as a conference paper at ICLR 2027

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

Abstract

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose oral and spotlight papers from top ML/NLP conferences by preserving the datasets, evaluation harness, and baseline implementations but withholding the paper’s proposed method. This results in seven containerized execution environments comprising 54 gradable evaluation tasks. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. Across bounded-time and extended evaluations of frontier LLM agents, agents improve over the provided baselines in only 57 of 235 evaluations and complete only 20% of tasks on average. We identify recurrent difficulties with time and resource management, confidence in weak hypotheses, and coordination of parallel experiments. Yet rare runs, agents match the performance of methods developed by expert researchers, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably.

Primary area:
datasets and benchmarks
Keywords:
AI R&D, Evaluation, Automation, AI for Research, Auto Research, LLM Agents, Long Horizon Benchmark, RL Environments, Frontier Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uiTHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ResearchGym converts seven AI research papers into executable environments with 54 scorable subtasks, retaining datasets, baselines, and evaluation code while withholding the proposed methods. Its core contribution is reusable infrastructure and trajectory resources for evaluating closed-loop research agents, accompanied by evaluations of eight LLMs across three agent frameworks. Agents outperform the provided baselines in only 57 of 235 runs, highlighting the gap between occasional strong results and reliable research performance.

Critical strengths
  1. Meaningful evaluation. Combines hypothesis generation and actual experimentation with objective grading and published expert references.
  2. Careful construction. Manually validated, containerized environments support reproducible evaluation on a single GPU.
Critical weaknesses
  1. Narrow independent coverage. The 54 subtasks derive from only 7 research environments, limiting generalizability.
  2. Remaining confounding. Single-seed cross-framework ablations with many missing results insufficiently disentangle model and framework effects.
Questions for authors

See Weaknesses. As a benchmark paper, attaching no material (e.g. linking to an anonymous repo, or zipping data samples) reduces credibility.

1 comment
Area_Chair_rygMOpenReview ↗

I like to encourage the reviewer to discuss with the authors on the replies

I like to encourage the reviewer to discuss with the authors on the replies

Reviewer_8ZrBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Tackling the problem of evaluating using AI (agents) for research, this work proposes a benchmark named ResearchGym to assess AI models' capabilities in doing research, including ideation and experimentation. The agent is given a task (the research goal, experimental constraints, and baseline scores) with a base repository and an execution-based (i.e., non-LLM-based) grader for scoring. For constructing the task set, the authors gather oral, spotlight, and highlight papers published at ICLR, ICML, NeurIPS, and ACL in 2025 with LLM-based filtering and human-involving quality assessment, followed by the "task packaging" stage where they build a starter/skeleton repository that excludes the original authors' implementations with validation for reproducibility for each paper, resulting in 7 papers (and thus 7 research environments) and 54 gradable tasks. For evaluation, they use "task-native scores," which align with the original papers' metrics (e.g., accuracy, F1 score), and "task-agnostic summary metrics," which include normalized scores (defined based on the baseline scores). Based on the analysis, the authors suggest that existing AI models can occasionally achieve performance comparable to expert-developed methods but remain unreliable in conducting end-to-end research, often struggling with time and resource management, overconfidence in weak hypotheses, and coordination of parallel experiments.

Critical strengths
  • Evaluating AI agents on scientific research is a complex task, and this work tries to provide a comprehensive evaluation suite with multiple considerations, such as reducing possible knowledge contamination, constructing realistic research environments from published papers, and keeping the tasks computationally accessible.
Critical weaknesses
  • Overall, I think this work has claims that are not sufficiently supported. For instance, while the authors state "ResearchGym combines the aspects of ideation and experimentation, evaluating LLM agents in executable research codebases with objective scores." (L073) and "(1) Full-loop evaluation: tasks must require both ideation and experimentation" (L098), the evaluation metrics measure the performance scores (e.g., task-native scores) rather than any ideation aspects. Good ideation can help increase performance scores, but higher performance scores don't necessarily mean good ideation, as they may also reflect implementation, debugging, hyperparameter tuning, benchmark-specific optimization, etc. Another example is "A scalable data curation pipeline for turning research papers with verifiable metrics into an execution environment, outlining a path towards live benchmarking high-quality trajectory generation (§2.2)." (L077), whereas the described pipeline involves human effort at multiple stages: "A subset of 100 extractions were manually validated every sample to be correct." (L156, for "Stage-1: Automated extraction and filtering"), "We manually assess the feasibility of 90 shortlisted papers and finalize 7 environments (Table 2) with diverse representation of various domains, resulting in 54 tasks." (L162, for "Stage-2: Human Selection"), "we manually resolve each of these with two-way diff reconciliation between two authors." (L1199, for task packaging), "To ensure reproducibility, we manually run the paper’s original method" (L1223, for reproducibility). Given these manual steps and the lack of reported human effort or construction time per environment, the scalability of the overall pipeline remains unclear.
  • Similarly, I think this work could have clearer positioning regarding its contributions and novelty. While it is great that this work cites multiple existing papers in this domain and provides a comparison (Table 1), I am not fully convinced that the comparison sufficiently establishes the distinct contributions of ResearchGym. For instance, InnovatorBench already constructs open-ended research tasks from published papers by removing the original methods from runnable repositories, and EXP-Bench proposes a semi-automated pipeline for constructing research tasks from papers. This is particularly relevant because "data curation pipeline" (L077) as a whole is claimed as one of the main contributions. While Table 1 highlights differences in benchmark properties, these differences do not necessarily establish substantial methodological contributions/novelty over the closest prior work.
Questions for authors

Please refer to the Critical Weaknesses section.

1 comment
Area_Chair_rygMOpenReview ↗

I like to encourage the reviewer to discuss with the authors on the replies

I like to encourage the reviewer to discuss with the authors on the replies

Reviewer_PWjGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new benchmark, ResearchGym, for ML research agents. Each task is based on an ML conference paper: the datasets, baselines, and metrics of a paper a given, and the method is withheld, and the agent is asked to achieve good scores on the metric.

Critical strengths

ML research tasks is an interesting domain, and the paper does a through job of curating the environments, including checking for contamination. The analysis of agent behavior on these tasks is fairly extensive.

Critical weaknesses
  • One big framing concern I have with the paper is that research is not just hill-climbing on a verifiable objective. First, defining the problem + evaluation is a big part of the research itself. Research is meant to be generalizable in a way that just doing well on a dataset doesn't capture. It's the same as working on a homework problem that has a solution is different from working on an open problem. I think what is done here is more akin to ML engineering, and I'd recommend that the authors discuss this. I don't think "open-ended research goal" is an accurate characterization. To be clear, I don't have a problem with the work, just the framing.
Questions for authors
  • It seems like the agent is simply rewarded on the metric, but not on whether it came up with a novel idea and whether that idea is similar to the one in the original paper. I think these metrics are important to understand the nature of the solutions.
  • I think the paper could give more intuition into the type of tasks. For example, for the 'continual learning' task, I couldn't find much information about the metrics and the task description. Without more details, it's hard to evaluate the quality of the benchmark.
  • Some of the Qualitative Analysis in the appendix is important enough that I think it should go into the main body of the paper.
  • It would be interesting to see a detailed breakdown of all the papers considered and what fraction of them failed what kind of checks, to understand the distribution of papers that have been accepted here.
  • This work was done before Astra and Opus 5.5, but I am very curious to know how far the current frontier has come.
1 comment
Area_Chair_rygMOpenReview ↗

I like to encourage the reviewer to discuss with the authors on the replies

I like to encourage the reviewer to discuss with the authors on the replies

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Dear Area Chair,

We thank you for encouraging the discussion. We summarise the discussion below:

The reviewers recognised ResearchGym as a meaningful evaluation of end-to-end research, combining ideation and real experimentation with objective grading against expert references. They also noted careful construction of the environments, thorough contamination checks, reproducibility on a single GPU and the extensive analysis of agent behaviour.

We replied to all reviewers on 11 Nov and sent follow up reminders, but have not yet received a response from any of them

Concerns raised and our response

  1. Is this research or hill-climbing on a metric? (PWjG) We agree that defining the problem and evaluation is outside our scope, and will reframe the paper as "open-ended method development on fixed research goals". Within that scope, the evaluation is intentionally designed to discourage narrow hill-climbing, since a method must generalise across several tasks per environment

  2. Does the benchmark separate ideation? (8ZrB, PWjG) Our ablations already show that ideation and execution are bottlenecks that cannot be cleanly separated, which supports full-loop evaluation. Additionally, the subjectivity of ideation evaluation is an unsolved problem, with major pitfalls (L40-43), which is one of the motivations of our work.

  3. How does this differ from EXP-Bench and InnovatorBench? (8ZrB) We present substantial methodological and empirical differentiations, particularly in withholding the method, reconstructing full paper-level environments, multi-layered contamination controls and overall evaluation integrity

  4. Are 7 environments enough? (uiTH) The count follows from strict selection, with 7 of 1,387 candidate papers passing all five principles, and from the cost of 24-72 hour runs. Coverage is comparable to or larger than RE-Bench, MLRC-Bench, MLGym and InnovatorBench, and the benchmark is designed to be extended with post-cutoff papers

New results

  1. Current frontier models While GPT-6 Astra and Opus 5.5 beat the baseline on 6/7 and 5/7 envs, they beat the source paper on only 3/7 each. The paper's conclusion therefore holds for the newest models

  2. Full candidate-pool breakdown We labelled all 1,387 candidate papers against our five selection principles and released the per-paper labels with rationales

Points clarified from existing material

Several concerns were already covered in the submission

  • No material attached (uiTH): the anonymous repository with environments, evaluation code and trajectories is linked in the main paper (L90-91)
  • Single-seed results (uiTH): the main results use 3 seeds with bootstrap CIs (Table 16), and only the supplementary cross-scaffold ablation is single seed
  • Missing results (uiTH): these are runs where the agent failed to produce a valid submission, not missing measurements
  • Task details (PWjG): task descriptions, metrics and baselines for every environment are in Appendix L
  • Nature of the solutions (PWjG, 8ZrB): the hint ablation, idea-similarity analysis and qualitative analysis are in Appendix I.1, I.3 and L

We believe our responses address every concern raised, and that there are no critical weaknesses remaining.

We thank you for your time and consideration

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.