- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ResearchGym converts seven AI research papers into executable environments with 54 scorable subtasks, retaining datasets, baselines, and evaluation code while withholding the proposed methods. Its core contribution is reusable infrastructure and trajectory resources for evaluating closed-loop research agents, accompanied by evaluations of eight LLMs across three agent frameworks. Agents outperform the provided baselines in only 57 of 235 runs, highlighting the gap between occasional strong results and reliable research performance.
Critical strengths
- Meaningful evaluation. Combines hypothesis generation and actual experimentation with objective grading and published expert references.
- Careful construction. Manually validated, containerized environments support reproducible evaluation on a single GPU.
Critical weaknesses
- Narrow independent coverage. The 54 subtasks derive from only 7 research environments, limiting generalizability.
- Remaining confounding. Single-seed cross-framework ablations with many missing results insufficiently disentangle model and framework effects.
Questions for authors
See Weaknesses. As a benchmark paper, attaching no material (e.g. linking to an anonymous repo, or zipping data samples) reduces credibility.
I like to encourage the reviewer to discuss with the authors on the replies
I like to encourage the reviewer to discuss with the authors on the replies