- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces CreativeBench2.0, a benchmark for evaluating the ability of AI agents to discover and improve machine learning methods. The benchmark consists of 50 tasks across eight ML domains, constructed from published papers and their associated repositories. Each task restricts modifications to a single algorithmic component while keeping the surrounding training and evaluation pipeline fixed. Performance is measured against reproduced baselines across visible and hidden OOD settings. The benchmark includes two tracks: Discovery, which evaluates agents’ ability to develop and experimentally improve methods, and Foresight, which evaluates their ability to predict method rankings and OOD sensitivity without execution. Experiments across seven models show that reliable method discovery remains challenging, with an overall success rate of 14.6%. The paper also analyzes the effects of reference context and iterative experimentation, as well as common failure patterns in agent trajectories.
Critical strengths
-
I appreciate the emphasis on distinguishing genuine algorithmic improvements from gains introduced by changes to the surrounding experimental pipeline. The restricted edit scope, reproduced baselines, and hidden OOD evaluation provide a more controlled and interpretable setting than benchmarks that allow unrestricted modifications or rely on reported results.
-
As AI agents increasingly automate research workflows, evaluating their ability to discover better methods is an important direction. I particularly appreciate the separation between Discovery and Foresight, which captures two complementary aspects of scientific research. The evaluation across multiple models, trajectory-level failure analysis, and the repository-grounded task synthesis pipeline provide useful insights into current limitations and opportunities for future research.
Critical weaknesses
Overall, I think this is a solid and useful benchmark. I have two concerns that I think are worth discussing.
-
What counts as method discovery? The benchmark mainly measures whether an agent can improve performance by modifying a specific algorithmic component. However, some successful cases seem to involve reusing existing methods or combining known techniques, rather than coming up with genuinely new ideas. I understand that measuring novelty is difficult, but I wonder to what extent the current success rate actually reflects the “creativity” that the paper aims to evaluate.
-
The Iterative setting allows only three tests per task, which seems quite limited for algorithm discovery. The paper finds that iterative experimentation brings only a small improvement over One-shot, but I’m not sure how much we can conclude from this given the small budget. It would be interesting to see whether the results change with more iterations, or at least have some discussion of this limitation.
Questions for authors
-
Have the authors considered a more systematic analysis of whether successful submissions introduce genuinely new algorithmic mechanisms, rather than recombining or reusing existing methods? For example, how many successful submissions demonstrate meaningful mechanistic novelty beyond the provided baselines?
-
What motivated the choice of three official tests for the Iterative setting? Do the authors have any preliminary observations on whether increasing the experimentation budget could substantially improve Discovery performance, particularly for more challenging tasks?