- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Preparing reliable research tasks and runnable experiments for AI agents requires substantial manual work. CSForge automates paper selection, task design, evaluation rules, and experiment setup. Across 280 construction attempts using prefiltered papers, 80% produced environments that passed human review; on 49 selected tasks, the best agent scored 38.9 out of 100 when required to improve on existing methods by at least 5%.
Critical strengths
-
Automating the conversion of published research artifacts into executable, improvement-oriented environments addresses a meaningful bottleneck in developing research agents.
-
The evaluation includes 280 construction attempts across seven CS subfields, while the attack-and-repair comparison uses matched source papers and produces 588 solver trajectories.
-
The paper acknowledges problem-formulation errors, difficulty interpreting figures, and reproduction failures caused by experimental simplification, providing useful insight beyond aggregate success rates.
Critical weaknesses
-
The authors exclude tasks that neither Claude Opus 5 nor DeepSeek V4 Pro can execute before sampling the 49 evaluation tasks (page 15). This selection may affect task difficulty and model rankings, limiting the conclusions to the filtered tasks. Please state the full selection criteria in the main text and report how many tasks were excluded at each stage.
-
The proctor agrees with human judgments on the ten examples used to calibrate it, but no independent test results are reported. Since a single violation judgment zeros the entire task reward, incorrect judgments could affect both model scores and cheating statistics. Please provide independent validation results and the number of submissions zeroed by the proctor for each model.
-
The authors check baseline variation across three executions but do not clearly explain how baseline scores are calculated or whether new methods are also evaluated repeatedly. Reporting averages and variation across repeated runs would help establish whether the improvements are reliable.
-
SWE-Universe and EXP-Bench already automate some environment and task setup, so the paper should explain more clearly what CSForge adds. Besides, the reported cost of $10.67 per successfully synthesized task also leaves unclear whether it covers failed tries, deployment, compute resources, and human work. Comparing it with other task-building methods and providing full cost breakdowns would better show its higher efficiency and less manual effort.
-
Minor: Page 9, line 440: “The reseoning effort of all models are set to high” should be “The reasoning effort is xxxxx.”
Questions for authors
See weeknesses
Dear Reviewers and Area Chair,
Thank you for your time and thoughtful feedback. We have uploaded a revised manuscript with changes marked in blue. The revision incorporates the following updates from the rebuttal discussion.
We have done our best to address all concerns raised. We are encouraged that Reviewer yXqf has acknowledged our responses and raised their assessment. With the discussion deadline approaching, we would greatly appreciate any feedback from the remaining reviewers and welcome any further questions.
Thank you again for your time and consideration!
Best regards, The Authors