CSForge: Autonomous Construction of Verifiable Environments for Computer Science Research Agents

LLM-based agents are increasingly being developed for end-to-end computer science research (CSR), with further progress depending on reliable environments that can be constructed at scale to support their evaluation and improvement. Existing studies provide essential resources for evaluating research agents, but converting these resources into executable tasks remains challenging, as research objectives, experimental constraints, and evaluation protocols should be made explicit. We introduce CSForge, an automated pipeline that transforms published research papers and their released artifacts into verifiable CSR environments through three stages: paper filtering, task synthesis, and environment deployment. Drawing on human research experience, CSForge formulates structured research tasks and refines their evaluation protocols through iterative attack-and-repair to address shortcut solutions. It then reproduces reference experiments and constructs executable environments, checking their consistency with the source artifacts, reported findings, and repeated runs. Across sampled filter-passing papers from seven CS subfields, 86.8% of construction attempts yield executable environments, and 80.0% yield environments that pass human quality review. Experiments with six frontier models show that, even with access to baseline methods and executable environments, current agents struggle to consistently match or improve upon baseline performance. These findings highlight persistent limitations in research-agent performance across real research tasks and point to opportunities for further improvement. We would like to publicly release the complete CSForge pipeline and the resulting environment dataset to support further research.