acceptodds
Under review as a conference paper at ICLR 2027

CSForge: Autonomous Construction of Verifiable Environments for Computer Science Research Agents

Abstract

LLM-based agents are increasingly being developed for end-to-end computer science research (CSR), with further progress depending on reliable environments that can be constructed at scale to support their evaluation and improvement. Existing studies provide essential resources for evaluating research agents, but converting these resources into executable tasks remains challenging, as research objectives, experimental constraints, and evaluation protocols should be made explicit. We introduce CSForge, an automated pipeline that transforms published research papers and their released artifacts into verifiable CSR environments through three stages: paper filtering, task synthesis, and environment deployment. Drawing on human research experience, CSForge formulates structured research tasks and refines their evaluation protocols through iterative attack-and-repair to address shortcut solutions. It then reproduces reference experiments and constructs executable environments, checking their consistency with the source artifacts, reported findings, and repeated runs. Across sampled filter-passing papers from seven CS subfields, 86.8% of construction attempts yield executable environments, and 80.0% yield environments that pass human quality review. Experiments with six frontier models show that, even with access to baseline methods and executable environments, current agents struggle to consistently match or improve upon baseline performance. These findings highlight persistent limitations in research-agent performance across real research tasks and point to opportunities for further improvement. We would like to publicly release the complete CSForge pipeline and the resulting environment dataset to support further research.

Primary area:
datasets and benchmarks
Keywords:
Agents, Environment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_frasOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Preparing reliable research tasks and runnable experiments for AI agents requires substantial manual work. CSForge automates paper selection, task design, evaluation rules, and experiment setup. Across 280 construction attempts using prefiltered papers, 80% produced environments that passed human review; on 49 selected tasks, the best agent scored 38.9 out of 100 when required to improve on existing methods by at least 5%.

Critical strengths
  1. Automating the conversion of published research artifacts into executable, improvement-oriented environments addresses a meaningful bottleneck in developing research agents.

  2. The evaluation includes 280 construction attempts across seven CS subfields, while the attack-and-repair comparison uses matched source papers and produces 588 solver trajectories.

  3. The paper acknowledges problem-formulation errors, difficulty interpreting figures, and reproduction failures caused by experimental simplification, providing useful insight beyond aggregate success rates.

Critical weaknesses
  1. The authors exclude tasks that neither Claude Opus 5 nor DeepSeek V4 Pro can execute before sampling the 49 evaluation tasks (page 15). This selection may affect task difficulty and model rankings, limiting the conclusions to the filtered tasks. Please state the full selection criteria in the main text and report how many tasks were excluded at each stage.

  2. The proctor agrees with human judgments on the ten examples used to calibrate it, but no independent test results are reported. Since a single violation judgment zeros the entire task reward, incorrect judgments could affect both model scores and cheating statistics. Please provide independent validation results and the number of submissions zeroed by the proctor for each model.

  3. The authors check baseline variation across three executions but do not clearly explain how baseline scores are calculated or whether new methods are also evaluated repeatedly. Reporting averages and variation across repeated runs would help establish whether the improvements are reliable.

  4. SWE-Universe and EXP-Bench already automate some environment and task setup, so the paper should explain more clearly what CSForge adds. Besides, the reported cost of $10.67 per successfully synthesized task also leaves unclear whether it covers failed tries, deployment, compute resources, and human work. Comparing it with other task-building methods and providing full cost breakdowns would better show its higher efficiency and less manual effort.

  5. Minor: Page 9, line 440: “The reseoning effort of all models are set to high” should be “The reasoning effort is xxxxx.”

Questions for authors

See weeknesses

Reviewer_yXqfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CSForge is a framework for autonomously constructing verifiable computer science research (CSR) agent environments from published research papers and their released artifacts. It consists of three stages: paper filtering, which identifies papers suitable for environment construction; task synthesis, which derives research tasks and experimental protocols from each paper’s main experiments, using an iterative attack-and-repair process to identify and mitigate evaluation loopholes; and environment deployment, which follows a reproduce-then-build approach to reproduce baseline experiments, extract streamlined implementations, and establish executable evaluation harnesses. The framework validates environments through implementation consistency, claim consistency, and run-to-run consistency checks. Evaluated on 16,962 papers from 13 top-tier conferences across seven CSRankings subfields, CSForge achieves an 86.8% executable-environment rate among sampled papers passing the viability filter, with 80.0% passing human quality review. On a 49-task benchmark, the best model achieves 70.8% when matching or outperforming the baseline, but only 38.9% when requiring at least 5% relative improvement, highlighting the remaining difficulty of producing meaningful research improvements under verifiable evaluation.

Critical strengths
  1. A systematic pipeline for constructing verifiable research environments. CSForge integrates paper filtering, task synthesis, and environment deployment to transform published research papers and their released artifacts into executable CSR environments. Its reproduce-then-build strategy grounds the evaluation in original experimental artifacts rather than relying solely on LLM-generated task specifications, making the resulting environments more practical and reproducible.
  2. Rigorous validation of evaluation protocols and environment quality. The iterative attack-and-repair procedure explicitly probes shortcut solutions, while implementation consistency, claim consistency, and run-to-run consistency checks address complementary aspects of reliability. The authors further validate the pipeline through human review and a downstream benchmark of 49 tasks across seven CS subfields. The reported 80.0% human-review pass rate provides empirical evidence that the approach can produce useful research environments at scale.
Critical weaknesses
  1. The reliability of automatically synthesized evaluation protocols remains uncertain. Although CSForge uses iterative attack-and-repair to identify shortcut solutions, the paper acknowledges that this process is less effective at detecting violations of method requirements expressed in natural language. This leaves room for agents to exploit weaknesses that automated checks fail to identify, potentially affecting the validity of benchmark scores.
  2. The validation does not fully establish the scientific validity of the generated environments. The consistency checks and human review provide useful evidence of implementation fidelity and reproducibility, but reproducibility alone does not guarantee that a task captures the intended scientific objective or that its evaluation protocol adequately measures genuine methodological progress. The 49-task benchmark demonstrates the utility of the environments for comparing agents, but does not by itself establish that the synthesized tasks comprehensively represent real-world research capability.
Questions for authors
  1. Attack-and-repair robustness The paper reports that attack-and-repair has limited effectiveness in detecting violations of method requirements stated in natural language. How much does this limitation affect the validity of the resulting evaluation protocols, and to what extent can the proctor agent mitigate it?
  2. Faithfulness of task synthesis The paper identifies cases where synthesized problem formulations depart from the prescribed template and evaluation protocols fail to align with the source paper’s main objectives. How are these issues reflected in the human quality review, and what criteria determine whether an environment faithfully captures the intended scientific problem?
Reviewer_bqycOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CSForge is an automated pipeline that turns published CS papers + their released artifacts into verifiable research environments for computer-science-research (CSR) agents. An environment is a tuple E = (P, R, C, {(m_j^base, d_j)}, H): a method-agnostic problem statement, curated references, a containerized execution environment, reproduced baseline values, and a scoring harness. Baselines and the harness are hidden from the solver; a proctor agent audits trajectories for shortcuts and can zero the reward. Three stages: (1) paper filtering​ — coarse rule-based screen (public artifacts, no exotic hardware, affordable scale) plus fine-grained LLM rubric (objective metrics, not theory/profiling/large systems); (2) task synthesis​ — reconstruct the scientific problem via a fixed template (background / challenges / limitations / target capability), then design an evaluation protocol with strong baselines and "coverage + headroom," refined by an attack-and-repair​ loop in which subagents hunt for shortcut solutions (≤5 rounds); (3) environment deployment​ — reproduce-then-build in Docker, then validate-and-repair against three consistency checks (implementation, claim, run-to-run), ending in PASSED / BLOCKED / REJECTED. Implemented as Claude Code skills with an LLM author agent (Opus 4.7 for synthesis), ~$10.67 per successful task.

Critical strengths
  1. It converts "scalable environment construction" from an aspiration into a measured, reproducible capability — and that is the field's actual bottleneck.​ Prior objective-metric environments (MLRC-Bench, AIRS-Bench, MLS-Bench, MLGym) are all manually curated, so their size is capped by human labor; CSForge's differentiator is precisely that it removes that cap. The paper does not merely assert this — it runs the pipeline over a large, real corpus (16,962 main-track papers, 13 conferences, 7 CSRankings subfields) and reports the yield where it counts: 86.8% of sampled filter-passing papers reach an executable environment and 80.0% reach one that survives three-expert majority review, i.e. 224 validated environments, at a stated $10.67 per successfully synthesized task. That cost-per-task figure is what makes the scalability claim falsifiable and actionable: a reader can now compute what a 1,000-task dataset costs, which no prior manual-curation paper lets you do. Combined with the concrete engineering contract (reproduce-then-build, three-way consistency checks on implementation/claim/run-to-run, an explicit PASSED/BLOCKED/REJECTED taxonomy, harbor-format output), the pipeline is specified well enough to be reimplemented and extended rather than only admired. Even if the true end-to-end yield over the full corpus is lower than the conditional 80%, demonstrating that ~4 in 5 filtered papers can be turned into a human-accepted environment by an agent is a substantive existence proof for the paper's central thesis, and the released dataset would be a genuine community resource.

  2. Attack-and-repair is a real methodological contribution to evaluation validity, not an incidental engineering step.​ The paper's key insight is that "objective metrics" are not automatically robust: a task can be gamed by exploiting a single dimension, leaking the test set, or satisfying a metric while violating a natural-language requirement. CSForge operationalizes shortcut-resistance as an explicit optimization target — independent subagents search for loopholes, confirmed loopholes are repaired by expanding metric coverage or tightening constraints, and the task is discarded if loopholes survive five rounds. This is measured, not assumed: across 588 trajectories from matched environment variants, data/test-set exploitation drops 12→3, trivial-method shortcuts 7→2, and measurement loopholes 6→1. The surrounding evaluation hygiene reinforces it — baselines and the scoring harness are withheld from the solver, a proctor audits trajectories with veto power, and task construction enforces both coverage and improvement headroom (the latter checked against a theoretical oracle so credit is not handed out on saturated metrics). The payoff is visible downstream: at r = 0 models score up to 70.8 but collapse to ≤38.9 at r = 0.05, and the ranking reorders (GLM 5.3 goes 4th→2nd), showing the environments measure "genuinely improve the baseline" separately from "match the baseline." A benchmark that separates those two abilities — and that is hard for every frontier model tested — is exactly the discriminative instrument the CSR-agent literature currently lacks.

Critical weaknesses
  1. The primary scoring threshold is at the same magnitude as the measurement noise the authors themselves treat as disqualifying, and no significance test is applied.​ Tasks with any metric showing RSD ≥ 5% across repeated baseline runs are excluded from the benchmark subset — yet the headline threshold r = 0.05 credits exactly a 5% relative improvement. Retained tasks can therefore carry baseline noise just under the bar that decides credit. Worse, the noise floor is characterized only for the baseline; the solver's result appears to come from a single execution, with no repeated runs, confidence interval, or test of difference. This is consequential because every downstream claim — "agents score only 38.9 at r = 0.05," the subfield table, the ranking reversal between r = 0 and r = 0.05 — is a statement about differences that may be indistinguishable from run-to-run variation, in either direction (spurious credit or spurious denial). It is compounded by the authors' own admission that reproduction failures "most often arise when, under time and cost constraints, the author agent uses smaller datasets or models or simplifies experimental settings": the hidden baseline may be a weakened version of the published method, so "5% better than baseline" does not license "better than the published state of the art." Since the paper's central selling point is that its environments give reliable, verifiable measurement, a reward whose noise floor equals its credit threshold undercuts the main contribution, not merely a peripheral experiment.

  2. The validity of the environments — the paper's core claim — is certified largely by the same LLM systems being evaluated, under human review whose rubric and reliability are unreported, and the headline rates are conditional on an unmeasured filter.​ Concretely: (a) the 86.8%/80.0% figures are computed over the 280 filter-passing papers; the paper never reports end-to-end yield on the 16,962-paper corpus (≈39.3% × 80% at best) nor the filter's precision on a random sample (only recall on unsuitable papers is given, plus 9/70 false rejections, with false accepts attributed anecdotally). "Scalable construction" is thus unquantified where it matters most. (b) The 224 "valid" verdicts come from three experts with majority vote, but the review criteria are referenced, not stated, and no inter-rater agreement (κ, unanimity rate) is reported — for a paper whose main empirical claim is a pass rate, this is the load-bearing number and it is currently unreproducible. (c) On the benchmark side, the proctor is Claude Opus 5 — also the top-scoring solver — calibrated on just 10 trajectories and given veto power that sets R = 0 regardless of metrics; its false-positive rate on real benchmark trajectories is never measured, so the reported scores are jointly produced by a competitor-family judge. Similarly, the attack-and-repair evidence is self-referential: loopholes are searched by same-family subagents and detected by the same proctor, so the 12→3 reduction cannot be attributed to the protocol rather than to the detector. This is consequential because CSForge's differentiation over prior work is precisely "objective, non-LLM-judge evaluation"; if the audit layer is an unvalidated LLM judge and the validity rate rests on an undisclosed rubric, the claimed advantage over LLM-based benchmarks largely dissolves. (Related confound: environments are authored by Claude Code/Opus 4.7 and five of six solvers run on Claude Code, while GPT-5.6 Sol runs on Codex — no scaffold-matched comparison, so cross-model gaps are not interpretable.)

Questions for authors

Clarification requests (answers could change my assessment) *End-to-end yield and filter precision: what fraction of a random sample of the full 16,962-paper corpus reaches a human-validated environment, and what is the false-accept rate of the filter on a random (not expert-selected) sample? *Noise vs. threshold: distribution of per-metric RSDs (how many tasks/metrics were dropped for RSD ≥ 5%, median RSD of retained ones); were solver results averaged over repeated runs; how many of the credited metrics at r = 0.05 survive a 3-run average or significance test? *Human review protocol: the actual review criteria, and inter-rater agreement statistics (unanimity rate, Fleiss' κ) — ideally with the 19 rejected environments' failure breakdown. *Proctor: how many benchmark submissions were zeroed by the proctor, what the rankings look like with proctor vetoes removed, and whether the proctor was blinded to model identity; was any cross-family (non-Claude) proctor used as a control? *Baseline fidelity: for how many of the 49 tasks was the reproduced baseline obtained under a simplified setting (smaller model/dataset) relative to the source paper, and how does the baseline compare to the published number? *Scaffold control: is there a Claude Code vs. Codex matched comparison for the same model, to separate scaffold effects from model effects? Suggestions for future work (not affecting this recommendation)

Significance-aware or effect-size-based scoring with repeated solver runs; reporting per-task RSD alongside released environments; a head-to-head comparison of automatically synthesized vs. expert-manually-curated environments on the same papers to establish construct validity; human-vs-proctor agreement on the full trajectory set, and an ensemble/cross-family proctor; extending filtering to theory, systems, and profiling contributions (currently excluded) to test whether the pipeline generalizes beyond small-scale empirical algorithmic papers; and probing whether high scores actually correspond to methods a human reviewer would judge as genuine contributions.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

Thank you for your time and thoughtful feedback. We have uploaded a revised manuscript with changes marked in blue. The revision incorporates the following updates from the rebuttal discussion.

  • Updated the scoring protocol to use three runs per method and require statistical significance for metric credit.
  • Added proctor validation experiments and reported detailed results on detected rule violations.
  • Expanded the description of the human review criteria and results, and documented how the evaluation tasks were selected.
  • Added experimental results comparing different agent harnesses.

We have done our best to address all concerns raised. We are encouraged that Reviewer yXqf has acknowledged our responses and raised their assessment. With the discussion deadline approaching, we would greatly appreciate any feedback from the remaining reviewers and welcome any further questions.

Thank you again for your time and consideration!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.