- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ProgramGen, a repository-to-environment compiler for training agents on whole-repository reconstruction. It supports two modes: ProgramGen-Doc retains original tests while synthesizing a specification, and ProgramGen-Exec retains an isolated executable reference while synthesizing behavioral tests. The authors construct 10,000 environments and collect 4,000 long-horizon trajectories. Fine-tuning produces large improvements on NL2Repo and SWE-bench, suggesting that reconstruction supervision transfers to software repair.
Critical strengths
- This paper addresses an important data bottleneck for long-horizon software agents.
- The dual-mode framework is a meaningful design choice, and supporting both regimes increase the range of repositories that can be converted into training environments.
- The proposed environment construction pipeline have paid careful attentions to source isolation, leakage checking, executable validation, and trajectory auditing.
- Experiments have shown impressive downstream gains, including transfer from repository reconstruction to SWE-bench tasks.
Critical weaknesses
-
ProgramGen-Exec substantially overlaps with the concurrent ProgramGym/Program-Env formulation. The main differentiation appears to be scale, stronger oracle isolation, and the additional Doc mode; this comparison should be made explicit.
-
Verifier validity remains uncertain. Passing inherited or synthesized tests does not establish behavioral equivalence, and testing against fixed degenerate implementations provides limited evidence against more realistic incorrect solutions.
-
Some ProgramBench comparisons use an adapted protocol, including different context lengths and reminders, making attribution to training data difficult.
-
Environment construction cost, rejection rates, failure categories, and reproducibility are not reported in sufficient detail.
Questions for authors
-
What is the environment-level novelty over ProgramGym, particularly its coverage-guided black-box test construction?
-
Can verifier quality be evaluated using mutation testing, independently generated implementations, or human audits?
-
What are the computational cost and success rate of converting candidate repositories into accepted environments?
-
Do the ProgramBench improvements remain under exactly matched prompts, context limits, and agent scaffolds?
4 comments
Comment 2 of 2 — Weakness 2 / Question 2 (verifier validity) and Weakness 4 / Question 3 (cost)
Response to Reviewer 1dCQ (2/2)
W2/Q2. Verifier validity.
The reviewer asks whether verifier quality can be evaluated with mutation testing, independently generated implementations or human audits, beyond the fixed degenerate implementations.
We thank the reviewer for suggesting mutation testing, which we have carried out (new Appendix A.6). On the code that they execute, the Doc and Exec verifiers detect about three quarters of the injected faults, comparable to the human-validated tests of SWE-bench Verified under the same mutation procedure.
The executed-code figures and the Rust and Go rows cover the tasks on which execution could be traced.
| Verifier | Detected, all mutants | Mutants in executed code | Detected, executed code |
|---|---|---|---|
| Doc (inherited tests) | 58.6% | 77% | 75.9% |
| Exec (synthesized tests) | 31.7% | 36% | 77.1% |
| Exec, Rust tasks only | 42.1% | 57% | 73.3% |
| Exec, Go tasks only | 20.7% | 25% | 81.6% |
Setup. Single-point faults, from standard mutation tools or designed by hand, were injected into the original implementation of 100 Doc and 100 Exec training environments. Each mutant was scored with the unmodified verifier. Unlike fixed degenerate programs, these are realistic incorrect solutions: the original with a single fault.
Coverage. The two modes detect faults at nearly the same rate on the code their tests run, and differ in how much of the program the tests run:
- Origin. A Doc verifier is the developers' own test suite, admitted only above 50% coverage. An Exec verifier is written by the construction agent, which must investigate the program itself within a bounded number of rounds (Section 3.3).
- Interface. An Exec verifier exercises the program only through its command-line interface, as the rollout agent observes it; inherited tests can call internal functions directly.
- Size. Exec programs are about twice as large: a median of 7.7k code lines against 3.4k for Doc, over all accepted environments. The acceptance criterion is a minimum of 200 tests that does not grow with the program, and accepted verifiers hold about 290 tests whatever the program size. Coverage therefore falls as programs grow (Spearman correlation of −0.60 across 54 traced tasks), and small programs are more often rejected for yielding fewer than 200 tests.
- Go tasks. These lower the Exec rate most. Many of them are Kubernetes operators, network services and API clients, whose logic needs a cluster or a remote service that the offline sandbox does not provide, and 6% of the traced mutants fall in vendored third-party code.
Human-written tests. They show the same pattern. Under the same procedure, the tests of SWE-bench Verified detect 79.6% of the faults placed in the functions that a task's reference patch changes, which is the code those tests were written to check, and 54.8% of the faults placed elsewhere in the same files.
W4/Q3. Construction cost, rejection rates and reproducibility.
The reviewer asks for the computational cost and success rate of converting candidates into accepted environments, with rejection rates, failure categories and reproducibility details.
We thank the reviewer for asking for these details. The revised Appendix A.4 reports the funnel, every rejection category, the cost, and the construction of each training corpus.
Complete run to date; the 5,000 + 5,000 environments of the experiments predate the submission. The token and time figures are those of an accepted environment, over all of its own attempts.
| Doc | Exec | |
|---|---|---|
| Processed candidates | 98,000 | 88,000 |
| Buildable | 8,218 (8.4%) | 29,305 (33.3%) |
| Accepted | 5,929 (72% of buildable; 6.0% end to end) | 8,117 (28%; 9.2%) |
| Largest rejection category | no runnable test suite (41,041) | build or install failure (35,532) |
| Synthesis tokens of an accepted environment | 1.4M to 1.9M | 3.9M to 7.3M |
| Build and synthesis time of an accepted environment | 1.0 h | 2.0 h |
- Rejections. The largest Doc rejection category reflects the inventory, which is selected by a heuristic relevance score and contains many machine learning repositories without a runnable test suite.
- Cost with rejected candidates. The token figures are then 2M to 3M and 12M to 29M, against approximately 83M per environment reported by ProgramGym.
- Why Exec costs more. A Doc environment inherits its tests, so synthesis writes only the specification. An Exec environment requires the verifier itself to be synthesized: at least 200 behavioral tests, each executed against the reference, over up to four rounds.
- Reproducibility. Appendix A.4 describes the construction of each training corpus, Appendix A.5 the scoring of Table 4, and Appendix A.3 the training recipes.
Thanks authors for your response. My two major concerns w.r.t the difference of ProgramGen and ProgramGym, and the ProgramGen's generalizability, have been addressed. I remain my currently positive scoring.
We thank the reviewer for the follow-up and for the positive assessment. We are glad that the comparison with ProgramGym and the generalization results resolve the two concerns.
Comment 1 of 2 — Weakness 1 / Question 1 (ProgramGym) and Weakness 3 / Question 4 (protocol)
Response to Reviewer 1dCQ (1/2)
We thank the reviewer for the review and for recognizing the data bottleneck that the work addresses, the dual-mode design, the careful source isolation, leakage checking and trajectory auditing, and the downstream gains with transfer to SWE-bench.
This comment covers the relation to ProgramGym and the ProgramBench protocol; the next one covers verifier validity and construction cost.
W1/Q1. Relation to ProgramGym / Program-Env.
We thank the reviewer for raising this comparison. ProgramGym is concurrent work: it appeared on arXiv on 9 October 2026, after the ICLR submission deadline, and studies the executable-reference setting that ProgramGen-Exec also covers. What ProgramGen adds is the second form of evidence: one compiler builds environments from inherited tests (Doc) and from executable references (Exec), so the two can be compared, combined and tested for transfer. The revision nevertheless states the comparison in Section 2 and adds ProgramGym's reported result to Table 4.
The differences are three:
ProgramGym as described in its paper.
W3/Q4. ProgramBench protocol.
We thank the reviewer for the question. In the revision, the improvements are measured under exactly matched prompts, context limits and scaffold. Within each model, the rows below differ only in the training data, so the gains can be attributed to it.
ProgramBench scores in %, from Table 4.