acceptodds
Under review as a conference paper at ICLR 2027

ProgramGen: Scalable Environment Construction for Long-Horizon Agent Training

Abstract

Building an entire software repository from scratch is a challenging long-horizon task for software agents, requiring sustained exploration, implementation, execution, and debugging across many files. Training agents for such zero-to-one repository construction is essentially a data problem: realistic repositories are abundant, but scalable training tasks with reliable executable verification and high-quality long-horizon trajectories are not. We introduce ProgramGen, a repository-to-environment compiler for constructing such training data from existing software repositories. ProgramGen supports two complementary forms of reconstruction: ProgramGen-Doc retains the original tests as a hidden verifier and synthesizes an aligned specification, while ProgramGen-Exec retains the original program as an execute-only behavioral oracle and synthesizes behavioral tests from its observable behavior. Through agentic construction, ProgramGen produces 10,000 environments across five programming languages and 4,000 verifier-selected trajectories of up to 512K tokens. Training Qwen3.5-35B-A3B-Base on only 1,000 ProgramGen-Doc trajectories raises NL2Repo pass rate from 0.5% to 23.0%. The supervision also transfers beyond repository reconstruction and remains effective after post-training, improving Qwen3.6-35B-A3B from 28.8% to 31.8% on NL2Repo and from 69.6% to 75.4% on SWE-bench Verified.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent, LLM SFT, Agent, Data Pipeline

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1dCQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ProgramGen, a repository-to-environment compiler for training agents on whole-repository reconstruction. It supports two modes: ProgramGen-Doc retains original tests while synthesizing a specification, and ProgramGen-Exec retains an isolated executable reference while synthesizing behavioral tests. The authors construct 10,000 environments and collect 4,000 long-horizon trajectories. Fine-tuning produces large improvements on NL2Repo and SWE-bench, suggesting that reconstruction supervision transfers to software repair.

Critical strengths
  • This paper addresses an important data bottleneck for long-horizon software agents.
  • The dual-mode framework is a meaningful design choice, and supporting both regimes increase the range of repositories that can be converted into training environments.
  • The proposed environment construction pipeline have paid careful attentions to source isolation, leakage checking, executable validation, and trajectory auditing.
  • Experiments have shown impressive downstream gains, including transfer from repository reconstruction to SWE-bench tasks.
Critical weaknesses
  • ProgramGen-Exec substantially overlaps with the concurrent ProgramGym/Program-Env formulation. The main differentiation appears to be scale, stronger oracle isolation, and the additional Doc mode; this comparison should be made explicit.

  • Verifier validity remains uncertain. Passing inherited or synthesized tests does not establish behavioral equivalence, and testing against fixed degenerate implementations provides limited evidence against more realistic incorrect solutions.

  • Some ProgramBench comparisons use an adapted protocol, including different context lengths and reminders, making attribution to training data difficult.

  • Environment construction cost, rejection rates, failure categories, and reproducibility are not reported in sufficient detail.

Questions for authors
  1. What is the environment-level novelty over ProgramGym, particularly its coverage-guided black-box test construction?

  2. Can verifier quality be evaluated using mutation testing, independently generated implementations, or human audits?

  3. What are the computational cost and success rate of converting candidate repositories into accepted environments?

  4. Do the ProgramBench improvements remain under exactly matched prompts, context limits, and agent scaffolds?

4 comments

Comment 1 of 2 — Weakness 1 / Question 1 (ProgramGym) and Weakness 3 / Question 4 (protocol)

Response to Reviewer 1dCQ (1/2)

We thank the reviewer for the review and for recognizing the data bottleneck that the work addresses, the dual-mode design, the careful source isolation, leakage checking and trajectory auditing, and the downstream gains with transfer to SWE-bench.

This comment covers the relation to ProgramGym and the ProgramBench protocol; the next one covers verifier validity and construction cost.

W1/Q1. Relation to ProgramGym / Program-Env.

The reviewer asks what the environment-level novelty is over the concurrent ProgramGym / Program-Env, in particular its coverage-guided black-box test construction.

We thank the reviewer for raising this comparison. ProgramGym is concurrent work: it appeared on arXiv on 9 October 2026, after the ICLR submission deadline, and studies the executable-reference setting that ProgramGen-Exec also covers. What ProgramGen adds is the second form of evidence: one compiler builds environments from inherited tests (Doc) and from executable references (Exec), so the two can be compared, combined and tested for transfer. The revision nevertheless states the comparison in Section 2 and adds ProgramGym's reported result to Table 4.

The differences are three:

  • Evidence and scale. ProgramGym builds environments from executable references only. ProgramGen also builds them from inherited tests, which adds Python repositories, and has 10,000 environments against 1,648.
  • Exec environments. ProgramGen runs the reference in a separate container, where ProgramGym uses filesystem isolation. It filters trivial tests with eight degenerate programs, where ProgramGym uses one. It sets test breadth by behavior categories, where ProgramGym uses code coverage.
  • Interaction of the two forms. With both forms in one pipeline, we can study how they interact: compare them at similar token budgets, mix them, and measure transfer from one form to the other's benchmark (Section 4.4).

ProgramGym as described in its paper.

ProgramGymProgramGen
Correctness evidenceexecutable referencesinherited tests (Doc) and executable references (Exec)
Environments1,6485,000 + 5,000
LanguagesGo, C++, Rust, CPython (Doc); Rust, Go, C, C++ (Exec)
Reference accessexecute-only copy, filesystem-enforcedexecute-only stub to a separate container
Trivial-test filterone dummy programeight degenerate programs
Test breadthcoverage-guided generationat least 200 tests over 8 behavior categories
Study of the two formsone formcomparison, mixture and transfer between forms

W3/Q4. ProgramBench protocol.

The reviewer asks whether the ProgramBench improvements remain under exactly matched prompts, context limits and agent scaffolds.

We thank the reviewer for the question. In the revision, the improvements are measured under exactly matched prompts, context limits and scaffold. Within each model, the rows below differ only in the training data, so the gains can be attributed to it.

ProgramBench scores in %, from Table 4.

ModelTraining dataProgramBench AlmostProgramBench Avg
Qwen3.8-27Buntuned4.048.0
Qwen3.8-27BExec 1k6.561.6
Qwen3.8-27BDoc 1.4k + Exec 1.4k7.060.4
Qwen3.5-35B-A3B-Baseuntuned0.01.1
Qwen3.5-35B-A3B-BaseExec 1k0.020.0
  • Protocol. All our ProgramBench results in Table 4 use the same OpenHands adapter, reinforced prompt, context-budget reminder and 512K context (Appendix A.5).
  • Untuned models. They are now evaluated with the same reinforced prompt and reminder as the fine-tuned models, so the starting points benefit from the protocol as well.
  • Gains. Exec data raises the ProgramBench average of both models, and Almost on the post-trained one.

Comment 2 of 2 — Weakness 2 / Question 2 (verifier validity) and Weakness 4 / Question 3 (cost)

Response to Reviewer 1dCQ (2/2)

W2/Q2. Verifier validity.

The reviewer asks whether verifier quality can be evaluated with mutation testing, independently generated implementations or human audits, beyond the fixed degenerate implementations.

We thank the reviewer for suggesting mutation testing, which we have carried out (new Appendix A.6). On the code that they execute, the Doc and Exec verifiers detect about three quarters of the injected faults, comparable to the human-validated tests of SWE-bench Verified under the same mutation procedure.

The executed-code figures and the Rust and Go rows cover the tasks on which execution could be traced.

VerifierDetected, all mutantsMutants in executed codeDetected, executed code
Doc (inherited tests)58.6%77%75.9%
Exec (synthesized tests)31.7%36%77.1%
Exec, Rust tasks only42.1%57%73.3%
Exec, Go tasks only20.7%25%81.6%

Setup. Single-point faults, from standard mutation tools or designed by hand, were injected into the original implementation of 100 Doc and 100 Exec training environments. Each mutant was scored with the unmodified verifier. Unlike fixed degenerate programs, these are realistic incorrect solutions: the original with a single fault.

Coverage. The two modes detect faults at nearly the same rate on the code their tests run, and differ in how much of the program the tests run:

  • Origin. A Doc verifier is the developers' own test suite, admitted only above 50% coverage. An Exec verifier is written by the construction agent, which must investigate the program itself within a bounded number of rounds (Section 3.3).
  • Interface. An Exec verifier exercises the program only through its command-line interface, as the rollout agent observes it; inherited tests can call internal functions directly.
  • Size. Exec programs are about twice as large: a median of 7.7k code lines against 3.4k for Doc, over all accepted environments. The acceptance criterion is a minimum of 200 tests that does not grow with the program, and accepted verifiers hold about 290 tests whatever the program size. Coverage therefore falls as programs grow (Spearman correlation of −0.60 across 54 traced tasks), and small programs are more often rejected for yielding fewer than 200 tests.
  • Go tasks. These lower the Exec rate most. Many of them are Kubernetes operators, network services and API clients, whose logic needs a cluster or a remote service that the offline sandbox does not provide, and 6% of the traced mutants fall in vendored third-party code.

Human-written tests. They show the same pattern. Under the same procedure, the tests of SWE-bench Verified detect 79.6% of the faults placed in the functions that a task's reference patch changes, which is the code those tests were written to check, and 54.8% of the faults placed elsewhere in the same files.

W4/Q3. Construction cost, rejection rates and reproducibility.

The reviewer asks for the computational cost and success rate of converting candidates into accepted environments, with rejection rates, failure categories and reproducibility details.

We thank the reviewer for asking for these details. The revised Appendix A.4 reports the funnel, every rejection category, the cost, and the construction of each training corpus.

Complete run to date; the 5,000 + 5,000 environments of the experiments predate the submission. The token and time figures are those of an accepted environment, over all of its own attempts.

DocExec
Processed candidates98,00088,000
Buildable8,218 (8.4%)29,305 (33.3%)
Accepted5,929 (72% of buildable; 6.0% end to end)8,117 (28%; 9.2%)
Largest rejection categoryno runnable test suite (41,041)build or install failure (35,532)
Synthesis tokens of an accepted environment1.4M to 1.9M3.9M to 7.3M
Build and synthesis time of an accepted environment1.0 h2.0 h
  • Rejections. The largest Doc rejection category reflects the inventory, which is selected by a heuristic relevance score and contains many machine learning repositories without a runnable test suite.
  • Cost with rejected candidates. The token figures are then 2M to 3M and 12M to 29M, against approximately 83M per environment reported by ProgramGym.
  • Why Exec costs more. A Doc environment inherits its tests, so synthesis writes only the specification. An Exec environment requires the verifier itself to be synthesized: at least 200 behavioral tests, each executed against the reference, over up to four rounds.
  • Reproducibility. Appendix A.4 describes the construction of each training corpus, Appendix A.5 the scoring of Table 4, and Appendix A.3 the training recipes.

Thanks authors for your response. My two major concerns w.r.t the difference of ProgramGen and ProgramGym, and the ProgramGen's generalizability, have been addressed. I remain my currently positive scoring.

We thank the reviewer for the follow-up and for the positive assessment. We are glad that the comparison with ProgramGym and the generalization results resolve the two concerns.

Reviewer_U11nOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProgramGen is a repository-to-environment compiler that turns existing software repositories into source-hidden, executable reconstruction environments for long-horizon agent training. It supports two modes: ProgramGen-Doc retains the original tests as a hidden verifier and synthesizes an aligned specification, while ProgramGen-Exec retains the original program as an execute-only behavioral oracle and synthesizes behavioral tests. Agentic construction produces 10,000 environments across five programming languages and 4,000 verifier-selected trajectories of up to 512K tokens.

Critical strengths
  1. Agentic construction yields 10,000 validated environments across five languages and 4,000 verifier-selected trajectories of up to 512K tokens, each environment providing an executable verifier for validating the task and scoring teacher rollouts. Training Qwen3.5-35B-A3B-Base on 1,000 ProgramGen-Doc trajectories raises NL2Repo-Bench from 0.5% to 23.0%.
  2. This paper reports that with the reference executable available inside the agent container, the teacher attempted to read, disassemble, or instrument it in 48.8% of trajectories. ProgramGen runs the reference in a separate container and exposes it only through an execute-only stub, with no shared file system, so the rollout agent observes the reference only through these channels — making the information boundary part of the environment itself rather than a matter of agent compliance.
Critical weaknesses
  1. The headline gain uses a weak starting point, and the gain shrinks on the post-trained model.The main comparison fine-tunes Qwen3.5-35B-A3B Base, which has not undergone agent post-training and so has not seen the tool-call format, agent scaffold, or multi-turn trajectories used at evaluation; its starting scores (0.5% on NL2Repo and 6.8% on SWE-bench Verified) partly reflect this format mismatch rather than reconstruction ability, so the rise from 0.5% to 23.0% cannot be attributed to ProgramGen supervision alone. On the already post-trained Qwen3.6-35B-A3B, where the starting point is strong, the gains are much smaller (NL2Repo from 28.8 to 31.8, SWE-bench Verified from 69.6 to 75.4), and ProgramBench even decreases from 25.7 to 24.2.
  2. ProgramGen-Exec, the novel mode, shows no demonstrated benefit. On NL2Repo the balanced mixture of 500 Doc and 500 Exec trajectories reaches 23.2, close to the 23.0 from 1,000 Doc trajectories alone, and Doc alone transfers more strongly to SWE-bench Multilingual than the mixture does (54.7 against 52.7).
Questions for authors
  1. The paper presents both a repository-to-environment compiler and a corpus of verifier-selected trajectories, and it should state which is the primary contribution. If it is the construction framework, please report the construction success rate, meaning the fraction of candidates that pass the buildability check and the Build, Leakage, and verifier-quality acceptance checks, and compare it with other agentic environment-construction pipelines such as SWE-smith, R2E-Gym, or Repo2Run.
  2. The corpus is at most 4,000 trajectories and the post-trained model is fine-tuned for one epoch, so the overall training volume is limited, and the value of the reconstruction data may only become visible after scaling. Please add an SFT experiment at a substantially larger data budget to show whether the gains continue to grow as more reconstruction data is added.
  3. The final metrics include SWE-bench Multilingual, but the environments cover only Python in ProgramGen-Doc and Rust, Go, C, and C++ in ProgramGen-Exec. Zero-to-one whole-repository construction would be more meaningful if it also covered common languages such as JavaScript and TypeScript and front-end repository scenarios; please consider adding them.
  4. All experiments use 35B-A3B models, so it is unknown whether the supervision helps at other scales. Please run the same SFT on larger models and report whether the reconstruction data yields performance gains across different model scales.
8 comments

Comment 1 of 3 — Weaknesses 1 and 2

Response to Reviewer U11n (1/3)

We thank the reviewer for the detailed review and for recognizing the scale of the validated environments, the gain from 1,000 Doc trajectories on the base model, and the execute-only isolation that makes the information boundary part of the environment itself.

In brief, the revision shows three things about the data. It is of high quality: an environment is accepted only after its verifier is validated, and 500 trajectories per mode are enough for strong gains. It improves a strong post-trained model on building repositories from scratch. And it generalizes to issue fixing, which it does not contain.

Weaknesses are answered here, Q1 in the second comment, and Q2 to Q4 in the third.

W1. Starting point of the headline result, and gains on a post-trained model.

The reviewer asks how much of the headline gain comes from the supervision, given a starting model with no agent training, and points to the smaller gains on the post-trained Qwen3.6 (ProgramBench 25.7 to 24.2).

We thank the reviewer for this point. The gains are not limited to learning the agent format: in the new experiments of the revision, ProgramGen supervision also improves a strong post-trained model under one evaluation protocol, including on issue fixing, which the training data does not contain.

The post-trained experiments now use Qwen3.8-27B, a stronger starting point than the Qwen3.6-35B-A3B of the submitted version. The mixture improves on the untuned model on all three benchmarks, and Exec data alone gives the highest ProgramBench average.

Qwen3.8-27B; scores in %, from the revised Table 4 (protocol in Appendix A.5).

Training dataNL2Repo PassNL2Repo FullProgramBench AlmostProgramBench AvgSWE-bench Verified
untuned39.34.84.048.076.8
Exec 1k40.24.86.561.671.4
Doc 1.4k + Exec 1.4k45.04.87.060.478.8

For reference, ProgramGym reports 34.9 to 42.8 on ProgramBench from the same Qwen3.8-27B under its own scaffold.

The Qwen3.6 run of the submitted version was trained for a single epoch, and its two ProgramBench figures (25.7 and 24.2) were obtained under different protocols; it is retained in Appendix A.4.

W2. Contribution of ProgramGen-Exec.

The reviewer asks what ProgramGen-Exec adds, comparing the 500 + 500 mixture with 1,000 Doc trajectories on NL2Repo (23.2 vs. 23.0) and on SWE-bench Multilingual (52.7 vs. 54.7).

We thank the reviewer for raising this. ProgramGen-Exec adds three things: repositories that Doc cannot use, the strongest data for ProgramBench, and, combined with Doc data, the highest NL2Repo Pass and SWE-bench Verified. The last two are results of the new Qwen3.8-27B experiments in the revision.

  • Coverage. Exec builds on an executable reference instead of inherited tests, so it applies to repositories with a runnable executable but no usable test suite, and extends the environments from Python to Rust, Go, C and C++. Its task also differs: the agent receives only usage documentation and establishes the behavior by querying the reference (Sections 3.1 to 3.3).
  • Exec alone. ProgramBench is the benchmark that poses the Exec task. There, Exec 500 outperforms Doc 1k with fewer tokens, and is also higher on SWE-bench Verified (rows 1 and 2).
  • Combined. With the trajectory counts compared in the review, Doc 500 + Exec 500 outperforms Doc 1k on NL2Repo Pass, both ProgramBench metrics and SWE-bench Verified (rows 1 and 3). At a similar token budget, the mixture is higher than Exec alone on NL2Repo Pass and SWE-bench Verified (rows 4 and 5).

Qwen3.8-27B; scores in %, from the corpus table of Appendix A.4. Rows 1–2 and rows 4–5 are the pairs closest in tokens (within 16%).

#Training dataTokensNL2Repo PassNL2Repo FullProgramBench AlmostProgramBench AvgSWE-bench Verified
1Doc 1k145M42.62.90.555.074.2
2Exec 500126M38.63.85.561.777.0
3Doc 500 + Exec 500197M47.92.97.560.679.0
4Doc 1.4k + Exec 1.4k551M45.04.87.060.478.8
5Exec 2k503M38.94.87.063.173.6

On the base model cited in the review, the 500 + 500 mixture is higher than Doc 1k on SWE-bench Verified (69.8 against 67.4). Adding Exec 1k to Doc 1k raises SWE-bench Multilingual from 54.7 to 60.7 (Table 4).

Comment 2 of 3 — Question 1

Response to Reviewer U11n (2/3)

Q1. Primary contribution and construction success rate.

The reviewer asks which contribution is primary and requests the construction success rate, with a comparison to SWE-smith, R2E-Gym or Repo2Run.

We thank the reviewer for the question. The primary contribution is the compiler: one pipeline that turns a repository into a verifiable, source-hidden environment from either form of correctness evidence. Its acceptance design (Sections 3.2 to 3.5) is what makes the data of high quality, and the revision reports both its success rate and the strength of the accepted verifiers. The revised introduction states the contribution.

Success rate. Appendix A.4 now reports it over the whole candidate inventory, which is selected by a relevance score (Appendix A.1), not on buildability or runnable tests.

Complete run to date; the 5,000 + 5,000 environments of the experiments predate the submission. The token row counts rejected candidates; ProgramGym reports about 83M per environment.

StageDocExec
Processed candidates98,00088,000
Buildable8,218 (8.4%)29,305 (33.3%)
Accepted5,929 (72% of buildable)8,117 (28%)
End-to-end success rate6.0%9.2%
Synthesis tokens per accepted environment2M to 3M12M to 29M
  • The build stage is the main filter. The Doc buildable rate is low because 42% of its candidates have no runnable test suite.
  • Verifier acceptance is selective in Exec: most of the rejected buildable candidates (17,237) keep fewer than 200 tests after validation.
  • Accepted verifiers are strong: in a mutation audit, they detect about three quarters of the faults injected into the code they execute, comparable to the tests of SWE-bench Verified (Appendix A.6).
  • SWE-smith, R2E-Gym and Repo2Run report output sizes or setup success on pre-selected repositories, not a rate over such an inventory.

Comment 3 of 3 — Questions 2 to 4

Response to Reviewer U11n (3/3)

Q2. Training length and data budget.

The reviewer asks for an SFT experiment at a larger data budget, noting the 4,000-trajectory corpus and the single training epoch.

We thank the reviewer for this point. The post-trained model is now trained for 8 epochs instead of one, and a new scaling series varies the data from 250 to 1,400 trajectories per mode, up to 4.4B training tokens (0.8B in the submitted version).

Gains already appear at 250 trajectories per mode, and as few as 500 trajectories per mode give strong performance, which reflects the quality of the selected data: that corpus reaches the highest NL2Repo Pass, ProgramBench average and SWE-bench Verified of the series.

Scores in %, from Table 4 and Appendix A.4. First two rows: runs of the submitted version (Qwen3.5-35B-A3B-Base, Qwen3.6-35B-A3B); other rows: Qwen3.8-27B.

Doc + Exec trajectoriesEpochsTokens per epochTotal tokensNL2Repo PassNL2Repo FullProgramBench AlmostProgramBench AvgSWE-bench Verified
Qwen3.5 base: 1,000 + 1,0005372M1.9B24.82.9––72.4
Qwen3.6: 2,000 + 2,0001829M0.8B31.82.90.024.275.4
untuned–––39.34.84.048.076.8
250 + 250898M0.8B45.45.84.552.877.0
500 + 5008197M1.6B47.92.97.560.679.0
1,000 + 1,0008395M3.2B44.94.85.559.878.8
1,400 + 1,4008551M4.4B45.04.87.060.478.8

The series stops at 1,400 per mode because selection by verifier score within difficulty layers leaves about 1,400 Doc trajectories; the 4,000-trajectory corpus of the submitted version (second row) is unchanged.

Q3. Language coverage.

The reviewer suggests covering JavaScript, TypeScript and front-end repositories.

We thank the reviewer for the suggestion. The compiler is language-agnostic and already processes JavaScript and TypeScript repositories, although their pass rate is currently low. The supervision also transfers to languages that the training data does not cover.

  • Coverage. The inventory was collected for Python, Rust, Go, C and C++, so such repositories entered only incidentally: 4 of 833 candidates were accepted in Doc, and 5 of the 34 buildable ones in Exec.
  • Transfer. SWE-bench Multilingual contains no Python, yet 1,000 Python-only Doc trajectories raise the base model on it from 6.7% to 54.7%, with gains in every language group, including from 5 to 20 of its 43 JavaScript and TypeScript tasks. On Qwen3.8-27B these tasks are solved at the same rate after fine-tuning (33 of 43; Appendix A.4).

Q4. Model scale.

The reviewer asks whether the supervision helps at other model scales.

We thank the reviewer for the question. The revision adds a second model of different scale and architecture, and the supervision is effective on both.

Untuned → fine-tuned on the Doc + Exec mixture (1k per mode on the base model, 500 on Qwen3.8-27B); scores in %, from Table 4.

ModelArchitectureParameters (total / active)NL2Repo PassSWE-bench Verified
Qwen3.5-35B-A3B-Basemixture of experts35B / 3B0.5 → 24.86.8 → 72.4
Qwen3.8-27Bdense27B / 27B39.3 → 47.976.8 → 79.0

The ProgramBench average also rises on both: from 1.1 to 20.0 on the base model (Exec 1k), and from 48.0 to 60.6 on Qwen3.8-27B (mixture). On both models, the gains thus cover building repositories from scratch and issue fixing.

Reviewer_iKbuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ProgramGen converts existing software repositories into executable environments for training agents to reconstruct entire repositories from scratch. It supports two modes: Doc synthesizes specifications while retaining original tests as hidden verifiers, and Exec provides an isolated executable reference while synthesizing behavioral tests. The pipeline produces 10,000 environments across five programming languages and 4,000 trajectories selected by verifiers, with sequences of up to 512K tokens, for supervised fine-tuning of Qwen models. Experiments show improvements in repository reconstruction and transfer to software repair, including an increase in SWE-bench Verified from 69.6% to 75.4% for Qwen3.6, which had already undergone agent training.

Critical strengths
  1. Repository reconstruction is an important task. The authors provide a complete pipeline for generating tasks, executable feedback, and interaction trajectories. Doc and Exec support repositories with existing tests and executable references, respectively, broadening the available data sources.
  2. Strong transfer across tasks as expected. Training on repository reconstruction improves software repair performance on SWE-bench. Doc data, collected from Python repositories, also improves multilingual repair performance.
  3. Detailed trajectory analysis and explicit limitations. The paper examines trajectory length, tool calls, execution before the first edit, and edit–execute cycles. It acknowledges that comparisons between modes are descriptive and that trajectory selection affects the observed relationships. The discussion of ProgramBench also discloses evaluation changes and limitations.
Critical weaknesses
  1. The evaluation setup for ProgramBench is inconsistent. The main text states that ProgramBench uses a 128K-token context window; however, Appendix A.4 specifies that the ProgramGen model employs a 512K-token context window, enhanced prompting, and additional reminders regarding the context budget, whereas the Qwen3.6 baseline uses the original prompts without such reminders.
  2. Using an identical number of trajectories does not adequately control for the total volume of data. Although the authors compare different modes using the same set of 1,000 trajectories, the average lengths of the selected corpora—162K tokens for Doc and 252.6K tokens for Exec—differ significantly. Consequently, an equal number of trajectories may still correspond to varying amounts of training data and computational requirements.
Questions for authors

See weaknesses

3 comments

Comment 1 of 1 — Weaknesses 1 and 2

Response to Reviewer iKbu

We thank the reviewer for the careful reading and for recognizing the importance of repository reconstruction, the complete pipeline whose two modes broaden the data sources, the transfer to software repair, and the detailed trajectory analysis. We respond to the two weaknesses below.

W1. ProgramBench evaluation protocol.

The reviewer notes that the main text gives a 128K context for ProgramBench while the appendix describes 512K, a reinforced prompt and context reminders, with the baseline run on the original prompt.

We thank the reviewer for pointing this out. In the revision, the untuned and the fine-tuned models are evaluated under one ProgramBench protocol, and the gains hold.

The post-trained experiments now use Qwen3.8-27B, a stronger starting point than the Qwen3.6-35B-A3B of the submitted version. Exec data alone gives the highest ProgramBench average. The mixture gives the highest NL2Repo Pass and SWE-bench Verified, and improves on the untuned model on all three benchmarks.

Qwen3.8-27B; scores in %, from the revised Table 4.

Training dataNL2Repo PassNL2Repo FullProgramBench AlmostProgramBench AvgSWE-bench Verified
untuned39.34.84.048.076.8
Exec 1k40.24.86.561.671.4
Doc 1.4k + Exec 1.4k45.04.87.060.478.8

Protocol. All our ProgramBench results in Table 4 use the same OpenHands adapter, reinforced prompt, context-budget reminder and 512K context (Appendix A.5). Section 4.3 now gives 512K.

W2. Trajectory count and data volume.

The reviewer notes that equal numbers of trajectories do not equalize data volume, since Exec trajectories are longer than Doc ones (253K vs. 162K tokens on average).

We thank the reviewer for raising this. The revision reports the token count of every corpus and compares the two modes at similar token budgets. The finding is unchanged: Exec data is the stronger on ProgramBench, and Doc data on NL2Repo.

  • Pair 1. Exec 500 has fewer tokens than Doc 1k and the higher ProgramBench average; Doc 1k has the higher NL2Repo Pass.
  • Pair 2. With similar tokens again, the mixture has the higher NL2Repo Pass and SWE-bench Verified, and Exec alone the higher ProgramBench average.

The differences therefore come from the type of data, not from its volume.

Scores in %, from Table 4 and the corpus table of Appendix A.4. The two pairs are the ProgramGen corpora closest in tokens (within 16%).

ModelTraining dataNL2Repo PassNL2Repo FullProgramBench AlmostProgramBench AvgSWE-bench Verified
Pair 1: similar tokens
Qwen3.8-27B + Doc 1k145M tokens42.62.90.555.074.2
Qwen3.8-27B + Exec 500126M tokens38.63.85.561.777.0
Pair 2: similar tokens
Qwen3.8-27B + Doc 1.4k + Exec 1.4k551M tokens45.04.87.060.478.8
Qwen3.8-27B + Exec 2k503M tokens38.94.87.063.173.6
Other long-horizon corpora, for reference
MindForge-27B†172M tokens (973 trajectories)23.5–2.549.573.8
ProgramGym-27B†64,505 segments from rollouts on 1,648 tasks (tokens not reported)––1.542.8–
DeNovoSWE-35B-A3B1.1B tokens (11,463 trajectories)19.11.00.522.164.2
  • Our corpora range from 126M to 551M tokens: the smaller pair is comparable to MindForge, and all are below DeNovoSWE.
  • † As reported in the original papers under their own scaffolds: MindForge fine-tunes Qwen3.6-27B, and ProgramGym fine-tunes Qwen3.8-27B under Claude Code.
  • DeNovoSWE-35B-A3B fine-tunes the post-trained Qwen3.5-35B-A3B. We evaluated the released model under our scaffold and protocol, and its token count is computed on the released trajectories.

Thank you for your response. It addresses my concern.

We thank the reviewer for the feedback, and we are glad that the response addresses the concern.

Comments on the paper

1 comment

To all reviewers and the Area Chair

We thank all reviewers for their constructive feedback. This comment gives an overview of the revision; each reviewer's points are answered one by one in a separate thread.

Background. ProgramGen builds training tasks in which an agent must rebuild a whole software repository whose source code is hidden, and an automatic verifier scores the result. Its two modes differ in what the agent is given:

  • ProgramGen-Doc: a written specification. The repository's own tests are the hidden verifier.
  • ProgramGen-Exec: the original program, which the agent can run but not read. The verifier is a synthesized test suite.

Three main results:

  • The data is of high quality. 500 trajectories per mode raise the post-trained Qwen3.8-27B from 39.3 to 47.9 on NL2Repo (from a specification) and from 48.0 to 60.6 on ProgramBench (from an executable).
  • It transfers and generalizes. The data contains only reconstruction tasks, yet issue fixing improves on both models (SWE-bench Verified: 6.8 to 72.4 on the base model) and, with Python-only data, in every language of SWE-bench Multilingual.
  • The verifiers are reliable. A mutation audit finds them comparable to the human-validated tests of SWE-bench Verified.

Revision. The reviews share three concerns. To address them, the revision adds nine fine-tuning runs on a stronger post-trained model, each evaluated on four benchmarks, a construction funnel over 186,000 repositories, and a mutation audit of 200 verifiers. For each concern, we state what is added and what it shows.

1. Do the gains come from the supervision, and do they hold beyond the base model? (iKbu W1; U11n W1, Q4; 1dCQ W3)

  • Concern. Whether the ProgramBench comparison uses one protocol, and whether the gains go beyond teaching a base model the agent format.
  • Added. Experiments on Qwen3.8-27B, a model already post-trained for agentic coding: 8 epochs per corpus at a 512K-token context, with the untuned and fine-tuned models under one evaluation protocol.
  • Result. The supervision improves both the base model Qwen3.5-35B-A3B-Base and the post-trained Qwen3.8-27B, and it transfers from whole-repository reconstruction to issue-level repair.

Before → after fine-tuning on the Doc + Exec mixture (1k per mode on the base model, 1.4k on Qwen3.8-27B); scores in %, from the revised Table 4.

ModelNL2Repo PassProgramBench AvgSWE-bench Verified
Qwen3.5-35B-A3B-Base (no agent training)0.5 → 24.8–6.8 → 72.4
Qwen3.8-27B (post-trained)39.3 → 45.048.0 → 60.476.8 → 78.8

Each mode alone also transfers to repair: on the base model, 1,000 Doc trajectories reach 67.4 on SWE-bench Verified, and 1,000 Exec trajectories 66.8.

2. How much data is needed, and what does each mode contribute? (iKbu W2; U11n W2, Q2)

  • Concern. Whether the corpora are compared at similar token budgets, what Exec adds to Doc, and how the results change with more data.
  • Added. Token counts for every corpus, comparisons of the two modes at similar token budgets, and a scaling series from 250 to 1,400 trajectories per mode.
  • Result. The supervision is data-efficient, and the two modes are complementary. As few as 500 trajectories per mode give strong performance: the 500 + 500 mixture reaches the highest NL2Repo Pass (47.9) and SWE-bench Verified (79.0) of the nine corpora. Exec data alone gives the highest ProgramBench average (63.1).

3. Are the environments and their verifiers reliable, and what does construction cost? (U11n Q1; 1dCQ W1, W2, W4)

  • Concern. The construction success rate and cost, the strength of the synthesized tests, and the difference from the concurrent ProgramGym.
  • Added. The construction funnel with rejection categories and cost, a mutation audit of 100 Doc and 100 Exec verifiers with the same procedure applied to SWE-bench Verified, and a comparison with ProgramGym in Table 4 and Section 2.
  • Result. On the code that they execute, the Doc and Exec verifiers detect 75.9% and 77.1% of the injected faults, against 79.6% for the human-validated tests of SWE-bench Verified in the functions their patches change. End to end, 6.0% of the Doc candidates and 9.2% of the Exec candidates become accepted environments. Unlike ProgramGym, ProgramGen compiles both forms of correctness evidence in one pipeline, which allows the two to be compared and mixed.

Revised PDF. The revised PDF, now uploaded, gives the full details:

  • Sections 1 to 3: contribution and method, revised around the two modes.
  • Section 4, Table 4 and Figures 3 to 5: the new experiments.
  • Appendix A.3 to A.6: training, corpora and funnel, evaluation protocol, and mutation audit.

Blue marks new results and requested additions, not rewording; in the appendix, the headings and captions of new parts. Numbering in our responses refers to this version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.