acceptodds
Desk rejected at ICLR 2027

Zero2Repo: Can Coding Agents Build Repositories from Scratch?

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks. We introduce \bench, a benchmark in which an agent receives a product requirements document, an interface contract, and an empty workspace, and must deliver a complete repository in the project's native ecosystem. Tasks are produced by a language-agnostic authoring pipeline that converts real, version-pinned open-source projects into behavioral specifications, reproducible environments, and hidden acceptance tests. Each task is validated by execution: a reference implementation derived from the upstream project must pass, and adversarial validation must show that the tests reject incorrect implementations. Evaluation runs production coding agents in isolated containers, withholds the acceptance tests until an explicit submission, and assigns a binary reward only when every test passes, with no LLM judge. The pipeline and harness make no language-specific assumptions and apply to mainstream programming ecosystems; the current release contains Python, TypeScript, Go, and C++ tasks. Even on 11 tasks drawn from repositories that frontier models have very likely seen during training, the strongest agent solves only 10, and every failing submission passes 89–99% of the hidden tests; for the two strongest agents, 87–100% of failed tests trace to a single omission or a low-frequency rule stated in the specification rather than to a missing subsystem, so each failure is a concrete target for improvement.

Primary area:
datasets and benchmarks
Keywords:
coding agents, large language models, code generation, software engineering, datasets and benchmarks, evaluation, repository construction, specification-to-code, hidden tests

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7fWfOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Zero2Repo, a benchmark for constructing complete software repositories from an empty workspace given a product requirements document and an Interface Contract. Tasks are derived from pinned open-source projects through a model-assisted authoring pipeline and are validated using an adapted reference implementation, adversarial negative controls, and author inspection. During evaluation, acceptance tests remain hidden until explicit submission and are executed in a fresh offline judge container; a task receives binary credit only if all tests pass. The current release contains 11 tasks across Python, TypeScript, C++, and Go. The paper evaluates three model-harness configurations once per task, obtaining pass rates of 10/11, 8/11, and 7/11, and analyzes failed hidden tests to argue that stronger configurations increasingly fail on last-mile specification details rather than missing subsystems.

Critical strengths
  1. The evaluation mechanics are carefully designed and consequential. The separation between solve and judge phases, sealed acceptance tests, fresh offline judge container, explicit submission protocol, and deterministic all-or-nothing scoring avoid the ambiguity of LLM judging and many common leakage channels (Sections 3.1-3.3; Figures 1-3). This is a meaningful step toward evaluating whether agents can deliver usable repositories rather than isolated code fragments.

  2. The benchmark offers unusually concrete diagnostic evidence. The paper reports per-task outcomes, time, cost, hidden-test counts, and root-cause narratives, and it candidly acknowledges likely training exposure (Tables 6-8; Appendix H). The combination of reference execution and adversarial negative controls is a useful foundation; if the specification-validity issue is repaired and the suite is expanded, these assets could support valuable evaluation and training research.

Critical weaknesses
  1. The benchmark’s specification-validity guarantee is contradicted by its own evidence. Appendix E and Section 5 claim that every tested behavior is publicly specified. However, Appendix H acknowledges that requirements involving default construction in task 008 and GIT_SSH in task 009 were not included in the PRD or Interface Contract. In task 008, the unstated requirement causes all 28 Claude failures before the matching logic is exercised. These failures constitute 61% of Claude’s failed tests and materially drive the abstract’s 87% claim; excluding them reduces the corresponding proportion to 67%. The affected specifications and tests therefore require correction, independent traceability review, and rerunning rather than merely revised wording.

  2. The empirical conclusions exceed the available evidence. The benchmark contains only 11 selected tasks, eight in Python, with one run per configuration. Model and harness are fully confounded, so the results compare complete configurations rather than model capability. Moreover, Table 3 counts correlated test failures separately even when one root cause produces 28 failures, making the proposed change in failure structure only a preliminary observation. Finally, because the tasks come from familiar repositories—and one submission substantially overlaps with upstream code—the evaluation may partly measure source memorization. A matched cohort of recent or unseen projects is needed to isolate specification-following ability.

As a minor presentation issue, Section 2 contains unresolved “(?)” citations in the discussions of FrontierSWE and SWE-Together; these references should be completed.

Questions for authors
  1. Can the authors correct tasks 008 and 009, verify that every hidden test maps to an explicit public requirement, and report how the corrected results affect Tables 2–3 and the 87–100% claim?
  2. Can the authors provide repeated runs to quantify variance? Otherwise, please restrict the conclusions to the evaluated model–harness configurations rather than model capability.
  3. Can the authors evaluate on recent or plausibly unseen repositories and provide sufficient details about the task-authoring pipeline to support its reproducibility and scalability?
Reviewer_mPddOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Zero2Repo, a benchmark that tests whether coding agents can build complete, runnable repositories from scratch. Agents start with an empty working directory and work from product requirements documents and interface contracts. The tasks are drawn from specific versions of real open-source projects. Acceptance tests are kept hidden during development, and submissions are evaluated in isolated containers without network access. The dataset currently includes 11 tasks across Python, TypeScript, Go, and C++. In the experiments, the three latest model–harness combinations completed 10, 8, and 7 of the 11 tasks, respectively.

Critical strengths
  1. The evaluation target is practical. Unlike generating a single function or patching an existing repo, it has the agent handle code organization, interface implementation, project setup, and delivering build artifacts, a more direct test of going from requirements to a complete project.

  2. The experiments include some of the latest frontier models, giving a useful picture of how well current models handle these tasks.

Critical weaknesses
  1. The experimental scale limits statistical reliability. The dataset has only 11 tasks (eight in Python, one each in the other languages). A single task is worth about 9.1 percentage points, so the current results cannot support robust model rankings or language-level conclusions.

  2. The authors say they’ll add more tasks in v1, so doesn’t that suggest the paper isn’t quite ready yet?

Questions for authors
  1. Building a repo from scratch is important, but how do you ensure the problem is solvable? And how do you ensure the interfaces designed in the PRD match the functional interfaces that the unit tests are meant to test?

  2. When building a repo from scratch here, roughly how much code is involved? How many steps does the agent run? Judging from the runtime, it doesn't seem to be a task with a large amount of code. For example, GPT-6 Astra's average runtime is only 14.4 minutes.

  3. Is the difficulty also a bit on the low side? Current SOTA models already have fairly high solve rates.

2 comments
Reviewer_mPddOpenReview ↗

Official Comments by Reviewer mPdd

Thank you for the detailed response. It has addressed most of my concerns, including those regarding the scale and difficulty of the dataset, and I sincerely appreciate your efforts. However, I still feel that there are quite a few similar works to Zero2Repo, such as NL2Repo and other approaches listed in the Related Work section, which also aim to build a repository from scratch given a task prompt. Could the authors further clarify and highlight the fundamental differences?

We sincerely thank the reviewer for the follow-up and are glad our previous response addressed the concerns about scale and difficulty. We agree that several recent benchmarks also build repositories from scratch, and we appreciate the chance to clarify how Zero2Repo differs. Zero2Repo is designed to be close to real-world vibe coding: the developer gives only requirements and an external interface, and the agent must deliver a complete, usable repository from an empty workspace.

  1. NL2Repo-Bench, the closest work, differs in four ways.
  • Language coverage. NL2Repo-Bench supports only Python, because its scoring restores the reference packaging and upstream tests and then runs pytest. Zero2Repo works with any language. It currently covers nine languages, and each repository must build in its native ecosystem.
  • Evaluation target. NL2Repo-Bench scores with the upstream repository's own tests. These tests check conformance to the original implementation (module paths, docstrings), so a correct but differently structured submission can fail. Zero2Repo instead follows a "different paths, same destination" principle. Any implementation that delivers the specified behaviour passes, whatever its architecture, module layout or internal design. Our tests check only observable outcomes: return values, exit codes, error handling and produced files.
  • Specification completeness. NL2Repo-Bench's descriptions often omit what the tests need. For example, the funcy tests call 127 functions, and 76 of them are absent from the description. Its own ablation shows the gap: revealing the tests raises the pass rate from 40.2% to 59.4%. In Zero2Repo, everything the tests depend on is specified, and each task is validated against a reference implementation before release.
  • Scoring environment. Some NL2Repo-Bench images preinstall the reference implementation, so an empty pytz submission passes 232 of 235 tests. Zero2Repo isolates the agent, blocks reuse of the upstream implementation, keeps the tests sealed until submission, and gives credit only when every test passes.
  1. RepoZero measures a different capability: cross-language output replication. The agent receives a driver file's source and four white-box test cases. Only the driver's standard output is scored, by exact string match. Errors, exit codes, written files and performance are not evaluated, and a correct port that prints True instead of true, or 1.0 instead of 1, is marked wrong. Zero2Repo gives only a PRD and an interface contract, and it judges behaviour, not exact output text.

  2. Other related benchmarks. Commit0 provides library skeletons with interactive test feedback and covers only Python. DevBench scores separate development stages, partly with an LLM judge. RepoGenesis targets only Python and Java web microservices. None of them requires a complete, multi-language, native-ecosystem repository built from an empty workspace with sealed tests and no LLM judge.

  3. Construction cost and sustainability. Most existing benchmarks rely heavily on manual work for each task. NL2Repo-Bench's 104 specifications are written by human annotators, and DevBench, Commit0 and FrontierSWE are hand-curated. This makes these benchmarks costly to extend and leaves them exposed to saturation and contamination over time.

In Zero2Repo, models draft the specifications, environments and tests, and executable checks decide what is released:

  • the environment must rebuild from its recipe;
  • a reference implementation must pass every test;
  • deliberately defective implementations must receive zero reward.

Humans only review the results. This efficiency allows regular versioned releases that draw on recently published repositories (twelve current tasks come from repositories created after December 2025), so the benchmark keeps its difficulty and its ability to separate systems.

Summary. To our knowledge, Zero2Repo is the only benchmark that combines all of the following:

  • a functional specification sufficient to solve each task without exposing the tests;
  • complete native-ecosystem repositories across multiple languages;
  • implementation-agnostic, behaviour-based acceptance ("different paths, same destination"), with sealed tests and all-or-nothing credit;
  • an isolated scoring environment that blocks reuse of the reference implementation;
  • a low-cost, largely automated construction pipeline that supports regular updates.

We are therefore confident that Zero2Repo is one of the more realistic and solid benchmarks for from-scratch vibe coding. At the same time, we see it as complementary to these valuable prior works. In the revision, we will add this discussion to the Related Work section and extend Table 1 with columns for the evaluation target, test visibility and construction effort. We thank the reviewer again for this constructive suggestion.

Reviewer_SvQCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Zero2Repo, a benchmark for constructing software repositories from an empty workspace. Agents receive a product requirements document and an interface contract, implement the specified functionality in the project’s native programming ecosystem, and submit a workspace for evaluation against hidden acceptance tests. A task receives credit only when every test passes. Tasks are derived from existing open-source projects through a model-assisted pipeline that produces specifications, environments, and acceptance tests. The current evaluation contains 11 tasks across four languages and three model-harness configurations.

Critical strengths
  1. The evaluation protocol is careful and well documented
  2. The detailed failure reporting is useful.
Critical weaknesses

W1: The biggest issue with the paper is the scale of the benchmark. It contains only 11 repositories with 8 of them being Python packages. Compared to similar benchmarks which have up to 104 tasks [1], Zero2Repo is substantially smaller. The authors promise to release more repositories in the future, which I believe will strengthen the benchmark. As of now, the benchmark remains a proof-of-concept, not a finished research product.

Moreover, each model-harness pair is evaluated just once. The ordering of results remains highly volatile due to these factors. The models differ from each other only in 1-2 repositories, which could very well be noise rather than a pattern. Any conclusion that the authors drew from these results remain questionable.

W2: The other primary concern is that hidden tests enforce unstated requirements, conflating agent failures with defects in the benchmark specification. Appendix H identifies two such cases. In the C++ URL parser task (008), the hidden test driver assumes that the compiled pattern result can be default-constructed. Claude’s implementation does not support this construction form, causing all 28 failed tests on the task to fail compilation before the relevant matching behavior is exercised. The public contract specifies the result type but does not establish this additional requirement. In the Go large-file client task (009), some tests require honoring the GIT_SSH environment variable, although neither the PRD nor the Interface Contract mentions it.

These specification defects affect the reported results under all-or-nothing scoring. The constructor issue alone accounts for one failed task and approximately 61% of Claude’s failed tests. The authors acknowledge these gaps and plan to clarify them in a future release, but the affected specifications or tests should be corrected and the evaluations rerun for the current paper. These examples also contradict the claim in Section 5 that every failure was detectable from the public specification alone.

W3: The contribution over existing repository-generation benchmarks needs stronger justification. NL2Repo-Bench already evaluates repository generation from an empty workspace using natural-language requirements and hidden executable tests [1]. RepoZero combines automated task construction with repository generation across languages and hidden execution-based evaluation [2]. Zero2Repo introduces useful distinctions, including native-ecosystem deliverables and its acceptance-test validation procedure, but the current experiments do not sufficiently establish the significance of these differences.

[1] Ding, Jingzhe, et al. "Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents." arXiv preprint arXiv:2512.12730 (2025).

[2] Zhang, Zhaoxi, et al. "RepoZero: Can LLMs Generate a Code Repository from Scratch?." arXiv preprint arXiv:2605.07122 (2026).

Questions for authors

See weaknesses above

Comments on the paper

1 comment
Area_Chair_2YsGOpenReview ↗

Reminder from AC

Dear Reviewer,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.