- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces Zero2Repo, a benchmark for constructing complete software repositories from an empty workspace given a product requirements document and an Interface Contract. Tasks are derived from pinned open-source projects through a model-assisted authoring pipeline and are validated using an adapted reference implementation, adversarial negative controls, and author inspection. During evaluation, acceptance tests remain hidden until explicit submission and are executed in a fresh offline judge container; a task receives binary credit only if all tests pass. The current release contains 11 tasks across Python, TypeScript, C++, and Go. The paper evaluates three model-harness configurations once per task, obtaining pass rates of 10/11, 8/11, and 7/11, and analyzes failed hidden tests to argue that stronger configurations increasingly fail on last-mile specification details rather than missing subsystems.
Critical strengths
-
The evaluation mechanics are carefully designed and consequential. The separation between solve and judge phases, sealed acceptance tests, fresh offline judge container, explicit submission protocol, and deterministic all-or-nothing scoring avoid the ambiguity of LLM judging and many common leakage channels (Sections 3.1-3.3; Figures 1-3). This is a meaningful step toward evaluating whether agents can deliver usable repositories rather than isolated code fragments.
-
The benchmark offers unusually concrete diagnostic evidence. The paper reports per-task outcomes, time, cost, hidden-test counts, and root-cause narratives, and it candidly acknowledges likely training exposure (Tables 6-8; Appendix H). The combination of reference execution and adversarial negative controls is a useful foundation; if the specification-validity issue is repaired and the suite is expanded, these assets could support valuable evaluation and training research.
Critical weaknesses
-
The benchmark’s specification-validity guarantee is contradicted by its own evidence. Appendix E and Section 5 claim that every tested behavior is publicly specified. However, Appendix H acknowledges that requirements involving default construction in task 008 and
GIT_SSHin task 009 were not included in the PRD or Interface Contract. In task 008, the unstated requirement causes all 28 Claude failures before the matching logic is exercised. These failures constitute 61% of Claude’s failed tests and materially drive the abstract’s 87% claim; excluding them reduces the corresponding proportion to 67%. The affected specifications and tests therefore require correction, independent traceability review, and rerunning rather than merely revised wording. -
The empirical conclusions exceed the available evidence. The benchmark contains only 11 selected tasks, eight in Python, with one run per configuration. Model and harness are fully confounded, so the results compare complete configurations rather than model capability. Moreover, Table 3 counts correlated test failures separately even when one root cause produces 28 failures, making the proposed change in failure structure only a preliminary observation. Finally, because the tasks come from familiar repositories—and one submission substantially overlaps with upstream code—the evaluation may partly measure source memorization. A matched cohort of recent or unseen projects is needed to isolate specification-following ability.
As a minor presentation issue, Section 2 contains unresolved “(?)” citations in the discussions of FrontierSWE and SWE-Together; these references should be completed.
Questions for authors
- Can the authors correct tasks 008 and 009, verify that every hidden test maps to an explicit public requirement, and report how the corrected results affect Tables 2–3 and the 87–100% claim?
- Can the authors provide repeated runs to quantify variance? Otherwise, please restrict the conclusions to the evaluated model–harness configurations rather than model capability.
- Can the authors evaluate on recent or plausibly unseen repositories and provide sufficient details about the task-authoring pipeline to support its reproducibility and scalability?
Official Comments by Reviewer mPdd
Thank you for the detailed response. It has addressed most of my concerns, including those regarding the scale and difficulty of the dataset, and I sincerely appreciate your efforts. However, I still feel that there are quite a few similar works to Zero2Repo, such as NL2Repo and other approaches listed in the Related Work section, which also aim to build a repository from scratch given a task prompt. Could the authors further clarify and highlight the fundamental differences?