- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a repository-level code generation approach based on agent-native reusable code primitives. The key idea is to construct a library of reusable code primitives that encapsulate component interface contracts, dependency closures, validation tests, provenance information, and a resident LLM. Given a repository-level generation task, the proposed LEGO framework retrieves and activates relevant primitives, adapts and integrates their implementations into the target repository, and iteratively revises the generated code based on execution and validation feedback.
Overall, the paper addresses an important problem and reports encouraging empirical results. However, I have several concerns regarding the construction and coverage of the primitive library, the technical novelty of the LEGO pipeline, the motivation for the new benchmark, and the strength of the experimental baselines.
Critical strengths
- Repository-level code generation is a timely and practically important problem. Leveraging module-level reusable code primitives is an intuitive and potentially effective approach for improving the reliability of repository generation.
Critical weaknesses
-
Insufficient discussion of the construction and coverage of the CodeFace primitive library. The paper constructs a CodeFace library containing 1,424 code primitives, but the process for constructing this library is not sufficiently described or analyzed. In particular, it is unclear what the fundamental challenges are in building such a library, how primitives are identified and selected, and how their quality and generality are ensured. More importantly, it is unclear whether the current library provides broad coverage of real-world public repositories or is particularly aligned with the proposed benchmark. I would like to see a controlled study that varies the size and coverage of the primitive library. Such an experiment would help quantify how much of the improvement comes from the LEGO methodology versus strong coverage provided by CodeFace.
-
The fundamental technical novelty of the LEGO pipeline is unclear. The overall pipeline consists of decomposition, retrieval, adaptation, repository editing, execution feedback, verification, and diagnosis. Individually, these stages appear similar to components commonly found in existing coding-agent systems. The paper should more clearly articulate what fundamental challenges arise specifically from integrating reusable code primitives into repository generation and which algorithmic or system-design contributions of LEGO address these challenges. As currently presented, it is difficult to distinguish the core technical novelty from a well-engineered integration of existing agentic coding techniques.
-
The motivation and novelty of the new benchmark need to be better justified. There are already several benchmarks targeting repository-level or zero-to-one repository generation, including RepoCraft, RepoZero, and RepoGenesis. The paper should more clearly explain which important capabilities or challenges are not captured by these existing benchmarks and why a new benchmark is necessary. Beyond introducing additional tasks, it would be useful to articulate what new scientific insights or evaluation dimensions the proposed benchmark enables.
-
The agentic baselines appear insufficiently competitive. LEGO is fundamentally an agentic solution, but the evaluation does not appear to include sufficiently recent state-of-the-art agentic systems for zero-to-one repository generation. In particular, the two agentic baselines included in the paper are from 2024. This makes it difficult to determine whether the reported improvements come from the proposed reusable-primitive methodology or simply from using a stronger contemporary agent pipeline. Comparisons against more recent strong agentic baselines would be important.
Questions for authors
-
How exactly are the 1,424 CodeFace primitives constructed and selected? What are the main technical challenges in building the library, and how do you ensure that its coverage generalizes beyond the evaluation benchmark?
-
How sensitive is LEGO's performance to the size and coverage of CodeFace? Could you provide an ablation where, for example, 1%, 10%, 50%, and 90% of the relevant library is available? This would help understand the contribution of primitive coverage.
-
What happens when the required functionality is missing from CodeFace or when retrieval returns an irrelevant primitive? How gracefully does LEGO fall back to generation or recover from incorrect retrieval?
-
What fundamental challenges arise specifically from integrating reusable code primitives into repository generation?
-
What important capability does the proposed benchmark evaluate that RepoCraft, RepoZero, RepoGenesis, or other existing repository-generation benchmarks do not? Could you provide a systematic comparison of task characteristics and evaluation dimensions?
-
How does LEGO compare with more recent state-of-the-art agentic repository-generation approaches under the same base model?
6 comments
W2/Q4. The fundamental technical novelty of the LEGO pipeline is unclear. The overall pipeline consists of decomposition, retrieval, adaptation, repository editing, execution feedback, verification, and diagnosis. Individually, these stages appear similar to components commonly found in existing coding-agent systems. The paper should more clearly articulate what fundamental challenges arise specifically from integrating reusable code primitives into repository generation and which algorithmic or system-design contributions of LEGO address these challenges. As currently presented, it is difficult to distinguish the core technical novelty from a well-engineered integration of existing agentic coding techniques. What fundamental challenges arise specifically from integrating reusable code primitives into repository generation?
R2/A4. We would like to clarify that the novelty of LEGO lies in a form of reuse in which the reused code becomes part of the delivered repository. The novelty also lies in one design choice for each challenge that this form of reuse raises. LEGO is a repository-construction pipeline built on CodeFace, a library of reusable Code Primitives.
Existing coding agents, such as SWE-agent, Agentless, and OpenHands, explore a repository, localize and edit code, execute it, and revise from feedback. They generate the components of a new repository themselves. Where prior systems reuse code, it is typically given to a generator as retrieved context, as in RepoCoder (Zhang et al., 2023). It can also be packaged as tools and agent-native primitives that are invoked from outside, as in Agent Primitives and Tool Primitives (Jin et al., 2026a;b). In both cases, the reused artifact stays fixed.
This creates two difficulties. First, writing every component from scratch remains hard at repository scale. For example, the strongest of the 13 backbones we evaluated reached a delivery score of 0.318 and scored zero on 41.0% of the tasks. Second, reused code ships inside the delivered repository, so its interfaces, dependencies, and configuration have to be brought in line with the target. A fixed artifact leaves this work to the central agent.
We address these two difficulties with reuse by adaptation, in which each primitive becomes part of the delivered repository and adapts the component it owns. Reuse in this form raises four challenges. LEGO addresses each with one design choice. First, an executable boundary and dependency closure must be recovered from raw code (C1), so primitives are mined and validated in isolation. Second, the reused code must change to fit the target (C2), so it is adapted inside the component that owns it. Third, the adapted code must be checked without the target's native suite (C3). The primitive does this with its carried tests and a rejection rule. Fourth, an interface change in one component can break the components that consume it (C4). Primitive collaboration therefore passes such interface changes to them in the same round. Retrieval adds a fifth challenge, since a large repository needs only a small part of the library (C5). LEGO meets this challenge with decomposition and on-demand retrieval. It issues 10.6 queries per task, each returning two candidates, so the whole library is never placed in context (Table 3).
We tested these choices in controlled comparisons. A generic editor given the retrieved primitive implementations (RAG + Edit) reached 0.4093, compared with 0.5134 for LEGO (Table 5). Giving this editor the carried tests, contract and provenance, and dependency closure raised its score to 0.5039 (Table 19). Primitive construction produces these fields. Most of LEGO's margin over generic editing therefore comes from what the primitive carries. Adaptation inside the owning component adds the remaining 0.0095 on all tasks (p = 0.49) and +.0351 on the 187 tasks where adaptations cross component boundaries (p = 0.004, Table 19). As an additional check on C4, we discarded the interface changes that primitive collaboration passes to consuming components. This lowered LEGO's score from 0.5134 to 0.4921. We also gave OpenHands, the highest-scoring external agent in Table 2 of the original submission, a CodeFace search tool under LEGO's budget. It reached 0.4106, compared with 0.5134 for LEGO. We have added a sentence on these design choices to Sec. 3.1, and the two new runs to Table 4 and Table 18 (Appendix G.3).
W3/Q5 (1/2). The paper should more clearly explain which important capabilities or challenges are not captured by these existing benchmarks and why a new benchmark is necessary. Beyond introducing additional tasks, it would be useful to articulate what new scientific insights or evaluation dimensions the proposed benchmark enables. What important capability does the proposed benchmark evaluate that RepoCraft, RepoZero, RepoGenesis, or other existing repository-generation benchmarks do not?
R3/A5 (1/2). We would like to clarify that the core of our paper is the method, LEGO. In Sec. 1 of the original submission, we introduced LEGO-REPO to evaluate LEGO. RepoCraft builds a large repository from a project description, RepoZero re-creates an existing program from scratch, and RepoGenesis builds a microservice repository from a README. NL2Repo-Bench builds a repository from a requirements document. All four grade the result by running tests.
With LEGO-REPO, we also ask how reliably frontier models build a complete repository end to end, and how this ability changes as repositories grow. To answer this question, LEGO-REPO adds three dimensions to test-based grading. All three rest on a fixed behavioral target, a complete importable package graded by its own native suite (Appendix M). First, scale-resolved difficulty bands separate the effect of repository size. The field mean of the 13 backbones falls from 0.393 on D1, with 6 modules on average, to 0.038 on D5, with 119 modules (Sec. 5.2). Second, a provenance-gated and floor-normalized delivery score credits only code that the system wrote. Floor normalization removes credit for native tests that pass without any reconstructed implementation (Sec. 4.2). The provenance gate gives no credit to runs in which a suite passes on the package installed in the environment. In the sweep that motivated this gate, 22 repositories appeared to pass in full in this way (Appendix E). Third, per-target donor eligibility controls which library entries each target may use. Excluding same-repository donors lowered LEGO's score from 0.5134 to 0.4743, mostly on D4 and D5 (Table 20).
W4/Q6. The agentic baselines appear insufficiently competitive. LEGO is fundamentally an agentic solution, but the evaluation does not appear to include sufficiently recent state-of-the-art agentic systems for zero-to-one repository generation. In particular, the two agentic baselines included in the paper are from 2024. This makes it difficult to determine whether the reported improvements come from the proposed reusable-primitive methodology or simply from using a stronger contemporary agent pipeline. Comparisons against more recent strong agentic baselines would be important. How does LEGO compare with more recent state-of-the-art agentic repository-generation approaches under the same base model?
R4/A6. Thank you for raising this concern. Following your suggestion, we evaluated three recent agents, RPG/ZeroRepo (2026), Codex CLI (2025), and mini-SWE-agent (2025), with GPT-5.6-terra on all 522 tasks. Each ran at a pinned release with its default settings, as for the agents in Table 2 of the original submission (Appendix I). RPG/ZeroRepo plans a repository graph before generation. Besides SWE-agent and Agentless (2024), Table 2 of the original submission includes OpenHands (ICLR 2025) and Claude Code (2025), its two highest-scoring agents. With Claude Sonnet 5, LEGO scored 0.4953, compared with 0.3170 for Claude Code.
We also report LEGO with CodeFace-Strict, which uses a library re-mined from repositories outside LEGO-REPO, without fork-network, vendoring, or near-duplicate donors of any target. The table reports all systems with GPT-5.6-terra. The OpenHands and LEGO results come from Table 2 of the original submission.
| System | D1 | D5 | All |
|---|---|---|---|
| OpenHands | .556 | .129 | .3263 |
| RPG/ZeroRepo | .631 | .171 | .3920 |
| Codex CLI | .583 | .121 | .3364 |
| mini-SWE-agent | .538 | .097 | .3046 |
| LEGO, CodeFace-Strict | .662 | .207 | .4442 |
| LEGO | .686 | .338 | .5134 |
RPG/ZeroRepo scored highest among the new agents. It came close to LEGO on D1 (.631 vs. .686) but not on D5 (.171 vs. .338). Overall, it reached .3920, compared with .5134 for LEGO, at a higher recorded-call cost (\2.08 per task). LEGO's cost excludes the one-time \$842.85 for building CodeFace. LEGO with CodeFace-Strict reached .4442, also above RPG/ZeroRepo. LEGO's own loop without the library (Feedback + Diagnosis) scored 0.3469, below RPG/ZeroRepo. With retrieval and primitive adaptation added, the same loop reached 0.5134 (Table 4). We added the new agents to Table 2 and their adapter settings to Appendix I.
Thank you for the thoughtful review. We are encouraged that you find repository-level code generation a timely and practically important problem, and reusable Code Primitives an intuitive and potentially effective approach. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We believe the remaining concerns can be sufficiently addressed, and we respond to them in detail below.
W1/Q1 (1/2). The process for constructing this library is not sufficiently described or analyzed. In particular, it is unclear what the fundamental challenges are in building such a library, how primitives are identified and selected, and how their quality and generality are ensured. How exactly are the 1,424 CodeFace primitives constructed and selected? What are the main technical challenges in building the library?
R1/A1 (1/2). Thank you for asking how CodeFace is built. We parsed each donor repository into a dependency graph (Sec. 3.3). Weighted community discovery on this graph proposed component boundaries. We then refined the boundaries against the neighboring code, documentation, and tests. Each component received its interface, dependency closure, and carried tests. Validation selects the primitives and checks their quality. An entry is admitted only when its closure executes, its own test suite passes, and its source repository, commit, and license are on record (Appendix F).
For the 1,107 donor repositories, we counted the rejected candidates by the first check they fail. The table reports the resulting funnel.
Together with 343 harvested and 210 web-sourced entries, these 871 mined primitives make up the 1,424 entries of CodeFace. Dependency closures that do not execute are the largest cause of rejection, followed by carried tests that still fail. The main challenges are therefore to recover an executable boundary with a closed set of dependencies, and to attach validation tests that reach the candidate and pass on it. We have added the funnel and these challenges to Appendix F.
W1/Q1 (2/2). It is unclear whether the current library provides broad coverage of real-world public repositories or is particularly aligned with the proposed benchmark. How do you ensure that its coverage generalizes beyond the evaluation benchmark?
R1/A1 (2/2). We check this in two ways. Every entry records its source repository, so we can remove the entries aligned with a target and measure what remains. We also evaluate LEGO on benchmarks that we did not build. The default CodeFace is partly aligned with LEGO-REPO through same-repository donors. Excluding these donors lowers its coverage of LEGO-REPO from 61.4% to 41.2% (Table 34). Filters comparable to those for the external benchmarks remove same-repository, fork, and near-duplicate donors. Under these filters, LEGO-REPO coverage is 33.1–41.2% (Table 34). This range overlaps the 29.6–34.8% that remains on the four external benchmarks after the entries traced to their targets are removed (Sec. 5.6). LEGO improved over the matched Feedback baseline on all four, for example from 53.8 to 62.9 on RepoZero (Table 9). In the coverage sweep of A2, LEGO's score rose with relevant coverage, from .3551 at 1% to .5134 at 100%.