acceptodds
Under review as a conference paper at ICLR 2027

Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

17 views

Abstract

Large language models (LLMs) equipped with development environments have moved code generation toward repository-scale construction, yet building complete repositories remains difficult because interacting modules, interfaces, configurations, tests, and dependencies must work together. We introduce Code Primitives, agent-native reusable executable components that carry their own interface contracts, dependency closures, validation tests, and provenance. Each primitive uses a resident LLM as a natural-language interface to assess relevance and adapt its implementation, interfaces, and dependencies to the target repository, and we organize validated primitives in CodeFace, a searchable library for repository construction. Each primitive adapts the component it owns. We introduce LEGO (Large-scale repository Engineering via aGent-native reusable cOde primitives), which activates the primitives a task requires, integrates their adapted implementations with task-specific code while resolving constraints that cross component boundaries, and revises the result against executed tests. To measure construction end to end we build LEGO-REPO, a benchmark of executable reconstruction tasks spanning seven software domains, capability tracks, and five difficulty levels, each scored against the package's native test suite between an empty-package floor and an original-source ceiling. The strongest of evaluated backbones reaches a delivery score of and scores zero on 41.0% of tasks; LEGO improves all by on average and raises GPT-5.6-terra from to (+61.4%). In controlled comparisons, adapted primitives achieve higher delivery scores than retrieved code supplied as context or vendored unchanged. The effect persists against independent repository agents, across three external benchmarks, and with a disjointly re-mined CodeFace, and using GPT-OSS-20B for adaptation and diagnosis retains 95.1% of the homogeneous score at 24.0% lower cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Repository Engineering, Code Primitives, Software Engineering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vUAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a repository-level code generation approach based on agent-native reusable code primitives. The key idea is to construct a library of reusable code primitives that encapsulate component interface contracts, dependency closures, validation tests, provenance information, and a resident LLM. Given a repository-level generation task, the proposed LEGO framework retrieves and activates relevant primitives, adapts and integrates their implementations into the target repository, and iteratively revises the generated code based on execution and validation feedback.

Overall, the paper addresses an important problem and reports encouraging empirical results. However, I have several concerns regarding the construction and coverage of the primitive library, the technical novelty of the LEGO pipeline, the motivation for the new benchmark, and the strength of the experimental baselines.

Critical strengths
  1. Repository-level code generation is a timely and practically important problem. Leveraging module-level reusable code primitives is an intuitive and potentially effective approach for improving the reliability of repository generation.
Critical weaknesses
  1. Insufficient discussion of the construction and coverage of the CodeFace primitive library. The paper constructs a CodeFace library containing 1,424 code primitives, but the process for constructing this library is not sufficiently described or analyzed. In particular, it is unclear what the fundamental challenges are in building such a library, how primitives are identified and selected, and how their quality and generality are ensured. More importantly, it is unclear whether the current library provides broad coverage of real-world public repositories or is particularly aligned with the proposed benchmark. I would like to see a controlled study that varies the size and coverage of the primitive library. Such an experiment would help quantify how much of the improvement comes from the LEGO methodology versus strong coverage provided by CodeFace.

  2. The fundamental technical novelty of the LEGO pipeline is unclear. The overall pipeline consists of decomposition, retrieval, adaptation, repository editing, execution feedback, verification, and diagnosis. Individually, these stages appear similar to components commonly found in existing coding-agent systems. The paper should more clearly articulate what fundamental challenges arise specifically from integrating reusable code primitives into repository generation and which algorithmic or system-design contributions of LEGO address these challenges. As currently presented, it is difficult to distinguish the core technical novelty from a well-engineered integration of existing agentic coding techniques.

  3. The motivation and novelty of the new benchmark need to be better justified. There are already several benchmarks targeting repository-level or zero-to-one repository generation, including RepoCraft, RepoZero, and RepoGenesis. The paper should more clearly explain which important capabilities or challenges are not captured by these existing benchmarks and why a new benchmark is necessary. Beyond introducing additional tasks, it would be useful to articulate what new scientific insights or evaluation dimensions the proposed benchmark enables.

  4. The agentic baselines appear insufficiently competitive. LEGO is fundamentally an agentic solution, but the evaluation does not appear to include sufficiently recent state-of-the-art agentic systems for zero-to-one repository generation. In particular, the two agentic baselines included in the paper are from 2024. This makes it difficult to determine whether the reported improvements come from the proposed reusable-primitive methodology or simply from using a stronger contemporary agent pipeline. Comparisons against more recent strong agentic baselines would be important.

Questions for authors
  1. How exactly are the 1,424 CodeFace primitives constructed and selected? What are the main technical challenges in building the library, and how do you ensure that its coverage generalizes beyond the evaluation benchmark?

  2. How sensitive is LEGO's performance to the size and coverage of CodeFace? Could you provide an ablation where, for example, 1%, 10%, 50%, and 90% of the relevant library is available? This would help understand the contribution of primitive coverage.

  3. What happens when the required functionality is missing from CodeFace or when retrieval returns an irrelevant primitive? How gracefully does LEGO fall back to generation or recover from incorrect retrieval?

  4. What fundamental challenges arise specifically from integrating reusable code primitives into repository generation?

  5. What important capability does the proposed benchmark evaluate that RepoCraft, RepoZero, RepoGenesis, or other existing repository-generation benchmarks do not? Could you provide a systematic comparison of task characteristics and evaluation dimensions?

  6. How does LEGO compare with more recent state-of-the-art agentic repository-generation approaches under the same base model?

6 comments

Thank you for the thoughtful review. We are encouraged that you find repository-level code generation a timely and practically important problem, and reusable Code Primitives an intuitive and potentially effective approach. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We believe the remaining concerns can be sufficiently addressed, and we respond to them in detail below.


W1/Q1 (1/2). The process for constructing this library is not sufficiently described or analyzed. In particular, it is unclear what the fundamental challenges are in building such a library, how primitives are identified and selected, and how their quality and generality are ensured. How exactly are the 1,424 CodeFace primitives constructed and selected? What are the main technical challenges in building the library?
R1/A1 (1/2). Thank you for asking how CodeFace is built. We parsed each donor repository into a dependency graph (Sec. 3.3). Weighted community discovery on this graph proposed component boundaries. We then refined the boundaries against the neighboring code, documentation, and tests. Each component received its interface, dependency closure, and carried tests. Validation selects the primitives and checks their quality. An entry is admitted only when its closure executes, its own test suite passes, and its source repository, commit, and license are on record (Appendix F).

For the 1,107 donor repositories, we counted the rejected candidates by the first check they fail. The table reports the resulting funnel.

StageCandidates
Proposed4,903
Rejected, closure not executable1,846
Rejected, no tests reach it774
Rejected, carried tests fail1,412
Admitted871

Together with 343 harvested and 210 web-sourced entries, these 871 mined primitives make up the 1,424 entries of CodeFace. Dependency closures that do not execute are the largest cause of rejection, followed by carried tests that still fail. The main challenges are therefore to recover an executable boundary with a closed set of dependencies, and to attach validation tests that reach the candidate and pass on it. We have added the funnel and these challenges to Appendix F.

W1/Q1 (2/2). It is unclear whether the current library provides broad coverage of real-world public repositories or is particularly aligned with the proposed benchmark. How do you ensure that its coverage generalizes beyond the evaluation benchmark?
R1/A1 (2/2). We check this in two ways. Every entry records its source repository, so we can remove the entries aligned with a target and measure what remains. We also evaluate LEGO on benchmarks that we did not build. The default CodeFace is partly aligned with LEGO-REPO through same-repository donors. Excluding these donors lowers its coverage of LEGO-REPO from 61.4% to 41.2% (Table 34). Filters comparable to those for the external benchmarks remove same-repository, fork, and near-duplicate donors. Under these filters, LEGO-REPO coverage is 33.1–41.2% (Table 34). This range overlaps the 29.6–34.8% that remains on the four external benchmarks after the entries traced to their targets are removed (Sec. 5.6). LEGO improved over the matched Feedback baseline on all four, for example from 53.8 to 62.9 on RepoZero (Table 9). In the coverage sweep of A2, LEGO's score rose with relevant coverage, from .3551 at 1% to .5134 at 100%.

W2/Q4. The fundamental technical novelty of the LEGO pipeline is unclear. The overall pipeline consists of decomposition, retrieval, adaptation, repository editing, execution feedback, verification, and diagnosis. Individually, these stages appear similar to components commonly found in existing coding-agent systems. The paper should more clearly articulate what fundamental challenges arise specifically from integrating reusable code primitives into repository generation and which algorithmic or system-design contributions of LEGO address these challenges. As currently presented, it is difficult to distinguish the core technical novelty from a well-engineered integration of existing agentic coding techniques. What fundamental challenges arise specifically from integrating reusable code primitives into repository generation?
R2/A4. We would like to clarify that the novelty of LEGO lies in a form of reuse in which the reused code becomes part of the delivered repository. The novelty also lies in one design choice for each challenge that this form of reuse raises. LEGO is a repository-construction pipeline built on CodeFace, a library of reusable Code Primitives.

Existing coding agents, such as SWE-agent, Agentless, and OpenHands, explore a repository, localize and edit code, execute it, and revise from feedback. They generate the components of a new repository themselves. Where prior systems reuse code, it is typically given to a generator as retrieved context, as in RepoCoder (Zhang et al., 2023). It can also be packaged as tools and agent-native primitives that are invoked from outside, as in Agent Primitives and Tool Primitives (Jin et al., 2026a;b). In both cases, the reused artifact stays fixed.

This creates two difficulties. First, writing every component from scratch remains hard at repository scale. For example, the strongest of the 13 backbones we evaluated reached a delivery score of 0.318 and scored zero on 41.0% of the tasks. Second, reused code ships inside the delivered repository, so its interfaces, dependencies, and configuration have to be brought in line with the target. A fixed artifact leaves this work to the central agent.

We address these two difficulties with reuse by adaptation, in which each primitive becomes part of the delivered repository and adapts the component it owns. Reuse in this form raises four challenges. LEGO addresses each with one design choice. First, an executable boundary and dependency closure must be recovered from raw code (C1), so primitives are mined and validated in isolation. Second, the reused code must change to fit the target (C2), so it is adapted inside the component that owns it. Third, the adapted code must be checked without the target's native suite (C3). The primitive does this with its carried tests and a rejection rule. Fourth, an interface change in one component can break the components that consume it (C4). Primitive collaboration therefore passes such interface changes to them in the same round. Retrieval adds a fifth challenge, since a large repository needs only a small part of the library (C5). LEGO meets this challenge with decomposition and on-demand retrieval. It issues 10.6 queries per task, each returning two candidates, so the whole library is never placed in context (Table 3).

We tested these choices in controlled comparisons. A generic editor given the retrieved primitive implementations (RAG + Edit) reached 0.4093, compared with 0.5134 for LEGO (Table 5). Giving this editor the carried tests, contract and provenance, and dependency closure raised its score to 0.5039 (Table 19). Primitive construction produces these fields. Most of LEGO's margin over generic editing therefore comes from what the primitive carries. Adaptation inside the owning component adds the remaining 0.0095 on all tasks (p = 0.49) and +.0351 on the 187 tasks where adaptations cross component boundaries (p = 0.004, Table 19). As an additional check on C4, we discarded the interface changes that primitive collaboration passes to consuming components. This lowered LEGO's score from 0.5134 to 0.4921. We also gave OpenHands, the highest-scoring external agent in Table 2 of the original submission, a CodeFace search tool under LEGO's budget. It reached 0.4106, compared with 0.5134 for LEGO. We have added a sentence on these design choices to Sec. 3.1, and the two new runs to Table 4 and Table 18 (Appendix G.3).

W3/Q5 (1/2). The paper should more clearly explain which important capabilities or challenges are not captured by these existing benchmarks and why a new benchmark is necessary. Beyond introducing additional tasks, it would be useful to articulate what new scientific insights or evaluation dimensions the proposed benchmark enables. What important capability does the proposed benchmark evaluate that RepoCraft, RepoZero, RepoGenesis, or other existing repository-generation benchmarks do not?
R3/A5 (1/2). We would like to clarify that the core of our paper is the method, LEGO. In Sec. 1 of the original submission, we introduced LEGO-REPO to evaluate LEGO. RepoCraft builds a large repository from a project description, RepoZero re-creates an existing program from scratch, and RepoGenesis builds a microservice repository from a README. NL2Repo-Bench builds a repository from a requirements document. All four grade the result by running tests.

With LEGO-REPO, we also ask how reliably frontier models build a complete repository end to end, and how this ability changes as repositories grow. To answer this question, LEGO-REPO adds three dimensions to test-based grading. All three rest on a fixed behavioral target, a complete importable package graded by its own native suite (Appendix M). First, scale-resolved difficulty bands separate the effect of repository size. The field mean of the 13 backbones falls from 0.393 on D1, with 6 modules on average, to 0.038 on D5, with 119 modules (Sec. 5.2). Second, a provenance-gated and floor-normalized delivery score credits only code that the system wrote. Floor normalization removes credit for native tests that pass without any reconstructed implementation (Sec. 4.2). The provenance gate gives no credit to runs in which a suite passes on the package installed in the environment. In the sweep that motivated this gate, 22 repositories appeared to pass in full in this way (Appendix E). Third, per-target donor eligibility controls which library entries each target may use. Excluding same-repository donors lowered LEGO's score from 0.5134 to 0.4743, mostly on D4 and D5 (Table 20).

W4/Q6. The agentic baselines appear insufficiently competitive. LEGO is fundamentally an agentic solution, but the evaluation does not appear to include sufficiently recent state-of-the-art agentic systems for zero-to-one repository generation. In particular, the two agentic baselines included in the paper are from 2024. This makes it difficult to determine whether the reported improvements come from the proposed reusable-primitive methodology or simply from using a stronger contemporary agent pipeline. Comparisons against more recent strong agentic baselines would be important. How does LEGO compare with more recent state-of-the-art agentic repository-generation approaches under the same base model?
R4/A6. Thank you for raising this concern. Following your suggestion, we evaluated three recent agents, RPG/ZeroRepo (2026), Codex CLI (2025), and mini-SWE-agent (2025), with GPT-5.6-terra on all 522 tasks. Each ran at a pinned release with its default settings, as for the agents in Table 2 of the original submission (Appendix I). RPG/ZeroRepo plans a repository graph before generation. Besides SWE-agent and Agentless (2024), Table 2 of the original submission includes OpenHands (ICLR 2025) and Claude Code (2025), its two highest-scoring agents. With Claude Sonnet 5, LEGO scored 0.4953, compared with 0.3170 for Claude Code.

We also report LEGO with CodeFace-Strict, which uses a library re-mined from repositories outside LEGO-REPO, without fork-network, vendoring, or near-duplicate donors of any target. The table reports all systems with GPT-5.6-terra. The OpenHands and LEGO results come from Table 2 of the original submission.

SystemD1D5All
OpenHands.556.129.3263
RPG/ZeroRepo.631.171.3920
Codex CLI.583.121.3364
mini-SWE-agent.538.097.3046
LEGO, CodeFace-Strict.662.207.4442
LEGO.686.338.5134

RPG/ZeroRepo scored highest among the new agents. It came close to LEGO on D1 (.631 vs. .686) but not on D5 (.171 vs. .338). Overall, it reached .3920, compared with .5134 for LEGO, at a higher recorded-call cost (\2.64vs.2.64 vs. \\2.08 per task). LEGO's cost excludes the one-time \$842.85 for building CodeFace. LEGO with CodeFace-Strict reached .4442, also above RPG/ZeroRepo. LEGO's own loop without the library (Feedback + Diagnosis) scored 0.3469, below RPG/ZeroRepo. With retrieval and primitive adaptation added, the same loop reached 0.5134 (Table 4). We added the new agents to Table 2 and their adapter settings to Appendix I.

Reviewer_JabBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Code Primitives, reusable executable components containing implementations, interface contracts, dependencies, validation tests, provenance, and an LLM-based adaptation interface. LEGO retrieves and adapts these components to reconstruct 522 existing Python packages from retained public interfaces and native tests after their original implementations are removed. The paper reports substantial improvements over generation without CodeFace, although the default evaluation permits target-related donors and repeated access to the tests used for final grading.

Critical strengths
  • The studied software-reuse problem is practically relevant.
  • The structured component representation is useful.
  • The empirical evaluation is extensive.
  • The paper is clearly presented.
Critical weaknesses
  • The task does not match the stated repository-generation motivation.
  • The default evaluation has substantial answer-derived reuse.
  • The contamination and test-leakage controls are insufficient.
  • The claimed agent-native mechanism is not isolated.
Questions for authors
  1. The benchmark appears to evaluate package reconstruction rather than repository generation from new requirements. Each task is derived from an existing package whose implementation is removed while its public interface and native tests are retained. What natural-language requirement, design document, README content, or behavioral specification is supplied beyond these interfaces and tests, and why should this setting be interpreted as zero-to-one repository engineering?

  2. The motivation and benchmark contribution are not clearly aligned. The paper motivates LEGO by the difficulty of repo-level generation, but existing benchmarks such as NL2Repo-Bench, RepoZero, RepoGenesis, and RepoCraft already evaluate commit-zero or zero-to-one repository construction. What previously unevaluated generation capability does LEGO-REPO measure, beyond test-driven reconstruction and its floor–ceiling scoring protocol?

  3. The default result has direct answer-derived reuse risk. CodeFace may contain primitives mined from the same repository being reconstructed, and excluding same-repository donors reduces the score from 0.5134 to 0.4743; on D5, only 44% of the default gain is retained. Why are target-derived primitives allowed in the main evaluation, and why is the fully disjoint CodeFace result not used as the headline estimate?

  4. The current provenance checks do not exclude model-side contamination. The 522 targets are popular public GitHub repositories with at least 1,000 stars, so the evaluated LLMs may have encountered their source code during training. Can the authors measure generated-to-reference similarity, use repositories created after the models’ training cutoffs, or evaluate on private or newly constructed targets?

  5. The native tests function as both development feedback and the grading key. LEGO repeatedly executes these tests during up to five revision rounds and is then scored using the same suite. How does the evaluation distinguish semantic reconstruction from test-guided fitting? Please report performance on held-out tests or independently generated behavioral checks that are unavailable during construction.

  6. The library decontamination controls remain incomplete. Repository identity, fork filtering, and token-level MinHash cannot exclude refactored copies, vendored implementations, semantically equivalent code, or documentation-derived solutions. Can the authors provide component-level source overlap and semantic-similarity analyses between each retrieved primitive and its target module?

  7. The central agent-native mechanism is not established by the ablations. In Table 19, a generic editor supplied with the primitive tests, contracts, provenance, and dependency closures reaches 0.5039, versus 0.5134 for LEGO, and the confidence interval for the difference includes zero. Does the evidence support resident-LLM adaptation, or primarily the value of supplying structured reusable code and metadata?

  8. Primitive collaboration is not independently validated. The paper attributes the improvement to requirement propagation and component-level collaboration, but does not isolate this mechanism. Please disable inter-primitive communication while retaining the same primitives, adaptation context, diagnosis, tests, and budget.

The paper presents a useful software-reuse system, but the concerns above currently prevent the results from establishing repository generation for genuinely new requirements. I am willing to raise my score if the rebuttal provides convincing explanations.

7 comments

We appreciate your thoughtful review and are encouraged by your recognition of our work: for finding the software-reuse problem practically relevant and the structured component representation useful, and for considering the evaluation extensive and the presentation clear. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We also appreciate your openness to revisiting the score, and we provide detailed responses below.


W1/Q1 (1/2). The task does not match the stated repository-generation motivation. The benchmark appears to evaluate package reconstruction rather than repository generation from new requirements. Why should this setting be interpreted as zero-to-one repository engineering?
R1/A1 (1/2). Thank you for raising this concern. The setting is zero-to-one construction because the system builds the whole package from an empty implementation tree. Every task removes the implementation files of the target package, so the system writes every delivered module (Sec. 4). Each task supplies a specification rather than a partial implementation. Repository-generation benchmarks differ in the form of this specification. NL2Repo-Bench supplies a requirements document, RepoGenesis supplies a README, and LEGO-REPO supplies the public interface with its docstrings and the native suite.

The task does not match the stated repository-generation motivation. The benchmark appears to evaluate package reconstruction rather than repository generation from new requirements. Why should this setting be interpreted as zero-to-one repository engineering?
Thank you for raising this concern. The setting is zero-to-one construction because the system builds the whole package from an empty implementation tree. Every task removes the implementation files of the target package, so the system writes every delivered module (Sec. 4). Each task supplies a specification rather than a partial implementation. Repository-generation benchmarks differ in the form of this specification. NL2Repo-Bench supplies a requirements document, RepoGenesis supplies a README, and LEGO-REPO supplies the public interface with its docstrings and the native suite.

To examine repository generation from natural-language requirements, we evaluated Feedback and LEGO with GPT-5.6-terra in two settings without native tests during construction. In LEGO-REPO-NL, each of the 522 tasks provides the README, docs, docstrings, and signatures. The native suite stays hidden until grading. On NL2Repo-Bench, we followed its published protocol. The table reports the scores of Feedback and LEGO in both settings.

BenchmarkFeedbackLEGO
LEGO-REPO-NL.2394.3962
NL2Repo-Bench37.643.9

LEGO scored .3962 on LEGO-REPO-NL, compared with .2394 for Feedback. On NL2Repo-Bench, LEGO reached 43.9, compared with 37.6 for Feedback. We added both settings to Table 9 (Sec. 5.6) and described their input construction and CodeFace filtering in Appendix I.

W2/Q3. The default evaluation has substantial answer-derived reuse. The default result has direct answer-derived reuse risk. CodeFace may contain primitives mined from the same repository being reconstructed, and excluding same-repository donors reduces the score from 0.5134 to 0.4743; on D5, only 44% of the default gain is retained. Why are target-derived primitives allowed in the main evaluation, and why is the fully disjoint CodeFace result not used as the headline estimate?
R2/A3. Thank you for raising this concern. We have revised the abstract to also report LEGO's gain on 106 tasks from repositories created after the backbone's knowledge cutoff. No CodeFace entry comes from these repositories. Target-derived entries are in the default CodeFace because it was mined from public repositories, which include the targets. The headline result uses the default library so that it stays comparable with the main comparisons of the original submission. These include the comparison with external agents (Table 2) and the 13-backbone sweep (Table 7).

To obtain a fully disjoint estimate, we built CodeFace-Strict. We started from the 518 primitives that we re-mined from 640 repositories outside LEGO-REPO in Appendix G.17 of the original submission. We then removed every entry in the fork network of any target and every entry from a donor that vendors any target. We also removed every entry whose AST-normalized similarity to any target file is above 0.6. This left 463 primitives. We re-ran LEGO (GPT-5.6-terra) on all 522 tasks and changed only the library. The table reports the D5 and overall scores, with the other results from Table 20.

ConfigurationD5All
Feedback.129.3180
LEGO, full library.338.5134
LEGO, same-repo donors excluded.221.4743
LEGO, CodeFace-Strict.207.4442

With CodeFace-Strict, LEGO raised the score from 0.3180 to 0.4442 and retained 64.6% of the default gain. On D5, it retained 37.3%, with a score of .207 compared with .129 for Feedback. In every difficulty band, LEGO with CodeFace-Strict scored above Feedback (Table 20). We added the CodeFace-Strict results to Tables 2, 4, and 20.

W3/Q4. The contamination and test-leakage controls are insufficient. The current provenance checks do not exclude model-side contamination. The 522 targets are popular public GitHub repositories with at least 1,000 stars, so the evaluated LLMs may have encountered their source code during training. Can the authors measure generated-to-reference similarity, use repositories created after the models’ training cutoffs, or evaluate on private or newly constructed targets?
R3/A4. Thank you for suggesting these analyses. We would like to clarify that LEGO and its from-scratch baseline use the same backbone (Tables 4 and 7). Any source code that the backbone has memorized is therefore available to both. With GPT-5.6-terra, the baseline reaches 0.3180 and scores zero on 41.0% of the tasks, so memorized code alone does not yield a working repository. Following your suggestion, we also measured generated-to-reference similarity and built new tasks from repositories created after the backbone's knowledge cutoff. We tested whether LEGO's outputs are closer to the original source than the baseline's, and whether its gain holds on these new tasks.

Both analyses used GPT-5.6-terra. For similarity, we compared every generated module of the LEGO-REPO runs with the removed original module by token Jaccard. For the post-cutoff analysis, we built 106 tasks from repositories created after the knowledge cutoff of GPT-5.6-terra and graded them in the same way as LEGO-REPO. The table reports both analyses.

ConfigurationMedian token JaccardPost-cutoff score
Feedback0.17.3418
LEGO0.21.4985
LEGO, CodeFace-Strict0.18.4726

Default LEGO's modules had a higher median similarity to the original than Feedback's (0.21 vs. 0.17). The overlap analysis in A6 also shows that same-repository donors supply 90.7% of default LEGO's high-overlap adaptations. With CodeFace-Strict, which removes these donors, the median similarity was close to Feedback's (0.18 vs. 0.17). With this library, LEGO scored 0.4442 on LEGO-REPO, compared with 0.3180 for Feedback.

On the post-cutoff tasks, default LEGO raised the score from .3418 to .4985. This gain is 83.1% of its gain on LEGO-REPO tasks from the same bands. This share is close to the 80.0% that LEGO retained on LEGO-REPO without same-repository donors (Table 20). The new repositories have no such donors. LEGO with CodeFace-Strict, which has no target-related donor on either task set, raised the score from .3418 to .4726. This gain of 0.1308 is close to its gain of 0.1262 on LEGO-REPO. The held-out tests in A5 show that LEGO's lead holds on tests hidden during construction (.4806, and 0.4113 with CodeFace-Strict, vs. .2871 for Feedback). We have added both analyses to the new Appendix G.18 (Table 36). The 106 tasks are listed in the anonymous repository of the original submission (benchmark/post_cutoff/).

Reviewer_nwwqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LEGO, a framework for repository-level code generation based on reusable Code Primitives. Each primitive includes an implementation, interface contract, dependencies, tests, and provenance, and can adapt itself to a target repository using an LLM. LEGO retrieves relevant primitives, coordinates changes across components, and uses execution feedback to revise the generated repository. The authors also introduce LEGO-REPO, a benchmark of 522 Python package reconstruction tasks.

The experimental results are encouraging. LEGO improves all 13 evaluated backbones, with GPT-5.6-terra increasing from 0.3180 to 0.5134 on LEGO-REPO. The ablation studies suggest that primitive adaptation contributes more than retrieval alone. However, the extent to which these gains come from the proposed abstraction itself, rather than the additional adaptation and validation machinery or potential overlap between donor repositories and evaluation tasks, remains an important question.

Critical strengths

A practical approach to repository-level code generation. The paper addresses a real limitation of coding agents: generating individual components is not enough when interfaces, dependencies, configurations, and tests must work together.

The adaptation mechanism is more interesting than retrieval alone. Each primitive can modify its implementation and dependencies to fit the target repository, while communicating changes that affect other components. This is a useful distinction from simply providing retrieved code as additional context.

The experimental evaluation is relatively broad. The authors evaluate 13 backbones, conduct ablation studies, compare against existing repository agents, and test transfer to three external benchmarks. The results provide more evidence than a single-model demonstration.

The paper reports meaningful failure cases. The analysis acknowledges that component reuse does not guarantee successful integration, particularly when repositories rely on plugin registries or other global conventions. This makes the evaluation more informative.

Critical weaknesses

The novelty of the overall framework needs to be established more clearly. LEGO combines code retrieval, component-level adaptation, dependency-aware coordination, and test-guided revision. These are useful mechanisms, but several are established ideas in software engineering and agent-based code generation. The paper should make clearer which part of the design is fundamentally new and why the proposed primitive abstraction is necessary beyond a well-engineered retrieval-and-editing pipeline.

Potential data leakage remains a concern. CodeFace contains primitives mined from repositories, harvested from previous construction runs, and sourced from the web. The authors acknowledge that repository-level exclusion cannot rule out forks, vendored code, near-duplicates, or cross-task contamination. The disjoint-library experiments are helpful, but the reduced gains suggest that provenance and library composition have a nontrivial effect on the results. Stronger evidence is needed to establish how much improvement comes from genuine adaptation rather than access to closely related implementations.

The benchmark measures test agreement rather than overall repository quality. LEGO-REPO evaluates reconstructed packages against their original test suites. This is reasonable for functional evaluation, but a high score does not necessarily imply that the implementation is maintainable, secure, architecturally sound, or correct on unseen inputs. The paper should be more cautious about generalizing these results to repository engineering in practice.

The ablation studies do not fully isolate the contribution of each design choice. Primitive construction and adaptation change several things at once, including component boundaries, interfaces, dependencies, tests, and the amount of task-specific editing. The authors acknowledge this limitation. As a result, the experiments show that the complete approach works better, but do not fully explain which properties of Code Primitives account for the improvement.

The cost analysis is incomplete. The reported per-task cost focuses on recorded model calls. Building and maintaining CodeFace also requires repository mining, dependency analysis, validation, storage, and library updates. These costs may be amortized over many tasks, but the paper does not establish the workload or reuse frequency needed to make the overall approach cost-effective.

Questions for authors

How much of LEGO's improvement remains when CodeFace is built exclusively from repositories that are strictly disjoint from all evaluation targets, including forks, vendored code, and near-duplicates? Can the authors report this result separately for each difficulty level?

Can the authors isolate the contribution of the primitive abstraction from that of additional LLM calls, component-level editing, and validation? In particular, how would a strong retrieval-augmented coding agent with the same token and execution budget perform?

What is the total cost of building and maintaining CodeFace, and how many construction tasks are needed to amortize this cost compared with a from-scratch agent?

How sensitive are the results to the quality of the mined component boundaries and inferred interface contracts? Does LEGO still perform well when primitives contain incomplete dependencies, weak tests, or incorrect metadata?

Since LEGO-REPO uses native tests as its primary correctness signal, have the authors evaluated the generated repositories on additional tests or unseen inputs to check whether the improvements generalize beyond the benchmark's test suites?

6 comments

We thank the reviewer for the constructive comments. We appreciate the recognition that LEGO is a practical approach to repository-level code generation and that its adaptation mechanism is more interesting than retrieval alone. We also appreciate the acknowledgment that our evaluation is relatively broad and that the reported failure cases make it more informative. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We address each of the raised weaknesses and questions below.


W1 (1/2). The novelty of the overall framework needs to be established more clearly. LEGO combines code retrieval, component-level adaptation, dependency-aware coordination, and test-guided revision. These are useful mechanisms, but several are established ideas in software engineering and agent-based code generation. The paper should make clearer which part of the design is fundamentally new.
R1 (1/2). Thank you for raising this point. We have revised Sec. 2 (Key Differences) and Sec. 3.1 to make the new part explicit. Code Primitives are a new unit of reuse. Each primitive carries its own interface contract, dependency closure, validation tests, and provenance, becomes part of the delivered repository, and is adapted and validated as one piece. Sec. 3.1 now also states what each design choice of LEGO is for.

W1 (2/2). Why the proposed primitive abstraction is necessary beyond a well-engineered retrieval-and-editing pipeline.
R1 (2/2). A retrieval-and-editing pipeline retrieves code and leaves four problems to the editor. First, raw files are not reusable units, so an executable boundary and a dependency closure have to be recovered from them. Second, the reused code has to change to fit the target. Third, the adapted code has to be checked without the target's native suite. Fourth, an interface change in one component can break the components that consume it. The primitive abstraction answers each problem. Mining with isolated validation recovers the boundary and the closure. The resident LLM adapts the component it owns. The carried tests and the rejection rule check the result before delivery. Primitive collaboration passes interface changes to the consumers.

Each of these is measured separately. File RAG + Edit retrieves whole donor files, without a recovered boundary, dependency closure, contract, or carried tests. It reaches 0.3528, close to the 0.3469 of the same loop without a library. Replacing those files with constructed primitives raises the score to 0.4093, and primitive adaptation raises it to 0.5134 (Table 5). Import + Call vendors a primitive unchanged and reaches 0.3430, compared with 0.3469 without a library. It falls 0.013, 0.047, and 0.075 below the no-library reference on D3, D4, and D5 (Sec. 5.4). Withholding the carried tests from every primitive costs 0.0338 (Table 19). Dropping the requirements that a primitive sends to its consumers costs 0.0213 (Table 18). An agent given the same library and LEGO's budget reaches 0.4106, compared with 0.5134 for LEGO (Table 4). Most of LEGO's gain therefore comes from constructed and adapted primitives rather than from retrieval and editing.

W2/Q1. Potential data leakage remains a concern. CodeFace contains primitives mined from repositories, harvested from previous construction runs, and sourced from the web. The authors acknowledge that repository-level exclusion cannot rule out forks, vendored code, near-duplicates, or cross-task contamination. The disjoint-library experiments are helpful, but the reduced gains suggest that provenance and library composition have a nontrivial effect on the results. Stronger evidence is needed to establish how much improvement comes from genuine adaptation rather than access to closely related implementations. How much of LEGO's improvement remains when CodeFace is built exclusively from repositories that are strictly disjoint from all evaluation targets, including forks, vendored code, and near-duplicates? Can the authors report this result separately for each difficulty level?
R2/A1. Thank you for raising this concern. With a library that excludes forks, vendored code, and near-duplicates of every target, LEGO retained 64.6% of its improvement. Its gain stayed positive at every difficulty level, which we report separately below. For cross-task contamination, removing the 296 harvested entries that come from LEGO-REPO tasks changed the score from 0.5134 to 0.5128 (Appendix G.17).

Following your suggestion, we built this library, CodeFace-Strict. We started from the 518 primitives that we re-mined from 640 repositories outside LEGO-REPO in the original submission (Appendix G.17). With them, LEGO scored 0.4528, compared with 0.3180 for Feedback and 0.5134 with the default library. This re-mined library still admitted forks, vendored code, and refactored near-duplicates of the targets. From the 518 primitives, we removed every entry in the fork network of any target and every entry from a donor that vendors any target. We also removed every entry whose AST-normalized similarity to any target file is above 0.6. This left 463 primitives. As a re-mined library, CodeFace-Strict contains no harvested or web-sourced entry. We re-ran LEGO (GPT-5.6-terra) on all 522 tasks and changed only the library. The table reports the delivery score per difficulty level, with the Feedback and LEGO results from Table 2 of the original submission.

ConfigurationD1D2D3D4D5All
Feedback.525.382.274.206.129.3180
LEGO.686.583.483.409.338.5134
LEGO, CodeFace-Strict.662.541.418.301.207.4442
Retained share85.1%79.1%68.9%46.8%37.3%64.6%

With CodeFace-Strict, LEGO raised the score from 0.3180 to 0.4442, compared with 0.5134 with the default library. The retained share fell with difficulty, from 85.1% on D1 to 37.3% on D5. On D5, LEGO with CodeFace-Strict scored .207, compared with .129 for Feedback.

As an additional check on access to closely related implementations, we compared each successful adaptation under CodeFace-Strict with the target module it replaced. Of these adaptations, 0.3% had a token Jaccard above 0.6, and 0.9% had an AST-normalized similarity above 0.6. Under an independent code-embedding model, 4.1% had a cosine similarity above 0.9. We added the CodeFace-Strict results to Tables 2 and 4, Table 20 (Appendix G.4), and Table 34 (Appendix G.17), and the similarity check to the new Appendix G.18 (Table 35).

W3. The benchmark measures test agreement rather than overall repository quality. LEGO-REPO evaluates reconstructed packages against their original test suites. This is reasonable for functional evaluation, but a high score does not necessarily imply that the implementation is maintainable, secure, architecturally sound, or correct on unseen inputs. The paper should be more cautious about generalizing these results to repository engineering in practice.
R3. Thank you for pointing this out. We have revised Sec. 4.2 to state that the delivery score measures functional reconstruction quality, and the conclusion to state LEGO's gains in these terms. A score of 1 means agreement with the original package on the frozen suite. It does not imply full semantic correctness, maintainability, or open-ended architecture design (Appendix L). Sec. 4.2 reports a non-degeneracy audit. It labels a module genuine only if, judging from its code, the module would also be correct on inputs that the graded tests do not cover. In this audit, LEGO had 102 genuine implementations out of 120, compared with 74 for Feedback (Appendix O).

W4/Q2 (1/2). The ablation studies do not fully isolate the contribution of each design choice. Primitive construction and adaptation change several things at once, including component boundaries, interfaces, dependencies, tests, and the amount of task-specific editing. The authors acknowledge this limitation. As a result, the experiments show that the complete approach works better, but do not fully explain which properties of Code Primitives account for the improvement. Can the authors isolate the contribution of the primitive abstraction from that of additional LLM calls, component-level editing, and validation?
R4/A2 (1/2). Thank you for suggesting these controls. Yes, we compared LEGO with a matched control for each of these factors. For additional LLM calls, Sec. 5.5 compares LEGO with Feedback at an attempt budget of eight per module (b = 8). At this budget, Feedback has about the same recorded-call cost as LEGO (\2.07vs.2.07 vs. \\2.08 per task, Table 8). It reached 0.3437, compared with 0.5134 for LEGO. For component-level editing, RAG + Edit gives a generic editor the retrieved primitive implementations under the same attempt budget. The editor does not receive the carried tests, metadata, or execution state of the primitive. RAG + Edit reached 0.4093, compared with 0.5134 for LEGO (Table 5).

For validation and the other primitive fields, Table 19 adds each field to this editor in turn, with retrieval, diagnosis, and the budget held fixed. The fields are added cumulatively, so each increment is measured given the fields added before it. Carried tests raised the editor's score from 0.4093 to 0.4431, contract and provenance to 0.4818, and dependency closure to 0.5039. With all three fields, the editor keeps every field of the primitive except its resident LLM. The remaining difference from LEGO (0.5039 vs. 0.5134) thus measures adaptation inside the owning component. Its 95% CI is

−0.0071,+0.0148−0.0071, +0.0148

(p = 0.49). Most of LEGO's margin over generic editing therefore comes from what the primitive carries. Primitive construction produces these fields. The robustness study in A4 also shows that coarser and finer component boundaries lowered LEGO's score from 0.5134 to .5011 and .4962.

As an additional check, we split the tasks by whether an adaptation changes a signature, representation, or configuration key that another primitive uses. We call such tasks cross-boundary. The table reports the difference between LEGO and the editor with all three fields, with the result on all tasks from Table 19.

TasksLEGO − editor95% CI
All+.0095
−.0071,+.0148−.0071, +.0148

| | Cross-boundary | +.0351 |

+.0109,+.0590+.0109, +.0590

| | Other | −.0048 |

−.0146,+.0051−.0146, +.0051

|

On the 187 cross-boundary tasks, LEGO scored .0351 higher than the editor. On the other 335 tasks, the difference was −.0048. We have added this task split as panel (b) of Table 19.

Q2 (2/2). In particular, how would a strong retrieval-augmented coding agent with the same token and execution budget perform?
A2 (2/2). Under LEGO's budget, a strong retrieval-augmented agent reached 0.4106, compared with 0.5134 for LEGO. Following your suggestion, we gave OpenHands (GPT-5.6-terra) a CodeFace search tool with the same embedding and top-2 retrieval as LEGO. OpenHands is the highest-scoring external agent in Table 2 of the original submission. With this tool, the agent could read, copy, and edit full primitives and run both the native and the carried tests. We ran it on all 522 tasks in two settings. In the first, we capped it at LEGO's mean recorded per-task usage of 1.03M input and 78k output tokens, with at most five suite runs per module. In the second, it used its default stopping rules. The table reports both runs next to the OpenHands and LEGO results of Table 2.

SystemAll$/task
OpenHands.32632.41
OpenHands + CodeFace, same budget.41062.05
OpenHands + CodeFace, default budget.42713.62
LEGO.51342.08

Under the same budget, the agent's score is close to the 0.4093 of RAG + Edit. It edited a retrieved primitive on 63.2% of the tasks and re-ran that primitive's carried tests on 14.2% of the tasks. LEGO reruns these tests after every adaptation. With its default stopping rules, the agent reached .4271 at \3.62pertask,comparedwith0.5134at3.62 per task, compared with 0.5134 at \\2.08 for LEGO. We added the same-budget run to Table 4, both runs to Table 18 (Appendix G.3), and the tool interface to Appendix I.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.