acceptodds
Withdrawn from ICLR 2027

ReArch-Coder: Requirement Alignment and Architecture Design for Reliable Repository-Level Code Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-source language models offer low cost and flexible deployment, making them attractive for practical coding agents. Yet they still struggle to generate executable repositories reliably from requirement documents. Existing approaches use harness engineering to organize repository generation, with closed-source frontier models serving as the underlying base models. However, applying these approaches to open-source models exposes two limitations: (1) Models may misinterpret requirements and implement functionality that departs from user needs. (2) Their limited architecture design capabilities can lead to flawed repository designs that prevent modules from working together. To address these challenges, we propose ReArch-Coder, a method for requirement alignment and repository architecture design optimization in open-source models. First, ReArch-Coder aligns the target model’s understanding of the requirement document through diagnostic questions and answers and factual hints. Then, it formulates architecture design as a trainable agentic decision-making process and optimizes it using GRPO with rewards based on requirement coverage and dependency modularity. Experimental results on representative repository-level code generation benchmarks demonstrate the effectiveness of our method. For example, ReArch-Coder outperforms the strongest coding-agent baseline on RepoGenesis by 21.8 percentage points in Passed and 19.6 percentage points in Resolved.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Repository-Level Code Generation, Requirement Alignment, Agentic Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_w2rkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ReArch‑Coder, a framework for requirement alignment and repository‑level architecture design optimization tailored for open‑source code models. ReArch‑Coder first calibrates the target model’s comprehension of requirement documents via diagnostic question‑answering and grounded factual hints. It then casts repository architecture design as a learnable agentic decision‑making procedure, which is optimized with GRPO using reward signals derived from requirement coverage and dependency modularity.

Critical strengths
  1. The problem definition is clear and practically significant. The paper targets two key failure modes of open‑source coding agents for repository‑level generation: requirement misunderstanding and poor architectural design.
  2. The requirement‑alignment module for the target (student) model is well‑grounded in classical software‑requirements‑engineering literature. The diagnostic question‑answering pipeline offers a principled way to detect and fix comprehension defects of the student model prior to implementation.
  3. The agent‑driven architecture‑design formulation is interesting. Treating multi‑step repository planning as a trainable decision trajectory optimized by GRPO with modularity‑aware rewards provides a fresh perspective for repository‑level code‑agent research.
Critical weaknesses
  1. The requirement‑alignment module heavily depends on a strong closed‑source teacher model to create diagnostic questions, generate corrective hints, and check whether those hints are valid. In effect, this approach performs a form of input‑based distillation: the deep understanding of software requirements comes almost entirely from the teacher model, instead of being newly learned by the target code model. Although this workflow yields practical improvements, this component has limited conceptual novelty. It does not fix the target model’s built‑in weaknesses in understanding requirements — it only compensates for them by rewriting the input prompt.

  2. The requirement‑coverage reward for GRPO optimization contains clear shortcomings. It only checks whether each requirement ID is assigned to some file in the proposed architecture. It does not check whether that file is logically appropriate to implement the given requirement. This creates an easy gaming opportunity: the model can naively attach requirement IDs to random files to inflate its reward score, while still producing poorly‑structured repository designs.

  3. The experiments lack sufficient ablations over the teacher‑model component. All main results rely on powerful closed‑source teacher models for requirement alignment. The paper does not report results when substituting the teacher with weaker open‑source alternatives. Therefore, we cannot tell how much performance would drop if one wants to deploy this whole pipeline without external proprietary LLM APIs. This limits our understanding of its real‑world portability.

Questions for authors

See the weaknesses

Reviewer_LeuNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReArch-Coder a repo-level code generation method with two parts.

  1. harness-side test-time requirement alignment and
  2. offline reinforcement learning for repository architecture design. At test time, ReArch-Coder addes additional two model with different roles: GPT-5.5 writes diagnostic questions for their model to anser, reference answers, and corrective hints; and Claude-Opus-4.6 judges the answers from the main model and verifies each hint before it is added to the requirement document. For the main model, the authors post-train different instruct models with GRPO on architecture-design trajectories, with rewards for requirement coverage and dependency modularity. At inference time, this model designs the repository architecture and then implements it. The paper evaluates three open-source base models across four agent harness on NL2Repo-Bench and RepoGenesis. The paper also provides analysis on RPG and Paper2Code comparisons.

In general, the results are promising, and the main functional evaluation is a strength. However, I do find this paper needs substaintial revision and explaination. I'd happy increase my rating if the authors could address my concerns.

Critical strengths
  1. The evaluation shows good improvements across the benchmarks and models.
  2. The component ablations support the design. Table 2 provide analysis all the propoposed additions in details. The architecture-trained system without alignment still achieves Passed rates of 22.2% and 35.6% on the two benchmarks, above the original Qwen baseline. One left is it does not isolate RL from the architecture-planning workflow (see weakness). The analyses by requirement dimension and by question budget show more about how alignment behaves. These detailed analyses cover only Qwen3-Coder-30B with Codex. Within that setting, they are useful checks of the components.
Critical weaknesses
  1. The two introduced methods are not connected, and both of them are but really well motivated from the presentation, so the methodology seems a bit ad-hoc. The abstract and introduction name requirement misunderstanding and poor architecture as the problems to solve, but does not justify why the authors think they are the gap that cause a repository to fail, compared with implementation errors or other causes for smaller models. This gap matters more for the architecture post-training. The paper does not show that incorrect sub-directory layout assignments are a recurring cause of failure. so it is hard to determine the soundness of the method v.s. just trained on more data.
  2. The incremental benefit of RL over explicit architecture planning is also unclear. Appendix C provides detailed instructions for requirement allocation, architecture composition, diagnosis, and revision. I think adding an ablation of RL v.s. providing these instructions as prompt to agent directly (via system prompt, or agent skills) should help, especially consider that the base models authors use are already instruction-tuned.
  3. Requirement alignment calls GPT-5.5 and Claude-Opus-4.6 at test time, while the coding baselines use the smaller models. This comparison shows improvement over those baselines. It leaves open whether running either frontier model directly in the same agent framework would give a better trade-off between performance and cost. I understand that the authors cannot do RL on these models, so building up from the previous point (i.e., system instruction or skill) should provide better insights.
Questions for authors

Please address the concerns in Weaknesses. And one clarification: For the experiments, does the requirement alignment run as external preprocessing or as partg of the harness's workflow? Please explain how the diagnostic and masked-recovery calls are executed and how the diagnostic context is managed in the coding session.

Reviewer_Jeh2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReArch-Coder, a two-stage method designed to enhance repository-level code generation from requirement documents for open-source Large Language Models (LLMs). In Stage 1 (Requirement Alignment), the system employs teacher models to generate up to 20 diagnostic questions based on the requirement document. It then evaluates the target model’s responses, generates factual hints for any incorrect answers, validates these hints via a masked recovery task, and integrates them into the original document. In Stage 2 (Repository Architecture Design Optimization), the architecture design task is structured as a four-step process i.e., i) Requirement Allocation; ii) Initial Architecture Synthesis; iii) Diagnosis; iv) Final Architecture Refinement. This step is trained using the GRPO reinforcement learning algorithm. The training objective utilizes a reward function that combines requirement coverage (RcovR_{cov}) and directory-based dependency clustering (measured via Newman-Girvan modularity, RdepR_{dep}). Experiments conducted with Qwen3-Coder-30B, Qwen3-30B, and CodeLlama-34B on the NL2Repo-Bench and RepoGenesis datasets demonstrate significant improvements in test pass rates (%Passed and %Resolved).

Critical strengths

Accurately identifying two key bottlenecks in repository-level code generation: The paper correctly pinpoints two primary reasons why open-source models often fail when building projects from scratch—misinterpreting behavioral or structural constraints defined in requirements documentation, and poor architectural design regarding module separation. Comprehensive evaluation across various agent frameworks and foundation models: The authors integrated and tested their approach using three open-source models combined with four popular agent frameworks (SWE-Agent, OpenHands, Qwen-Agent, and Codex), as well as two specialized methods (RPG and Paper2Code), covering both Python library projects and Python/Java-based microservices.

Critical weaknesses

A fundamental contradiction regarding incentives and an unfair advantage derived from closed-source models ("Frontier Teacher Leakage") during inference: In the Abstract and Introduction, the authors emphasize the optimization of open-source models, citing lower costs and security requirements. However, in Section 4.1 and Appendix E, the "Requirement Alignment" process utilizes GPT-5.5 to analyze requirement documents, formulate questions, generate ground-truth answers, and produce hints. Furthermore, it employs Claude-Opus-4.6 to perform diagnostic scoring and validate these hints.

The 4-step agentic trajectory is nothing more than a single-turn JSON generation. Although, Section 3.2 characterizes the architectural design process as an agentic decision-making process involving a sequence of four steps, Appendix C (Table 11) reveals that all four components (M,A(0),Δ,AM, A^{(0)}, \Delta, A) are generated within a single prompt, which requests a JSON object containing four distinct fields. There is no interaction with an environment, nor is there any intermediate execution feedback between these steps—precluding any legitimate claim to agentic RL.

The RL reward functions (RcovR_{\text{cov}} and RdepR_{\text{dep}}) are superficial and highly susceptible to reward hacking. As detailed in Appendices D.1 and D.2, the coverage reward Rcov(A)R_{\text{cov}}(A) merely checks for the presence of identifier strings (e.g., "R1", "R2", ...) within the "requirement_ids" array of any file in the output JSON, without verifying semantic compatibility.

There is also a significant risk of overfitting, given the extremely small training set—comprising only 52 projects from NL2Repo-Bench and 53 from RepoGenesis—coupled with the fact that the authors performed 500 training iterations using GRPO (with G=8G=8 samples per document) for a 30B/34B parameter model. Furthermore, there is a lack of transparency regarding token costs.

Questions for authors

Why does a method designed for an open-source framework rely on both GPT-5.5 and Claude-Opus-4.6 during the inference stage for each individual requirement document?

What is the content referenced in Appendix ?? (at line 1321, Appendix D.1)? Not clear - you must proof read the article.

Do the figures of 8.4K/6.8K tokens for the alignment phase in Figure 5b account for the entirety of the prompts and responses from GPT-5.5 (checklist generation) and Claude-Opus-4.6 (diagnosis plus up to 3 rounds of verification per suggestion)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.