acceptodds
Under review as a conference paper at ICLR 2027

ArchRSBench: Enhance LLM Reasoning for Repository Resolution

Abstract

While Large Language Models (LLMs) excel at localized code generation, scaling to repository-level resolution remains a formidable challenge. In practice, monolithic implementations are intractable; developers must decompose features into a structured “Chain of Patches” managed as a Directed Acyclic Graph (DAG) for safe parallelization and review. However, existing benchmarks evaluate end-to-end execution, entirely overlooking the a priori architectural reasoning required to plan these structures. We introduce ArchRSBench, a rigorous benchmark designed to isolate and evaluate system-level architectural decomposition. It formalizes feature planning as a DAG generation task, challenging models to define semantic subtasks, localize files, and map dependencies. We propose a Dual-Evaluation Framework combining deterministic graph-theoretic metrics with LLM-as-a-Judge assessments, supported by a strict anti-contamination pipeline. Our curation yields 1,525 high-fidelity instances across 13 diverse repositories. Empirical evaluations expose critical vulnerabilities: the leading frontier model achieves a zero-shot deterministic score of only 67%. Crucially, architectural planning requires rigorous constraint scaffolding; our specialized Tri-Agent pipeline unlocks latent reasoning capacity, yielding a 12% absolute improvement. Furthermore, augmenting prompts with file contents provides negligible gains over operating strictly on the directory tree topology. Thus, current LLMs over-rely on localized lexical pattern matching, fundamentally lacking the higher-order topological reasoning required for true architectural intelligence.

Primary area:
datasets and benchmarks
Keywords:
Repository-Level Code Generation, Architectural Reasoning, Software Benchmarking, Task Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_oxjyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ArchRSBench evaluates repository-level architectural planning as DAG generation, covering subtask semantics, target file mapping, execution order, and granularity. It contains 1,525 instances from 13 repositories, with ground truth DAGs reconstructed from historical pull requests. Validation includes deterministic metrics, LLM-as-a-Judge evaluation, five model baselines, a Tri-Agent pipeline, and context representation ablations.

Critical strengths
  • An interesting evaluation target. Explicitly evaluating architectural decomposition addresses a meaningful aspect of software development that final code correctness alone does not capture.

  • Useful data curation infrastructure. The collection of real pull requests across multiple ecosystems, together with documented curation prompts and evaluation schemas, provides a potentially valuable foundation for further research.

Critical weaknesses
  1. The reliability of the ground truth DAGs is insufficiently established.
  • Historical commit boundaries do not necessarily correspond to appropriate architectural subtasks.
  • Hunk overlap and historical co-change frequencies do not necessarily establish required execution dependencies. Assigning direction according to commit chronology does not resolve this ambiguity.
  • The annotation unit and coverage of missing dependencies remain unclear in Appendix D.1, leaving the validity of the benchmark targets uncertain.
  1. The baseline evaluation does not adequately support the broader architectural reasoning claims.
  • The Single-Call and Tri-Agent baselines use supplied candidate paths obtained through BM25 retrieval and golden file injection. They do not evaluate autonomous repository exploration, code inspection, or iterative tool use.
  • The Full Content ablation does not specify whether it includes the same candidate set, potentially containing 150 or more files, or report input lengths, token budgets, and truncation policies.

So the evaluation does not establish performance under an interactive coding agent paradigm or isolate the effect of code content from context length and generation failures.

Questions for authors
  1. Does the confusion matrix in Appendix D.1 count PR pruning decisions, commit pruning decisions, or individual dependency edges?

  2. Did the human audit independently assess dependency necessity, direction, and missing edges?

  3. Please clarify file counts, input token distributions, input and output limits, truncation policies, and how failed or incomplete outputs were scored. Actually, it needs coding agent baselines.

Reviewer_D41bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ArchRSBench, a benchmark designed to isolate and evaluate the system-level architectural decomposition capability of repository-level code generation. The benchmark aims to separate architectural reasoning, such as decomposing a feature into components and determining their relationships.

Critical strengths
  1. Repository-level code generation is an important and practically relevant problem, particularly as coding agents increasingly operate on large and complex software repositories.

2, The motivation of isolating system-level architectural decomposition from code generation is interesting. A dedicated benchmark for architectural reasoning could help better understand where current repository-level coding generation succeed or fail, beyond simply measuring whether the final generated code passes tests.

Critical weaknesses
  1. Relationship to prior repository planning and graph-based approaches. Several related works, such as CodexGraph, RepoGraph, and RPG, use repository structure or code graphs to support repository-level reasoning and to separate planning/repository understanding from code generation. The current paper would benefit from a clearer discussion of how repository topology generation and the proposed feature decomposition task differ conceptually and empirically.

  2. Limitations of deterministic decomposition metrics. The deterministic scores include Smap​, Sord​, and Sgran metrics​. However, architectural decomposition is inherently non-unique. A new feature may admit multiple valid decompositions with different component boundaries, dependency structures, implementation orders, or levels of granularity. Therefore, requiring a predicted decomposition to closely match the ground-truth decomposition may penalize alternative solutions that are architecturally reasonable or even superior. I think the paper needs stronger evidence that these metrics measure decomposition quality rather than similarity to the particular decomposition chosen by the original developers.

  3. Limited evaluation against agentic coding systems. The current evaluation appears to focus primarily on models using a single inference call. However, modern repository-level coding increasingly relies on agentic systems and coding harnesses that iteratively inspect repositories, plan modifications, invoke tools, and refine their solutions. It would substantially strengthen the paper to compare against strong agentic baselines or harnesses where applicable. Such experiments could help determine whether ArchRSBench captures a fundamental limitation of current models or a limitation that can be mitigated through iterative agentic reasoning.

  4. Lack of efficiency and cost analysis for the multi-agent setting. The paper evaluates the performance of the proposed Tri-Agent approach, but the improvement may come with substantially greater inference cost. Since a multi-agent architecture naturally requires additional model calls and context processing, performance should ideally be considered together with computational cost. Reporting token usage, number of model calls, latency, and/or estimated inference cost would provide a more complete picture of the trade-off between decomposition quality and inference overhead.

Questions for authors
  1. How does feature decomposition in ArchRSBench fundamentally differ from repository topology, planning, or graph construction used in prior repository-level approaches such as CodexGraph, RepoGraph, and RPG? What capability does ArchRSBench measure that these approaches do not?

  2. How does the benchmark handle cases in which there are multiple architecturally valid decompositions for the same feature? Could a decomposition receive a low Smap​, Sord, and Sgran​ scores despite being a valid solution? Have the authors conducted human evaluation, or another independent evaluation, to verify that the deterministic scores correlate with human judgments of architectural decomposition quality? Such an analysis would substantially strengthen the validity of the proposed benchmark.

  3. How would strong iterative or agentic coding systems perform on ArchRSBench compared with the single-call baselines? Can the authors include representative coding-agent or repository-planning baselines where feasible?

  4. What is the inference overhead of Tri-Agent relative to the single-call baselines in terms of token consumption, number of model calls, latency, and/or monetary cost?

Reviewer_3YD7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ArchRSBench, a benchmark for evaluating repository-level feature planning before implementation. Models are asked to produce a directed acyclic graph whose nodes contain subtask descriptions and target files, and whose edges represent execution dependencies. The dataset contains 1,525 instances from 13 repositories, with reference graphs reconstructed from historical commits using diff overlap, historical co-change, commit messages, and issue relationships. Evaluation combines four deterministic metrics with an LLM jury assessing corresponding dimensions. The experiments compare five models, a three-stage generation pipeline, and alternative representations of repository context. The reported deterministic score for the strongest single-call baseline is 0.67, increasing to 0.79 with the Tri-Agent pipeline.

Critical strengths
  1. Linking subtask descriptions to actual file changes.

I like the use of commits from merged GitHub PRs to define subtasks. Pairing natural-language commit messages with the files changed in those commits gives concrete examples of how descriptions of development work relate to actual repository changes.

  1. Dataset construction effort.

The authors collected and curated 1,525 instances across 13 repositories, including filtering PRs and commits, extracting file changes, and reconstructing dependency graphs. I appreciate the effort involved in assembling these records into a common format for evaluating repository-level planning.

Critical weaknesses
  1. The scoring can miss clear errors in a plan.

Equations (1)-(4) assess task descriptions, files, and ordering separately, without checking that they agree. For example, suppose task A creates a database schema in schema.sql, task B implements an API in api.py, and A must precede B. A prediction could keep both descriptions, swap their file assignments, and reverse the edge to B - A. The descriptions, file union, and node count remain unchanged. Equation (3) then matches reference A to predicted B and reference B to predicted A based on their files, so it detects no ordering error. Under the stated formulas and intended normalization, this incorrect plan can receive a perfect score.

The order metric can also miss deleted dependencies if the chosen topological ordering still agrees with the reference.

An LLM judge might catch these errors, but that does not make the separately reported mathematical score reliable. I would like to see tests showing that wrong file assignments and missing or unnecessary dependencies actually lower the scores.

  1. It is unclear whether the reference edges represent necessary dependencies.

Section 3 says an edge means one task must precede another. Yet files being changed together in the past, or edits touching nearby lines, does not establish that requirement. Commit order may reflect how a developer happened to organize the work rather than a necessary execution order.

Appendix D.1 describes human checks of which PRs and commits to retain or remove, but it is unclear whether the reviewers also checked whether each dependency was necessary, whether its direction was correct, and whether any dependencies were missing. I would like to see those results, including agreement between annotators and accuracy for each extraction rule. Without this validation, a reasonable alternative plan could be penalized simply for differing from the historical implementation, which makes the benchmark scores difficult to interpret.

Questions for authors

Q1 — Scorer definition and targeted validation.

Does the released evaluator implement additional constraints beyond Equations (1)-(4) that would penalize the twonode counterexample above? In particular, is a shared correspondence used to jointly assess semantics, file assignments, and dependencies, and does the order metric compare prerequisite reachability or only one topological ordering? Please provide the scores for controlled perturbations that swap task-file assignments, remove required edges, or introduce unnecessary serialization while preserving node count. This would help distinguish an incomplete mathematical description from a limitation of the evaluator itself.

Q2 — Scope of the existing human validation.

What is the unit of the confusion matrix in Appendix D.1: a DAG-level filtering decision, a commit-level filtering decision, or an individual dependency edge? Were annotators asked to establish necessary prerequisites directly from the repository and changes, independently of the mined graph? Please report any existing edge-level validation, including how missing dependencies were assessed and whether the results differ across the four extraction rules.

Convincing independent validation would materially affect my assessment of the dataset.

Q3 — Experimental protocol and numerical consistency.

Why does the single-call, paths-only Claude baseline obtain a deterministic score of 0.67 in Tables 2-3 but 0.78 in Table 4? Section 4.3 describes a single inference call, and Appendix J states that the foundational instructions remain the same. Were different instances, candidate files, inference settings, or evaluation procedures used?

Additionally, the Full Content row in Table 4 reports component scores of 0.62, 0.35, 0.60, and 0.48, whose equalweight harmonic mean is approximately 0.487, while the reported overall score is 0.51. Please clarify the aggregation protocol and reconcile these values. A simple change from aggregating component means to averaging per-instance harmonic means would not explain an overall score above this value when the same instances and weights are used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.