acceptodds
Desk rejected at ICLR 2027

LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks with large reference implementations. Many benchmarks evaluate coding agents' implementation capability to produce correct code edits from detailed specifications. However, practical modular development tasks also require the perception capability of grounding user intent and high-level design to derive a specification. We introduce LoLBench to evaluate both capabilities through the entire proposal-to-implementation process on large software systems. It is a multilingual benchmark of 100 tasks across 29 software systems in five domains. Each task provides a human-written enhancement proposal with user intent and high-level design. On average, proposals contain about 5,000 words, software systems contain 2.4 million source lines of code (LoC), and implementation pull requests (PRs) change approximately 5,500 LoC. Across 28 agents we evaluated, the best agent resolves only 14% of tasks and achieves a 52.7% Fail-to-Pass (F2P) pass rate. Failure analysis identifies incomplete code localization as a major bottleneck, while providing reference-derived file trees alongside API specifications improves resolved rates by 16–22 percentage points (2.4–17), reaching at most 34%. These results show that both perception and implementation remain central challenges for coding agents in practical modular development on large software systems.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Coding Agents, Long Horizon Tasks, Modular Development, Software System, Code Generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_onEcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a benchmark to evaluate coding agents throughout the proposal-to-implementation process on large software systems. The authors design a data pipeline to collect high-quality modular development tasks, which involves enhanced proposals, matched PRs and augmented test suits. Extensive evaluations are performed on multiple combinations of models and harnesses.

Critical strengths
  1. The benchmark is challenging in terms of solution size.
  2. Extensive experiments are performed with different combination of models and harnesses.
Critical weaknesses
  1. For the verification of task completion, it is proposed to use the so-called system level public interfaces for F2P tests. But how can such tests be identified and ensured with completeness or sufficiency? The supp describes some enhancement of F2P tests, but it is still not clear why the agent could give a correct testable implementation if not given the exact interfaces. Besides, the completeness and accuracy of tests are not validated. This is particularly important for large-scale coding tasks that involve many modules and span over 2k LoC. It is shown that agents achieve large improvements when given specification. This also raise questions whether previous failures are mainly due to lack of clear interface defintions rather than drawbacks in model capabilities.
  2. It is unclear what are the conclusions of the assessment. The section 4.2~4.6 provide very limited information. Many paragraphs emphasies the same thing that localization is more of the bottleneck, which is already in wide agreement. The text describes the phenomenon and numbers but give little in-depth analysis and insights. Reading the experimental section makes the reviewer feel confused. What are the drawbacks of model capabilities? How can the assessment results guide the future research on enhancing the model?
  3. Can the data pipeline be used for scalable task generation so that one can use it for SFT/RL data production?
  4. Are the tasks verified by human experts to ensure their quality and solvability?
Questions for authors

Were the result numbers obtained with multiple runs for each model-harness combination?

Reviewer_xLW9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LoLBench, a benchmark testing whether coding agents can turn high-level feature proposals into working changes in large, existing software systems. Its central idea is that successful development requires both perception—understanding user intent, navigating the codebase, and identifying all affected components—and implementation—writing correct changes without breaking existing behavior. The benchmark contains 100 tasks across 29 projects, with reference implementations changing about 5,500 lines on average, and evaluates functionality alongside regression tests. Across 28 agent configurations, the best fully solved only 14% of tasks; a major failure was missing relevant code and dependencies across modules. Providing file trees and API specifications derived from reference solutions increased success by 16–22 percentage points, reaching 34%. The main takeaway is that improving agents’ ability to translate broad goals into complete, concrete implementation plans is essential for reliable software development.

Critical strengths

A key strength of LoLBench is that it evaluates a realistic and demanding part of software development: turning high-level intent into coordinated changes within an existing system. Unlike benchmarks that provide detailed specifications or narrowly scoped bug reports, it starts with human-written enhancement proposals, requiring agents to interpret requirements, understand architecture, and identify dependencies across modules. Its 100 tasks span 29 software systems across five domains, with proposals averaging roughly 5,000 words, codebases averaging 2.4 million source lines, and reference changes averaging 5,500 lines. This scale makes the distinction between perception (determining what needs to chang) and implementation (making those changes correctly) particularly useful for understanding agent capabilities.

Another strength is its combination of behavioral evaluation, partial-progress measurement, and diagnostic experiments. System-level tests exercise public interfaces, allowing valid solutions to differ structurally from the reference implementation, while regression tests check whether existing behavior remains intact. The authors strengthen test coverage through deliberately incorrect solution variants and coverage-guided test generation, then map tests to proposal sections to measure progress beyond an overall pass or fail. Their analysis identifies incomplete cross-module localization as a major failure mode, and a targeted experiment provides supporting evidence: adding reference-derived file trees and API specifications improves task resolution by 16–22 percentage points across three agents, reaching 34%. This makes the benchmark useful for identifying concrete research priorities, especially repository understanding and comprehensive change planning, alongside code generation

Critical weaknesses

The main weakness is that the evidence does not cleanly isolate “perception” from implementation difficulty. Providing reference-derived file trees and API specifications improves results, but these hints also reveal architectural decisions and reduce the solution space. The improvement therefore supports the value of detailed guidance more directly than it proves that perception is the dominant bottleneck. Similarly, measuring localization against files and functions changed by the reference implementation can undervalue alternative valid designs. The failure analysis offers useful diagnostic evidence, but reference-based comparisons and LLM judgments leave uncertainty about the underlying causes of failure.

A second weakness is limited generalizability and statistical reliability. The benchmark selects large, completed open-source changes linked to a single merged PR with existing tests, favoring unusually well-documented, testable work over the ambiguity, collaboration, and evolving requirements common in practice. Tests and mutations built around reference solutions may also miss requirements or unintentionally favor particular implementation choices, despite using public interfaces. Finally, each agent receives only one attempt per task, so the reported results do not establish how stable performance is across repeated runs. With 100 tasks, a one-percentage-point difference represents just one solved task, making small ranking differences difficult to interpret confidently. These limitations qualify the paper’s broader conclusions without diminishing its value as a challenging benchmark.

Questions for authors
  1. How can you isolate perception difficulty from implementation difficulty? Could you separately test file-location hints and API specifications to determine which drives the improvement?

  2. How do you ensure evaluation accepts alternative valid implementations? Have independent developers checked whether the tests and reference-based localization metrics unfairly penalize solutions with different architectures?

  3. How stable are the reported results? Would repeated runs and confidence intervals change agent rankings or the conclusion that incomplete code localization is the dominant bottleneck?

Reviewer_A1zFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LoLBench, a benchmark covering 100 coding tasks across 29 large open-source software systems, five domains and eight programming languages. The benchmark is designed to evaluate coding agents on two dimensions of difficulty: implementation complexity, which is ensured due to the large size of required updates, and perception complexity, which is the difficulty of grounding user intent and high-level design, which is realized by providing EPs to the agents. The benchmark evaluates 28 agents (six models and five scaffolds) and reports the number of tasks resolved by determining software tests' pass-to-pass and fail-to-pass rates, which are extracted from the original software system and PRs implementing the EPs, which were augmented with LLM-generated tests to increase coverage.

On the benchmark, Claude Opus 5 with Claude Code or mini-SWE-agent achieves the highest score of 14% of tasks resolved, and the main identified failure mode is incomplete code localization by the agent. The paper tests the effect of additionally providing the file tree and API specifications derived from the reference PR implementing the EP to the agent, which increases the resolved rate by 16-22 percentage points to at most 34%.

Critical strengths
  1. The benchmark tests for implementation complexity, measured by the size of code changes required to implement the EPs (in LoC and number of files, classes, and functions changed). The covered repositories average 2.4M LoC, and the reference PRs change about 5.5k LoC across 65 files, which as shown in Table 1 is larger than prior benchmarks. The addition of systematically LLM-generated tests increases the test coverage and the mutation score from 61% to 96% and the F2P line coverage from 60% to 78.5% (Table 9).

  2. The perception complexity is an important and underexplored aspect of coding agent benchmarks. Most prior benchmarks specify implementation tasks from a detailed specification (e.g., a narrowly scoped bug report). LoLBench starts from design-level proposals written before the implementing PR, so agents must make high-level decisions themselves about which modules and interfaces to touch before writing code.

Critical weaknesses
  1. The central claim about perception complexity isn't directly supported. The paper's main conceptual contribution is the distinction between perception and implementation complexity (Figure 1), but perception complexity is never measured directly. All evidence for perception complexity is indirect:
  • The share of target class names mentioned in the EP (7.3% in Table 1) shows that guidance is missing, but doesn't measure how hard it is to ground the intent.
  • The specification experiment (Table 4) adds a file tree and API specifications derived from the reference solution. This can also lower implementation difficulty and can reveal what parts of the code are tested, so the 16-22 point gain cannot be credited to perception alone.

Additionally, the perception complexity and implementation complexity are confounded in the benchmark, as every LoLBench task is high on both axes, so failures cannot be attributed to one or the other. A per-task analysis relating success to perception proxies (such as EP length, number of implementable sections) versus implementation proxies (such as PR size) would make the claim more convincing.

  1. The results lack uncertainty estimates. Each model–scaffold combination attempts each task only once (acknowledged in Appendix G), and no confidence intervals or significance tests are reported. With 100 tasks, a 14% resolved rate has a 95% confidence interval of roughly 8.5-22%, following the Wilson score method for n=100n = 100 and p=14%p = 14\%. Many reported differences are therefore within noise, for example Opus 5 across agent scaffolds (10-14%) or the ordering of the remaining agents (0-5%).
Questions for authors
  1. What evidence distinguishes perception difficulty from implementation difficulty in the localization analysis and specification experiment?
  2. Beyond removing test-plan and validation sections (Appendix H), were EPs modified, e.g., by adding interface details? How many were changed, by whom (human or LLM), and was the 7.3% in Table 1 computed on the original or the modified EPs?
  3. All EPs and PRs are public, and many predate the evaluated models’ training cutoffs (PEP 617 is from 2020). Do results differ for tasks whose PRs were merged after each model’s cutoff?
Reviewer_navBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LoLBench evaluates coding agents on substantial changes to existing software systems. Its 100 tasks use community-written enhancement proposals across 29 projects and eight languages. Agents must translate these proposals into implementations within offline environments. Public-interface behavior tests, regression tests, and section-level scores measure their progress. Mutation-guided augmentation strengthens the test suites. Across 28 model-scaffold combinations, the best resolved rate is 14%. Reference-derived file and API guidance raises this to at most 34%. The study highlights the difficulty of connecting broad requirements to large codebases.

Critical strengths
  1. The benchmark addresses a useful gap between high-level proposals and working implementations. Its tasks span large repositories and substantial cross-module changes. Community-written proposals make the setting relevant to practical feature development (Sections 3.1 and 3.5).

  2. The evaluation deliberately supports public behavior rather than reference-specific private functions. Regression checks and mutation-guided augmentation strengthen the verifier. The added tests raise retained-mutant detection from 61.08% to 95.97%, providing concrete evidence of improved test coverage.

  3. The evaluation covers 28 model-scaffold combinations and reports quality, partial progress, and cost. Providing file and API guidance improves resolution by 16-22 percentage points. This intervention gives useful evidence beyond a difficult leaderboard (Tables 2 and 4).

Critical weaknesses
  1. The localization analysis uses reference-edit overlap as a proxy for missing context. A different implementation strategy may leave some reference files untouched. The paper acknowledges diagnostic uncertainty, but the link between these omitted locations and actual failed requirements needs clearer evidence.

  2. The Completed metric needs a clearer account of its denominator. The construction allows unmapped implementable sections, and supplied proposals may omit original sections. Their treatment in the score is not explicit. This is a reproducibility question, rather than evidence of incorrect results.

Questions for authors
  1. Section 4.4 attributes most localization failures to partial coverage of reference files. How do the rules distinguish necessary missing context from a different implementation strategy? Please provide one existing audited case linking the omitted context to a failed requirement. This would strengthen the interpretation of the attribution beyond reference overlap.

  2. Your definition of perception includes deriving a detailed implementation specification. For Table 4, could you describe one existing example of the added file and API guidance? Please identify which details were explicit, inferable, or unspecified in the original task input. This would clarify the information supplied by the intervention.

  3. Section 4.1 defines Completed through tests mapped to implementable sections. Does every scored section have at least one F2P test? Please clarify how unmapped sections and any sections removed from the supplied proposal enter the denominator. This would make the partial-progress score easier to interpret and reproduce.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.