acceptodds
Under review as a conference paper at ICLR 2027

ComBench: A Repository-level Real-world Benchmark for Compilation Error Repair

Abstract

Compilation errors pose pervasive and critical challenges in software development, significantly hindering productivity. Therefore, Automated Compilation Error Repair (ACER) techniques are proposed to mitigate these issues. Despite recent advancements in ACER, its real-world performance remains poorly evaluated. This can be largely attributed to the limitations of existing benchmarks, i.e., decontextualized single-file data, lack of authentic source diversity, and biased local task modeling that ignores crucial repository-level complexities. To bridge this critical gap, we propose ComBench, the first repository-level, reproducible real-world benchmark for C/C++ compilation error repair. ComBench is constructed through a novel, automated framework that systematically mines real-world failures from the GitHub CI histories of large-scale open-source projects. Our framework contributes techniques for the high-precision identification of ground-truth repair patches from complex version histories and a high-fidelity mechanism for reproducing the original, ephemeral build environments. To ensure data quality, all samples in ComBench are execution-verified—guaranteeing reproducible failures and build success with ground-truth patches. Using ComBench, we conduct a comprehensive evaluation of 12 modern LLMs under both direct and agent-based repair settings. Our experiments reveal a significant gap between a model's ability to achieve syntactic correctness (a 73% success rate for GPT-5) and its ability to ensure semantic correctness (only 41% of its patches are valid). We also find that different models exhibit distinct specializations for different error types. ComBench provides a robust and realistic platform to guide the future development of ACER techniques capable of addressing the complexities of modern software development.

Primary area:
datasets and benchmarks
Keywords:
Compilation Error Repair, Automated Debugging, Code Benchmark, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_naovOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • Presents a repository-level C/C++ compilation-repair benchmark mined from CI histories.
  • Provides reproducible environments and evaluates both compilability and semantic correctness.
Critical strengths
  • The benchmark addresses realistic repository context and execution-based validation.
  • The distinction between compiling patches and semantically correct patches is useful.
Critical weaknesses
  • The benchmark is drawn from a limited set of projects, which may restrict diversity.
  • Semantic correctness annotations and their scalability remain important sources of evaluation uncertainty.
Questions for authors
  • How will the benchmark be extended beyond the current project set?
  • Can the semantic-correctness labels be independently audited or updated?
Reviewer_D5XEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ComBench, a repository-level benchmark for evaluating automated compilation error repair (ACER) in real-world C/C++ projects. The authors develop an automated pipeline to extract compilation failures from GitHub Actions logs, identify corresponding developer fixes, and reconstruct reproducible build environments using Docker. The resulting benchmark contains 200 compilation errors across seven categories from four large-scale open-source projects (OpenSSL, Bitcoin, RocksDB, and LLVM). Through an evaluation of 12 LLMs under direct and agent-based repair settings, the authors demonstrate a substantial gap between compilation success and semantic correctness, highlighting the challenges of reliable repository-level compilation error repair.

Critical strengths

Compilation error repair is a practically important software engineering problem.

The authors develop a systematic pipeline for extracting compilation failures from CI logs, identifying developer repair patches, and reconstructing historical build environments.

The paper evaluates 12 LLMs and considers both direct repair and agent-based repair settings.

Critical weaknesses

(1). Limited novelty and insufficient justification for a standalone benchmark. While ComBench provides real-world compilation error repair tasks, its unique value over existing repository-level repair benchmarks (e.g., SWE-bench) is not sufficiently established. The paper primarily compares dataset characteristics with traditional compilation error benchmarks (e.g., DeepFix and TRACER), without systematically investigating whether existing repository-level benchmarks already cover similar tasks or whether ComBench reveals fundamentally different challenges or model limitations.

(2) Limited dataset scale and diversity. ComBench contains only instances from four repositories, with 84.5% originating from LLVM and RocksDB. This highly imbalanced distribution raises concerns about representativeness and potential project-specific biases. Moreover, given the proposed automated collection pipeline, it is unclear why the benchmark is limited to only four repositories and whether the pipeline can effectively scale to more diverse projects.

(3). Insufficient characterization of benchmark difficulty. Although the paper emphasizes repository-level compilation repair challenges, it does not sufficiently characterize the actual complexity of its instances. For example, the reported average of 200+ dependent files does not necessarily imply complex repairs. It remains unclear how many tasks genuinely require cross-file reasoning or coordinated modifications rather than simple local fixes. Furthermore, the primary evaluation provides ground-truth modified files through perfect fault localization, potentially simplifying the task substantially.

(4). Concerns about the validity and reproducibility of Semantic Correctness. Although the authors report high inter-annotator agreement (0.83), it relies primarily on manual inspection without systematic functional testing to validate the absence of behavioral regressions. Moreover, the metric conflates functional correctness with repository convention compliance and uses developer patches as authoritative references, potentially introducing evaluation bias. The reliance on manual annotation also limits the benchmark's reproducibility and scalability for evaluating future repair methods.

Questions for authors

Q1. What unique challenges or evaluation capabilities does ComBench provide beyond existing repository-level repair benchmarks such as SWE-bench? Could the authors quantify how many compilation-related issues are already covered by existing benchmarks (SWE-Bench series)?

Q2. Given the limited number of repositories and the highly imbalanced dataset distribution, how do the authors establish the representativeness of ComBench? What are the main obstacles to scaling the collection pipeline to more projects?

Q3. Could the authors quantify the actual repair complexity of benchmark instances, including patch sizes, numbers of modified files, and the proportion requiring cross-file reasoning rather than simple local modifications?

Q4. How can the authors ensure that manually assessed SC reflects actual functional correctness rather than similarity to developer patches or compliance with repository conventions? Could functional tests or independent expert assessments provide additional validation? How can future researchers reproducibly evaluate new repair methods using SC without repeating expensive manual evaluations?

Reviewer_XRQDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ComBench, a repository-level benchmark for automated compilation error repair in real-world C/C++ projects. It contains 200 execution-verified errors collected from GitHub Actions across four open-source repositories, with developer fixes and reproducible Docker environments. The authors evaluate 12 LLMs under perfect fault localization and two models in an OpenHands agent setting. Results reveal a substantial gap between compilation success and semantic correctness, as well as model-dependent benefits from agent-based repair.

Critical strengths
  1. This paper collects real compilation errors from large open-source repositories, providing a more realistic evaluation setting than existing single-file benchmarks.

  2. This paper reconstructs historical build environments using Docker and validates each instance through a fail–fix–pass process, improving the reliability and reproducibility of the benchmark.

  3. This paper considers compilation success, semantic correctness, and exact match, revealing limitations that compilation success alone cannot capture.

Critical weaknesses
  1. The paper emphasizes repository-level fault localization and cross-file reasoning, but its main evaluation uses Perfect Fault Localization (PFL), which directly provides the files modified by the ground-truth patch. This bypasses a key challenge that the benchmark aims to address.

  2. Although the benchmark is designed for realistic repository-level repair, only two of the 12 models are evaluated using the OpenHands agent framework. This limits the evidence for its effectiveness in end-to-end autonomous repair scenarios.

  3. Semantic correctness relies primarily on manual assessment against developer intent rather than comprehensive functional testing. Although the authors report high inter-annotator agreement, this approach cannot fully guarantee that generated patches preserve program behavior.

Questions for authors
  1. Given that repository-level fault localization is a central motivation of ComBench, could the authors explain why the main evaluation relies on Perfect Fault Localization (PFL), which provides the files modified by the ground-truth patch? Would a broader end-to-end evaluation better demonstrate the benchmark's intended capabilities?

  2. Why are only Claude Sonnet 4 and Qwen3-32B evaluated under the OpenHands framework? Could the authors extend the agent-based evaluation to more models to better support the conclusions about the benefits and limitations of iterative repair?

  3. Since semantic correctness is assessed manually, how do the authors ensure that patches judged as correct do not introduce functional regressions? Could additional functional tests provide stronger evidence beyond inter-annotator agreement?

  4. Could the authors clarify why OpenHands was selected as the agent framework? Since agent performance can be affected by the underlying harness, it would be useful to discuss whether the observed improvements or degradations are specific to OpenHands and whether similar results could be expected with other coding agent frameworks, such as Claude Code or Codex.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.