(1). Limited novelty and insufficient justification for a standalone benchmark. While ComBench provides real-world compilation error repair tasks, its unique value over existing repository-level repair benchmarks (e.g., SWE-bench) is not sufficiently established. The paper primarily compares dataset characteristics with traditional compilation error benchmarks (e.g., DeepFix and TRACER), without systematically investigating whether existing repository-level benchmarks already cover similar tasks or whether ComBench reveals fundamentally different challenges or model limitations.
(2) Limited dataset scale and diversity. ComBench contains only instances from four repositories, with 84.5% originating from LLVM and RocksDB. This highly imbalanced distribution raises concerns about representativeness and potential project-specific biases. Moreover, given the proposed automated collection pipeline, it is unclear why the benchmark is limited to only four repositories and whether the pipeline can effectively scale to more diverse projects.
(3). Insufficient characterization of benchmark difficulty. Although the paper emphasizes repository-level compilation repair challenges, it does not sufficiently characterize the actual complexity of its instances. For example, the reported average of 200+ dependent files does not necessarily imply complex repairs. It remains unclear how many tasks genuinely require cross-file reasoning or coordinated modifications rather than simple local fixes. Furthermore, the primary evaluation provides ground-truth modified files through perfect fault localization, potentially simplifying the task substantially.
(4). Concerns about the validity and reproducibility of Semantic Correctness. Although the authors report high inter-annotator agreement (0.83), it relies primarily on manual inspection without systematic functional testing to validate the absence of behavioral regressions. Moreover, the metric conflates functional correctness with repository convention compliance and uses developer patches as authoritative references, potentially introducing evaluation bias. The reliance on manual annotation also limits the benchmark's reproducibility and scalability for evaluating future repair methods.