- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
To address the lack of benchmarks for evaluating semantic equivalence verification in repository-level coding tasks, this paper introduces RepoEquiv, a benchmark constructed from existing repository-level code generation datasets. Motivated by the high cost of execution-based verification, RepoEquiv aims to facilitate the development of efficient LLM-based semantic verifiers. The authors evaluate several LLMs on RepoEquiv and investigate the effects of fine-tuning, repository context, and confidence-based selective execution. The results demonstrate improvements in verification performance and potential reductions in execution costs.
Critical strengths
- Practical motivation and realistic benchmark construction. The paper addresses an important problem: reducing the high execution costs associated with verifying repository-level code generation. Compared with prior equivalence benchmarks based on standalone functions and synthetic perturbations, RepoEquiv incorporates real-world repositories and candidate implementations generated by diverse coding LLMs, making the evaluation setting more representative of practical coding workflows.
- Comprehensive empirical evaluation with practical implications. The paper evaluates multiple verifier models across general and scientific coding benchmarks, including fine-tuning, calibration analysis, context ablation, and verification cost measurements. The improvements from fine-tuning and the demonstrated cost-accuracy trade-offs of selective execution provide useful empirical evidence for the potential of learned verification.
Critical weaknesses
- Insufficient validation of test-based labels as a proxy for semantic equivalence. The paper defines semantic equivalence as two implementations producing identical outputs for all valid inputs. However, in Section 3.1, equivalence labels are assigned solely based on whether a candidate passes the provided unit tests. Passing a finite test suite does not necessarily imply semantic equivalence, particularly when tests have incomplete coverage. Consequently, RepoEquiv may primarily measure the ability to predict test-based correctness rather than semantic equivalence between implementations.While test-based supervision is a reasonable practical approximation, the paper neither quantifies the potential labeling errors nor investigates their impact on the reported results. This raises concerns about the construct validity of the benchmark, especially given that semantic equivalence is its central research objective.
- Insufficient evidence that the benchmark genuinely requires repository-level semantic reasoning. A key claimed contribution of RepoEquiv is the incorporation of repository-level context and dependencies, distinguishing it from existing function-level equivalence benchmarks. However, the experimental evidence does not sufficiently establish the necessity of repository-level reasoning.As shown in Table 3, the benefit of repository context is mixed across benchmarks and models, with many differences being statistically insignificant. More importantly, the context ablation removes both in-file and cross-file information simultaneously, making it difficult to determine whether the observed improvements arise from understanding repository-level dependencies or simply from additional local code context.Furthermore, the dataset construction does not explicitly ensure that equivalence judgments require cross-file dependency reasoning. As a result, it remains unclear to what extent RepoEquiv introduces a genuinely repository-dependent verification challenge rather than repackaging function-level equivalence checking with additional contextual information. This weakens the empirical support for the paper's central novelty claim.
Questions for authors
- Regarding the reliability of test-based equivalence labels: Could the authors provide additional details about the coverage, comprehensiveness, and reliability of the unit tests used to construct RepoEquiv? Since passing the provided tests is directly treated as semantic equivalence, it is important to justify whether these tests are sufficiently comprehensive to serve as reliable proxies for semantic equivalence, rather than merely indicating correctness on a limited set of test cases.
- Regarding the contribution of repository-level context: Could the authors provide a more fine-grained ablation separating (a) no context, (b) in-file context only, and (c) in-file plus cross-file repository context? This would help clarify whether the observed context gains genuinely originate from cross-file dependency reasoning.
- Regarding the repository-level nature of the benchmark: What proportion of RepoEquiv instances actually require cross-file dependency information to determine equivalence? Could the authors identify a subset of explicitly repository-dependent cases and report verifier performance on this subset separately? Such an analysis would help substantiate the claimed distinction from existing function-level equivalence benchmarks.
Response to the reviewer
We thank the reviewer for recognizing that RepoEquiv addresses the important problem of cost reduction in repository-level code evaluation, the realistic scenarios of repository-level coding tasks that RepoEquiv is aiming at, and the comprehensive empirical evaluation with practical implications that RepoEquiv offers. We address the reviewers' concerns and questions as follows:
Question 1: Please see the response point 1 above.
Question 2: Please see the response point 2 above.
Question 3 (Regarding the repository-level nature of the benchmark): Please see the context-based performance breakdown in the response point 2 above. We additionally provide as follows the proportion of RepoEquiv instances based on their context requirements: