acceptodds
Under review as a conference paper at ICLR 2027

RepoEquiv: Benchmarking Large Language Models' Understanding for Repository-Level Semantic Equivalence

Abstract

Large language models (LLMs) for code and agentic coding systems are rapidly improving at complex repository-level programming tasks. However, training and evaluating these systems largely rely on execution-based signals from unit tests, which are costly to obtain and difficult to scale across web-scale repositories and heterogeneous execution environments. We advocate a complementary approach: learning an LLM-based verifier for semantic equivalence that determines whether two pieces of code produce identical outputs for all valid inputs, thereby reducing the need for explicit execution. We introduce RepoEquiv, a benchmark and dataset suite for repository-level semantic equivalence constructed from repository-level code infilling datasets spanning both general-purpose and scientific programming domains. Unlike prior benchmarks focused on standalone functions with synthetic perturbations, RepoEquiv features repository-level context and dependencies, as well as realistic candidate code snippets generated by diverse coding LLMs. Evaluating 5 representative LLMs on RepoEquiv shows that semantic equivalence verification in repository contexts remains challenging for off-the-shelf models. Fine-tuning on RepoEquiv substantially improves verifier performance on general coding repositories, demonstrating the feasibility of learned semantic verifiers for scalable, execution-free evaluation, and yields uniform gains across all four scientific coding benchmarks, indicating promising out-of-domain generalization for learned verifiers. We further conduct calibration analysis and find that fine-tuning reduces calibration error, and we measure the resulting cost-accuracy trade-off directly: routing low-confidence cases to execution reaches 95% accuracy on the scientific benchmarks while executing 58% of cases, against 80% under confidence-agnostic routing. RepoEquiv establishes a foundation for developing learned semantic verifiers that can significantly reduce the cost of large-scale evaluation and learning for repository-level code generation.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Datasets and Benchmarks, Code Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kWszOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the lack of benchmarks for evaluating semantic equivalence verification in repository-level coding tasks, this paper introduces RepoEquiv, a benchmark constructed from existing repository-level code generation datasets. Motivated by the high cost of execution-based verification, RepoEquiv aims to facilitate the development of efficient LLM-based semantic verifiers. The authors evaluate several LLMs on RepoEquiv and investigate the effects of fine-tuning, repository context, and confidence-based selective execution. The results demonstrate improvements in verification performance and potential reductions in execution costs.

Critical strengths
  1. Practical motivation and realistic benchmark construction. The paper addresses an important problem: reducing the high execution costs associated with verifying repository-level code generation. Compared with prior equivalence benchmarks based on standalone functions and synthetic perturbations, RepoEquiv incorporates real-world repositories and candidate implementations generated by diverse coding LLMs, making the evaluation setting more representative of practical coding workflows.
  2. Comprehensive empirical evaluation with practical implications. The paper evaluates multiple verifier models across general and scientific coding benchmarks, including fine-tuning, calibration analysis, context ablation, and verification cost measurements. The improvements from fine-tuning and the demonstrated cost-accuracy trade-offs of selective execution provide useful empirical evidence for the potential of learned verification.
Critical weaknesses
  1. Insufficient validation of test-based labels as a proxy for semantic equivalence. The paper defines semantic equivalence as two implementations producing identical outputs for all valid inputs. However, in Section 3.1, equivalence labels are assigned solely based on whether a candidate passes the provided unit tests. Passing a finite test suite does not necessarily imply semantic equivalence, particularly when tests have incomplete coverage. Consequently, RepoEquiv may primarily measure the ability to predict test-based correctness rather than semantic equivalence between implementations.While test-based supervision is a reasonable practical approximation, the paper neither quantifies the potential labeling errors nor investigates their impact on the reported results. This raises concerns about the construct validity of the benchmark, especially given that semantic equivalence is its central research objective.
  2. Insufficient evidence that the benchmark genuinely requires repository-level semantic reasoning. A key claimed contribution of RepoEquiv is the incorporation of repository-level context and dependencies, distinguishing it from existing function-level equivalence benchmarks. However, the experimental evidence does not sufficiently establish the necessity of repository-level reasoning.As shown in Table 3, the benefit of repository context is mixed across benchmarks and models, with many differences being statistically insignificant. More importantly, the context ablation removes both in-file and cross-file information simultaneously, making it difficult to determine whether the observed improvements arise from understanding repository-level dependencies or simply from additional local code context.Furthermore, the dataset construction does not explicitly ensure that equivalence judgments require cross-file dependency reasoning. As a result, it remains unclear to what extent RepoEquiv introduces a genuinely repository-dependent verification challenge rather than repackaging function-level equivalence checking with additional contextual information. This weakens the empirical support for the paper's central novelty claim.
Questions for authors
  1. Regarding the reliability of test-based equivalence labels: Could the authors provide additional details about the coverage, comprehensiveness, and reliability of the unit tests used to construct RepoEquiv? Since passing the provided tests is directly treated as semantic equivalence, it is important to justify whether these tests are sufficiently comprehensive to serve as reliable proxies for semantic equivalence, rather than merely indicating correctness on a limited set of test cases.
  2. Regarding the contribution of repository-level context: Could the authors provide a more fine-grained ablation separating (a) no context, (b) in-file context only, and (c) in-file plus cross-file repository context? This would help clarify whether the observed context gains genuinely originate from cross-file dependency reasoning.
  3. Regarding the repository-level nature of the benchmark: What proportion of RepoEquiv instances actually require cross-file dependency information to determine equivalence? Could the authors identify a subset of explicitly repository-dependent cases and report verifier performance on this subset separately? Such an analysis would help substantiate the claimed distinction from existing function-level equivalence benchmarks.
1 comment
AuthorsOpenReview ↗

Response to the reviewer

We thank the reviewer for recognizing that RepoEquiv addresses the important problem of cost reduction in repository-level code evaluation, the realistic scenarios of repository-level coding tasks that RepoEquiv is aiming at, and the comprehensive empirical evaluation with practical implications that RepoEquiv offers. We address the reviewers' concerns and questions as follows:

  1. Insufficient validation of test-based labels as a proxy for semantic equivalence: We agree that passing a finite test suite does not necessarily imply semantic equivalence, and this gap would affect the reliability of the derived equivalence labels. Similarly to our answer to reviewers KY4d and Qxdu, we address this concern empirically by conducting a coverage analysis and augmenting test cases across both general and scientific domains. In particular, we examine execution branches not covered by the original test cases in each benchmark/dataset, augment them with additional tests, and validate all test cases against the reference implementation with repeated execution. As shown in the results in table 2 of the response to all the reviewers, our test case enhancement substantially boosts the test completeness and fixes a large number of label errors. In the meantime, even with the improved test and label quality, learning from RepoEquiv can only establish a proxy for universal equivalence. However, a learned equivalence proxy is still a strong alternative/complement to execution-based verification, as the latter is confined by the same finite set of test cases.
  2. Insufficient evidence that the benchmark genuinely requires repository-level semantic reasoning: As we demonstrated with the new experiments in the response to all the reviewers (Table 1), our new experiments provide further breakdown of the effect of supporting context information on the verifiers' judgments across tasks that require in-file or cross-file context. The results show nearly uniform gains of performance when context information is provided, indicating the critical role of context in the repository-level semantic equivalence judgment enabled by RepoEquiv.

Question 1: Please see the response point 1 above.

Question 2: Please see the response point 2 above.

Question 3 (Regarding the repository-level nature of the benchmark): Please see the context-based performance breakdown in the response point 2 above. We additionally provide as follows the proportion of RepoEquiv instances based on their context requirements:

Dependency categorypairs (%)
Pair-local4,513 (50.07%)
In-file2,437 (27.04%)
Cross-file899 (9.97%)
Reviewer_QxduOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RepoEquiv repurposes repository-level Python function-infilling benchmarks into reference-based verification tasks. Candidate functions generated by several LLMs are paired with reference implementations and repository context, and labeled using execution outcomes. A verifier is fine-tuned on general coding repositories and evaluated on both held-out general repositories and scientific coding datasets. Experiments examine macro-F1, context ablations, calibration, and selective execution, with wall-clock comparisons between learned verification and test execution.

Critical strengths
  1. The benchmark studies a practically relevant verification setting with realistic LLM-generated candidates and repository context. Repository-disjoint general-domain splits, heterogeneous candidate generators, exact-duplicate removal, and diversity/difficulty-aware selection are useful safeguards against an overly narrow collection of trivial code pairs. Repeating execution three times with unanimity and excluding instances whose reference fails the tests also helps control unstable execution labels. These design choices make the dataset a useful starting point for studying reference-based verification of repository-conditioned function infilling.

  2. The experiments connect verifier quality to potential practical use rather than reporting classification accuracy alone. Fine-tuning improves macro-F1 from 67.7 to 78.7 on RepoST and from 59.8 to 66.6 on DevEval, with accompanying context ablations and calibration analyses. The selective-execution curves compare confidence-based deferral with random and oracle policies, providing a concrete way to evaluate the utility of confidence. The runtime study also identifies both favorable settings and the RepoST exception where direct execution is faster. Acknowledging that exception and discussing lifecycle costs helps define where learned verification may be worthwhile.

Critical weaknesses
  1. The reliability gap between the intended verification target and the available supervision remains unmeasured. The abstract and main motivation describe identical behavior over all valid inputs, whereas positive labels certify passing the supplied tests. Appendix A acknowledges this limitation, but does not quantify residual discrepancies. Three-run unanimity reduces execution instability; it does not establish test completeness. Consequently, both verifier accuracy and the hybrid policy's reported 95% accuracy are measured against test-derived labels, and do not reveal how often an accepted candidate has an untested behavioral difference. This is particularly relevant when confident predictions are used to skip execution or are proposed as future training signals. Test-based supervision is a reasonable practical approach, but the current evidence does not yet characterize the risk associated with that intended use. Revising terminology alone would not resolve the evidence gap.

  2. The strength and scope of scientific-domain transfer need a more careful uncertainty analysis. The positive direction of all four scientific results is encouraging, but Table 2 describes p-values of 0.152, 0.220, and 0.422 as mildly significant, which is not supported by conventional significance thresholds. In addition, candidates share masked functions and repositories, whereas the stated paired bootstrap samples individual examples. This matters for small splits such as LMR-Bench's 66 candidates from 13 functions and AutoExperiment's 134 candidates from 42 functions in two repositories. Pooled scientific calibration and routing results are useful, but can also conceal differences across these domains. The current results support promising directional transfer; stronger claims of reliable cross-domain verification require clearer evidence about dependence, per-domain uncertainty, and the behavior of confident errors.

Questions for authors
  1. Label reliability. Could you provide a small, explicitly sampled audit of test-passing candidates, including high-confidence verifier acceptances, for plausible behavioral differences not covered by the original tests? Please report sample size, selection procedure, and the types of checks used, distinguishing confirmed discrepancies from inconclusive cases. A bounded manual or targeted differential check would be useful; I am not asking for a formal equivalence proof, exhaustive testing, or a new benchmark. Please also make the test-based target explicit when discussing the hybrid accuracy.

  2. Grouped uncertainty. Could you recompute the base-versus-SFT differences by resampling masked functions with all their candidate groups, and assess repository-level variability where the number of repositories permits it? For very small repository counts, please state what can and cannot be inferred rather than relying on an unstable interval. Identify which transfer conclusions remain supported, correct the significance wording, and reconcile the 10,000 versus 20,000 bootstrap counts. This requires reanalysis of existing predictions, not new model training.

  3. What repository context contributes. Table 3 shows both helpful and harmful effects of context, and the manuscript already acknowledges this mixed picture. Could you give a few matched examples where cross-file information changes a correct or incorrect verification decision, and clarify how dependency context is selected and truncated? This would help distinguish actual use of repository dependencies from effects of additional prompt length. The request is for explanation and representative examples, not a new retrieval system or a larger ablation grid.

  4. Candidate difficulty and confident errors. Since the construction separates easy/hard positives and negatives, could you report the existing base/SFT performance by these groups, together with false-positive and false-negative rates among confidently retained cases? A brief generator-wise breakdown, where sample sizes permit, would also indicate whether the gains extend across candidate styles. This would make the benchmark's difficulty and the practical risk of skipping execution easier to understand than macro-F1 or pooled calibration alone.

  5. Actual cost of hybrid routing. The reported 58% executed fraction counts cases, while Table 4 shows large variation in per-case execution time across benchmarks. Could you estimate the hybrid policy's end-to-end verification time at the stated accuracy target using the existing timing data, including verifier calls for all cases and execution for deferred cases? Please distinguish per-case medians, measured total time, and any approximate estimate. A per-domain breakdown would help show whether case-count savings translate into time savings; a new deployment-scale experiment is not required.

2 comments
AuthorsOpenReview ↗

Response to the reviewer

We appreciate the reviewer for recognizing the practical relevance of developing equivalence verifiers enabled by RepoEquiv, our rigorous data curation pipeline, the empirical performance gain through training on RepoEquiv, as well as the comprehensiveness of the analysis and ablation. We address the reviewers' concerns and questions as follows:

  1. The reliability gap between the intended verification target and the available supervision: We agree that passing finite tests is not a guarantee of semantic equivalence, and this gap would affect the reliability of the derived equivalence labels. Similarly to our answer to reviewer KY4d, we address this concern empirically by conducting a coverage analysis and augmenting test cases across both general and scientific domains. In particular, we examine execution branches not covered by the original test cases in each benchmark/dataset, augment them with additional tests, and validate all test cases against the reference implementation with repeated execution. And as the results shown in Table 2 in the response to all the reviewers, our test case enhancement substantially boosts the test completeness and fixes a large number of label errors. In the meantime, even with the improved test and label quality, learning from RepoEquiv can only establish a proxy for universal equivalence. However, a learned equivalence proxy is still a strong alternative/complement to execution-based verification, as the latter is confined by the same finite set of test cases.

  2. Uncertainty analysis of scientific-domain transfer: We agree that the description of "mildly significant" is questionable for p-values of 0.422 and will update the manuscript for more rigorous significance wording. We additionally provide the bootstrap analysis using 20,000 paired bootstrap resamples of masked functions, retaining all candidates belonging to each sampled function:

BenchmarkCandidates / functions / repositoriesSFT − base macro-F1Function-group 95% CIRepository-group 95% CI
LMR-Bench66 / 13 / 13+6.21[−2.04, 14.75][−1.97, 14.72]
SciReplicate231 / 66 / 28+4.65[−2.95, 12.77][−0.89, 10.40]
AutoExperiment134 / 42 / 2+10.42[2.22, 19.84]Not estimated
ResearchCodeBench590 / 173 / 18+1.42[−1.98, 4.91][−1.05, 4.11]
AuthorsOpenReview ↗

Response to the reviewer (continued)

Question 1 (Label reliability): Please see the response in 1. Above.

Question 2 (Grouped uncertainty): We provide the grouped uncertainty analysis as follows:

BenchmarkFunctions / repositoriesSFT − baseFunction-group 95% CIRepository-group 95% CI
RepoST246 / 83+10.99[6.66, 15.32][5.35, 17.09]
DevEval285 / 24+6.72[3.70, 9.83][4.66, 9.06]
LMR-Bench13 / 13+6.21[−2.04, 14.75][−1.97, 14.72]
SciReplicate66 / 28+4.65[−2.95, 12.77][−0.89, 10.40]
AutoExperiment42 / 2+10.42[2.22, 19.84]Not estimated
ResearchCodeBench173 / 18+1.42[−1.98, 4.91][−1.05, 4.11]

Question 3 (What repository context contributes): As we demonstrated with the new experiments in the response to all the reviewers (Table 1), our new experiments provide further breakdown of the effect of supporting context information on the verifiers' judgments across tasks that require in-file or cross-file context. The results show nearly uniform gains of performance when context information is provided, indicating the critical role of context in the repository-level semantic equivalence judgment enabled by RepoEquiv.

Question 4 (Candidate difficulty and confident errors): We provide further analysis of the difficulty groups as follows:

BenchmarkEasy positiveHard positiveHard negativeEasy negative
RepoST76.3 → 94.7 (38)54.0 → 82.1 (224)86.7 → 86.7 (30)77.2 → 76.0 (425)
DevEval52.7 → 70.6 (201)32.7 → 57.0 (223)60.6 → 58.8 (279)88.8 → 78.6 (402)
LMR-Bench47.4 → 47.4 (19)14.3 → 14.3 (7)55.6 → 72.2 (18)54.5 → 63.6 (22)
SciReplicate76.9 → 84.6 (13)61.3 → 77.4 (31)52.2 → 73.9 (23)87.2 → 84.1 (164)
AutoExperiment26.7 → 53.3 (30)34.6 → 38.5 (26)58.6 → 72.4 (29)87.8 → 83.7 (49)
ResearchCodeBench55.7 → 67.8 (115)52.7 → 58.8 (131)80.7 → 73.9 (88)89.1 → 83.6 (256)

Question 5 (Actual cost of hybrid routing): We provide an estimate of the total cost as the combination of verifier cost and the deferred execution cost as follows:

DomainExecutedTest-label agreementExecution-only estimate (s)Verifier for all cases (s)Hybrid estimate (s)Saving
RepoST308/71795.12%353.082.2316.210.4%
DevEval788/1,10595.02%4,879.5416.63,932.919.4%
Pooled general coding1,096/1,82295.06%5,232.5498.84,249.018.8%
LMR-Bench44/6693.94%412.971.5380.67.8%
SciReplicate115/23197.84%18,351.568.68,810.552.0%
AutoExperiment78/13490.30%10,478.331.97,306.330.3%
ResearchCodeBench354/59095.08%2,134.8963.02,282.3−6.9%
Pooled scientific591/1,02195.00%31,377.51,135.118,779.640.1%

Across all the benchmarks, the average estimated hybrid running time costs are reduced by 40.1% on scientific benchmarks and 18.8% on general coding benchmarks. On all except for ResearchCodeBench, hybrid verification achieves substantial time savings.

Reviewer_KY4dOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RepoEquiv, a benchmark for evaluating LLMs on repository-level semantic equivalence verification. The benchmark is constructed from six existing code generation datasets, by incorporating repository context and LLM-generated candidate implementations, with equivalence labels derived from unit test results. The authors evaluate several LLMs on RepoEquiv and fine-tune Qwen2.5-Coder-7B to improve verification performance. They further investigate the effects of repository context, confidence calibration, and the cost-accuracy trade-off of combining learned verification with selective execution.

Critical strengths
  1. The paper explores an interesting direction of using learned semantic verifiers to reduce the cost of execution-based code evaluation.

  2. The evaluation covers multiple coding domains and includes analyses of repository context, confidence calibration, and verification efficiency.

Critical weaknesses
  1. Motivation, Novelty, and Necessity over Existing Benchmarks: It's unclear why RepoEquiv is necessary given existing semantic equivalence benchmarks such as EquiBench. The two main distinctions claimed by RepoEquiv are repository-level context and LLM-generated candidate implementations. However, simply providing repository context does not indicate that equivalence verification requires reasoning about cross-file dependencies. Although the context ablation shows performance improvements in some settings, its overall results are mixed and cannot reflect how much of RepoEquiv actually requires repository-level semantic reasoning beyond what existing function-level benchmarks evaluate. Similarly, using LLM-generated candidates may improve the diversity of the benchmark, but it does not necessarily introduce new semantic reasoning or equivalence verification challenges. For example, EquiBench already includes independently written implementations that differ substantially in their algorithms. It is unclear what limitations that existing benchmarks fail to capture are exposed by RepoEquiv.

  2. Validity of Semantic Equivalence Labels: The paper defines semantic equivalence as two implementations producing identical outputs for all valid inputs. However, it constructs equivalence labels solely from unit test results, treating candidates that pass the provided tests as semantically equivalent to the reference. Since passing a finite set of tests cannot guarantee identical behavior on all valid inputs, this labeling strategy may incorrectly classify non-equivalent implementations as equivalent. The paper does not quantify such labeling errors or provide independent validation of the ground truth. As a result, the reported performance only demonstrates the models' ability to predict test-based labels, leaving their actual capabilities of semantic equivalence reasoning unverified.

  3. Dependence on Ground-truth Implementations: The proposed verifier requires both the generated candidate and a ground-truth implementation as inputs. This assumption holds for the function-infilling tasks used to construct RepoEquiv, since the original implementations are readily available. However, in many real-world repository-level development tasks, e.g., implementing new features or fixing unresolved bugs, correct reference implementations are unavailable, such that the proposed verifier cannot be directly applied to these scenarios. Given that the paper motivates its approach as a scalable correctness signal for repository-level code generation, the authors need to justify how this requirement aligns with the intended applications.

Questions for authors
  1. What specific semantic equivalence challenges does RepoEquiv capture that existing benchmarks such as EquiBench do not, particularly regarding cross-file dependencies and LLM-generated implementations?

  2. How reliable are the equivalence labels derived from unit tests, and to what extent could incorrectly labeled equivalent implementations affect the validity of the benchmark and its evaluation results?

  3. In which realistic repository-level code generation scenarios are ground-truth implementations available, and how will the proposed verification approach apply to tasks where such implementations are unavailable?

1 comment
AuthorsOpenReview ↗

Response to the reviewer

We thank the reviewer for recognizing the value of learned semantic verifiers for cost reduction in code evaluation, as well as the comprehensiveness of the domain and the analysis of RepoEquiv. We address your concerns and questions as follows:

  1. Motivation, Novelty, and Necessity over Existing Benchmarks: As shown in Table 1 of the response to all the authors above, our new experiments provide further breakdown of the effect of supporting context information on the verifiers' judgments across tasks that require in-file or cross-file context. The results show nearly uniform gains of performance when context information is provided, indicating the critical role of context in the repository-level semantic equivalence judgment enabled by RepoEquiv. Compared with EquiBench, RepoEquiv features a different emphasis on diversity with respect to different downstream use cases. Aiming mostly at evaluating LLMs’ understanding of semantic equivalence, EquiBench generates its candidates through pre-defined sets of transformations, which may not reflect the candidates distribution generated from LLMs. On the other hand, to foster the development of equivalence verifiers as reward models/evaluators for LLM-generated code, RepoEquiv highlights candidate source diversity across 5 LLMs with difficulty controls, as well as repository and application diversity covering general software and scientific implementations where candidates interact with repo-specific helpers and conventions.

  2. Validity of Semantic Equivalence Labels: We agree that passing finite tests is not a guarantee of semantic equivalence. To address this concern empirically, we conduct a coverage analysis and augment test cases across both general and scientific domains. In particular, we examine execution branches not covered by the original test cases in each benchmark/dataset, augment them with additional tests, and validate all test cases against the reference implementation with repeated execution. The results are shown in Table 2 in the response to all reviewers above. As shown in the table, our test case enhancement substantially boosts the coverage and fixes a large number of label errors. In the meantime, even with the improved test and label quality, learning from RepoEquiv can only establish a proxy for universal equivalence. However, a learned equivalence proxy is still a strong alternative/complement to the execution-based verification, as the latter is confined by the same finite set of test cases.

  3. Dependence on Ground-truth Implementations: We want to first make a clarification regarding the scope of application of 1) Semantic verifiers learned through RepoEquiv versus 2) Code generation models trained/evaluated using 1) as main/auxilary reward models and/or evaluators. For 1), it is required by the construction of RepoEquiv to have both ground-truth implementations and test cases. And as shown by our results in Table 2 in the paper, it is viable to train a verifier by leveraging existing benchmarks/datasets with ground-truth and test cases that can also generalize to unseen repos/domains. For 2) one can always construct a large amount of training/evaluation data for the code generation model where ground-truth implementations are available. In particular, given any trusted repository, one can mask out arbitrary code snippets of interest and task the model to generate the missing snippets. In such cases, the masked-out snippets are automatically ground-truth implementations, and even when test cases/exectuions are not available, the verifier learned from 1) can serve as the reward model/evaluator by judging if the model generation is equivalent to the masked ground-truth. This is based on the OOD generalization design of RepoEquiv as well as the OOD results in Table 2, which show that a learned verifier from 1) is able to provide reliable judgments in these unseen training cases.

Questions: We answer the questions from reviewers in each of the corresponding bullet points above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all the reviewers with new experiments

We appreciate all the reviewers for recognizing the research and practical value of RepoEquiv with respect to the research question of cost- and label-efficient evaluation of code generation evaluation, the rigorous data compilation pipeline, and the comprehensive analysis, as well as all the constructive feedback and questions.

Regarding the two main concerns shared by the reviewers: 1) the detailed breakdown of the effect of repository context for semantic equivalence judgements; and 2) the gap between finite test cases' behavior vs. universal semantic equivalence, we conduct the following additional experiments:

  1. The performance changes of whether supporting context is provided to the verifier on each category of tasks, where we perform further categorization of each data item in the evaluation set based on dependency analysis, i.e., whether determining the equivalence requires additional in-file and/or cross-file context. As shown in Table X below, across all the in-file and cross-file conditions, having additional context almost universally boosts the judge accuracy across all models, demonstrating that RepoEquiv presents real context-dependent challenges for semantic equivalence judgements on repository-level coding.
ModellocalIn-file-dependentCross-file-dependent
GPT-4o+2.37+4.79+14.80
GPT-5.3-Codex−0.86+2.56+1.15
Gemini-3-Flash+3.74+3.06+5.24
Claude-Sonnet-4.6+1.58+5.08+4.60
Qwen base+0.30−0.54+3.09
Qwen SFT+3.18+1.50+13.72
  1. We conduct a coverage analysis and augment test cases across both general and scientific domains. In particular, we examine execution branches not covered by the original test cases in each benchmark/dataset, augment them with additional tests, and validate all test cases against the reference implementation with repeated execution. The results are shown in Table Y: our test case enhancement substantially boosts the coverage and fixes a large number of label errors.
Measured populationOriginal testsOriginal + added testsGain
Reference targets85.44%97.24%+11.80 pp
Candidate targets59.32%76.88%+17.56 pp
Originally test-positive candidates68.55%82.48%+13.93 pp

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.