- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces BenchMutator, a framework for refreshing existing benchmarks through task-adaptive transformations. It first derives and calibrates transformation rules from a benchmark subset, then applies these rules to individual instances while updating reference answers and tests, and checking validity. Experiments cover six benchmarks, two generator models, and four target model families, with analyses of diversity, human-assessed validity, performance shifts, and consistency across generation runs. Additional experiments fine-tune three target model families on original benchmark instances to examine the effects of known additional exposure.
Critical strengths
-
The paper addresses an important ambiguity in benchmark evaluation, since performance drops after transformation can reflect difficulty changes, invalid instances, or sensitivity to wording and memorization. Evaluating validity, diversity, and consistency is useful. The controlled exposure experiment is also a sensible way to study this issue without assuming that the base models' pretraining data were contamination-free.
-
The framework combines reusable benchmark-level rules with instance-level transformation and coordinated answer and test updates, addressing a practical challenge in maintaining usable evaluation data after modification. Comparing two generators and many target model families also helps reveal how much the resulting evaluation depends on the models involved.
Critical weaknesses
-
Section 5.4 defines CR as normalized Hamming distance but interprets higher values as greater agreement, while Section 5.1 treats higher self-BLEU as greater diversity without specifying an inversion. These inconsistencies make the contamination-resistance and diversity conclusions difficult to assess.
-
The human audit includes only 30 distinct items, excludes QuixBugs, and shows low annotator agreement (Fleiss' kappa of 0.212). A near-zero mean difficulty shift can mask substantial changes across individual items, providing limited support for difficulty preservation.
-
Original scores differ substantially across methods in Table 4, leaving the comparability of evaluation subsets and settings unclear. Without explaining these differences and how variability is aggregated across target models, the performance-shift and consistency comparisons are difficult to assess.
Questions for authors
-
What are the exact formulas and aggregation procedures for CR and self-BLEU diversity? In particular, is CR a distance or an agreement measure? What does each entry of its evaluation vector represent, and how do the CR values relate to the before/after results in Table 8? Please clarify whether correcting the descriptions changes any method rankings or conclusions.
-
Why do the Original scores in Table 4 differ across methods on the same benchmark? Please specify the source-item counts, filtering rules, and evaluation settings for each method, and explain how the reported standard deviations are aggregated across three generation runs and four target models. If the evaluated subsets differ, do the performance-shift and consistency conclusions hold on shared source items?
-
Appendix A.1 mentions BBC-latest scores above 1.0, but the corresponding entries in Table 7 appear to remain below 1.0. Could the authors reconcile this description with the table and clarify whether any reported results are affected by scoring errors?
-
Can the authors report the item-level difficulty distribution and separate audit results for answer correctness, test consistency, and rejection reasons? These would help assess whether the near-zero mean masks substantial changes and whether annotator disagreement is primarily stylistic, as claimed.
2 comments
Thanks for the response. I still have a few concerns:
-
In the rebuttal, the authors state that "CR is an agreement measure, not a distance. Following Sun et al. (2025)." However, Line 420 of the latest version still states: "Contamination resistance (CR) is the normalized Hamming distance between the question-level evaluation vectors of the clean and contaminated models on the mutated benchmark." This description remains inconsistent with the definition provided in Line 903. While the authors claim that their CR definition follows [1], they have not explained why it contradicts the description in the main text.
-
Line 260 introduces "self-BLEU diversity across the set of mutated instances produced for each dataset (Montahaei et al., 2019)." However, the cited paper [2] explicitly states in Section 3.1.2: "By averaging these BLEU scores (obtained for generated sentences), a metric that is called Self-BLEU is achieved where its lower values shows more diversity." In contrast, the authors introduce a different formula in Line 912 of the latest version. I would like the authors to explain why their Self-BLEU definition differs from the cited work [2], rather than simply adding a new formula to the appendix without justifying this discrepancy.
-
Line 903 still contains an incomplete citation: "Following ?"
Overall, I do not believe the authors have adequately addressed these fundamental concerns regarding metric definitions and consistency. Simply adding definitions to the appendix does not resolve the inconsistencies between the main text, the cited literature, and the implemented metrics. Therefore, I will maintain my rating.
[1] "The Emperor’s New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination" (ICML 2025)
[2] "Jointly Measuring Diversity and Quality in Text Generation Models"(NAACL 2019 NeuralGen workshop)
Response to Reviewer NpX8
We thank the reviewer for recognising the value of the validity, diversity and consistency perspective and of the controlled-exposure design.
W1 / Q1. Definitions of CR and self-BLEU diversity.
CR is an agreement measure, not a distance. Following Sun et al. (2025):
CR=1−H(R(Mbase,DS),R(Mft,DS))
Here R(M,DS)∈{0,1}n, and entry i is 1 if model M passes mutated item i. H is the normalised Hamming distance, so CR is the fraction of mutated items on which the base and fine-tuned models reach the same pass/fail verdict.
Relation to Table 8. The number of items on which two models disagree is at least the difference in their pass counts. Hence CR≤1−∣FT Mut−Base Mut∣. CR also captures item-level flips that cancel out in the aggregate accuracies of Table 8.
Self-BLEU diversity is a set-level measure. For each (system, dataset) group, we take the mutated inputs (at most 300, randomly subsampled), and for each item hi, we draw 5 references at random from the other items and compute sacreBLEU
sentence_bleu. The item's diversity is 1−BLEUi/100, so higher means more diverse. The lexical and semantic axes are instead per-item 1−cos between original and mutation (TF-IDF). Full definitions of all three metrics are in Appendix A.6.W2 / Q4. Human audit: distribution, criteria and rejection reasons. We agree that the audit is small, and that agreement is low (κ=0.212). It supports difficulty preservation but does not establish it.
W3 / Q2. Why the Original scores differ in Table 4. Original is the pass rate on each method's own surviving source items, over that method's own set of target models:
All methods share the same evaluation settings having pass@1 with greedy decoding (temperature 0). Each reported standard deviation is a population standard deviation (ddof = 0) over one pooled list of pass@1 values, with one value per (target model, generation run). It therefore mixes between-model and between-run variance. Section 5.3 wrongly describes it as run-to-run variation.
Shared items and a common set of target models. We restricted the analysis to items shared by all methods and to the three common target models (Gemma, Llama and Qwen):
HumanEval (135 shared items). The Original scores become comparable. Every shift keeps its sign:
The order of BenchMutator-DeepSeek and DyCodeEval reverses, but both values are small, so we draw no conclusion from the difference between them.
In the revised paper, the main Table 11 now use shared items, the common target models.
Q3. BBC-latest scores above 1.0. The sentence in Appendix A.1 comes from an analysis run before we fixed a scoring bug. The reviewer is right that Table 7 contains no score above 1.0, because Tables 4, 5, 7 and 8 were all recomputed after the fix. No reported result is affected by the bug, and we will remove the outdated sentence. We also fixed a formatting error in the table 8 where an entry printed as
40299000should read0.403.