acceptodds
Under review as a conference paper at ICLR 2027

BenchMutator : Agent Mutated Benchmarks

Abstract

Benchmark data contamination undermines the evaluation of large language models. Transforming the instances of an existing benchmark is a lower-cost alternative to constructing a new one, but current transformation methods have two limitations. First, most are task-specific and do not transfer across benchmarks. Second, their effectiveness is commonly inferred from the performance degradation they induce, although degradation can also result from invalid instances, increased difficulty, or variance across generation runs. We introduce BenchMutator, a task-adaptive multi-agent framework that calibrates transformation rules from benchmark examples and coordinates specialized agents to mutate inputs, update reference answers, and validate the resulting instances. We evaluate BenchMutator on six benchmarks spanning code synthesis, program repair, mathematical reasoning, and text-based question answering, using four model families. To control for contamination, we evaluate both base models and models deliberately fine-tuned on benchmark instances. We assess four properties, namely transformation diversity, human-assessed validity, consistency across generation runs, and model response to mutation. BenchMutator produces broader lexical and semantic variation than existing mutation approaches. A human audit reports 91.7% acceptance of transformed instances, with a near-zero mean shift in perceived difficulty. On GSM8K, BenchMutator yields lower run-to-run performance variation than the dynamic baselines Self-Evolving and DyVal. Mutation effects vary across tasks, generators, and model families, including for models fine-tuned on benchmark instances. These results indicate that performance degradation alone is not a reliable indicator of contamination mitigation, and that benchmark transformations should be assessed jointly on validity, diversity, and evaluation consistency. BenchMutator makes this assessment practical across task families.

Primary area:
datasets and benchmarks
Keywords:
benchmark mutation, data contamination, agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NpX8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces BenchMutator, a framework for refreshing existing benchmarks through task-adaptive transformations. It first derives and calibrates transformation rules from a benchmark subset, then applies these rules to individual instances while updating reference answers and tests, and checking validity. Experiments cover six benchmarks, two generator models, and four target model families, with analyses of diversity, human-assessed validity, performance shifts, and consistency across generation runs. Additional experiments fine-tune three target model families on original benchmark instances to examine the effects of known additional exposure.

Critical strengths
  1. The paper addresses an important ambiguity in benchmark evaluation, since performance drops after transformation can reflect difficulty changes, invalid instances, or sensitivity to wording and memorization. Evaluating validity, diversity, and consistency is useful. The controlled exposure experiment is also a sensible way to study this issue without assuming that the base models' pretraining data were contamination-free.

  2. The framework combines reusable benchmark-level rules with instance-level transformation and coordinated answer and test updates, addressing a practical challenge in maintaining usable evaluation data after modification. Comparing two generators and many target model families also helps reveal how much the resulting evaluation depends on the models involved.

Critical weaknesses
  1. Section 5.4 defines CR as normalized Hamming distance but interprets higher values as greater agreement, while Section 5.1 treats higher self-BLEU as greater diversity without specifying an inversion. These inconsistencies make the contamination-resistance and diversity conclusions difficult to assess.

  2. The human audit includes only 30 distinct items, excludes QuixBugs, and shows low annotator agreement (Fleiss' kappa of 0.212). A near-zero mean difficulty shift can mask substantial changes across individual items, providing limited support for difficulty preservation.

  3. Original scores differ substantially across methods in Table 4, leaving the comparability of evaluation subsets and settings unclear. Without explaining these differences and how variability is aggregated across target models, the performance-shift and consistency comparisons are difficult to assess.

Questions for authors
  1. What are the exact formulas and aggregation procedures for CR and self-BLEU diversity? In particular, is CR a distance or an agreement measure? What does each entry of its evaluation vector represent, and how do the CR values relate to the before/after results in Table 8? Please clarify whether correcting the descriptions changes any method rankings or conclusions.

  2. Why do the Original scores in Table 4 differ across methods on the same benchmark? Please specify the source-item counts, filtering rules, and evaluation settings for each method, and explain how the reported standard deviations are aggregated across three generation runs and four target models. If the evaluated subsets differ, do the performance-shift and consistency conclusions hold on shared source items?

  3. Appendix A.1 mentions BBC-latest scores above 1.0, but the corresponding entries in Table 7 appear to remain below 1.0. Could the authors reconcile this description with the table and clarify whether any reported results are affected by scoring errors?

  4. Can the authors report the item-level difficulty distribution and separate audit results for answer correctness, test consistency, and rejection reasons? These would help assess whether the near-zero mean masks substantial changes and whether annotator disagreement is primarily stylistic, as claimed.

2 comments
AuthorsOpenReview ↗

Response to Reviewer NpX8

We thank the reviewer for recognising the value of the validity, diversity and consistency perspective and of the controlled-exposure design.

W1 / Q1. Definitions of CR and self-BLEU diversity.

CR is an agreement measure, not a distance. Following Sun et al. (2025):

CR=1−H(R(Mbase,DS),  R(Mft,DS))\mathrm{CR} = 1 - H\big(R(M^{\text{base}}, D^S),\; R(M^{\text{ft}}, D^S)\big)

Here R(M,DS)∈{0,1}nR(M, D^S) \in \{0,1\}^n, and entry ii is 1 if model MM passes mutated item ii. HH is the normalised Hamming distance, so CR is the fraction of mutated items on which the base and fine-tuned models reach the same pass/fail verdict.

Relation to Table 8. The number of items on which two models disagree is at least the difference in their pass counts. Hence CR≤1−∣FT Mut−Base Mut∣\mathrm{CR} \le 1 - |\text{FT Mut} - \text{Base Mut}|. CR also captures item-level flips that cancel out in the aggregate accuracies of Table 8.

Self-BLEU diversity is a set-level measure. For each (system, dataset) group, we take the mutated inputs (at most 300, randomly subsampled), and for each item hih_i, we draw 5 references at random from the other items and compute sacreBLEU sentence_bleu. The item's diversity is 1−BLEUi/1001-\mathrm{BLEU}_i/100, so higher means more diverse. The lexical and semantic axes are instead per-item 1−cos⁡1-\cos between original and mutation (TF-IDF). Full definitions of all three metrics are in Appendix A.6.

W2 / Q4. Human audit: distribution, criteria and rejection reasons. We agree that the audit is small, and that agreement is low (κ=0.212\kappa = 0.212). It supports difficulty preservation but does not establish it.

  • Difficulty distribution. 80% of difficulty ratings are "same". The mean shift is +0.03 for DeepSeek and +0.06 for Mistral.
  • Rejections. Five of the 60 judgments were rejections:
    • two on HumanEval, both from the Mistral generator, where the updated tests were inconsistent with the mutated prompt;
    • one on TruthfulQA, from Mistral, with an incorrect reference answer;
    • two on BBC-latest, one per generator, where the question was absent in the context by the generation token limit.

W3 / Q2. Why the Original scores differ in Table 4. Original is the pass rate on each method's own surviving source items, over that method's own set of target models:

  • BenchMutator keeps only items that pass the task-specific checker (code execution, numeric check, or LLM judge for text). Item counts therefore vary by run. Specifically, 143–154 on HumanEval with DeepSeek, and 99–128 with Mistral.
  • DyCodeEval keeps verifier-approved rewrites (90–120 on HumanEval).
  • BenchAgent generates 200 items per task, with no source items.
  • The different rule-based transformations keeps every item.

All methods share the same evaluation settings having pass@1 with greedy decoding (temperature 0). Each reported standard deviation is a population standard deviation (ddof = 0) over one pooled list of pass@1 values, with one value per (target model, generation run). It therefore mixes between-model and between-run variance. Section 5.3 wrongly describes it as run-to-run variation.

Shared items and a common set of target models. We restricted the analysis to items shared by all methods and to the three common target models (Gemma, Llama and Qwen):

  • HumanEval (135 shared items). The Original scores become comparable. Every shift keeps its sign:

    MethodPooled shiftShared-item shift
    BenchMutator-Mistral−0.09−0.16
    BenchMutator-DeepSeek+0.13+0.06
    DyCodeEval+0.11+0.08

The order of BenchMutator-DeepSeek and DyCodeEval reverses, but both values are small, so we draw no conclusion from the difference between them.

  • GSM8K. The BenchMutator shifts are −0.03 (DeepSeek) and −0.08 (Mistral), against −0.07 and −0.08 in the pooled table. We attribute the remaining difference to the change in target models rather than to the item subset.

In the revised paper, the main Table 11 now use shared items, the common target models.

Q3. BBC-latest scores above 1.0. The sentence in Appendix A.1 comes from an analysis run before we fixed a scoring bug. The reviewer is right that Table 7 contains no score above 1.0, because Tables 4, 5, 7 and 8 were all recomputed after the fix. No reported result is affected by the bug, and we will remove the outdated sentence. We also fixed a formatting error in the table 8 where an entry printed as 40299000 should read 0.403.

Thanks for the response. I still have a few concerns:

  1. In the rebuttal, the authors state that "CR is an agreement measure, not a distance. Following Sun et al. (2025)." However, Line 420 of the latest version still states: "Contamination resistance (CR) is the normalized Hamming distance between the question-level evaluation vectors of the clean and contaminated models on the mutated benchmark." This description remains inconsistent with the definition provided in Line 903. While the authors claim that their CR definition follows [1], they have not explained why it contradicts the description in the main text.

  2. Line 260 introduces "self-BLEU diversity across the set of mutated instances produced for each dataset (Montahaei et al., 2019)." However, the cited paper [2] explicitly states in Section 3.1.2: "By averaging these BLEU scores (obtained for generated sentences), a metric that is called Self-BLEU is achieved where its lower values shows more diversity." In contrast, the authors introduce a different formula in Line 912 of the latest version. I would like the authors to explain why their Self-BLEU definition differs from the cited work [2], rather than simply adding a new formula to the appendix without justifying this discrepancy.

  3. Line 903 still contains an incomplete citation: "Following ?"

Overall, I do not believe the authors have adequately addressed these fundamental concerns regarding metric definitions and consistency. Simply adding definitions to the appendix does not resolve the inconsistencies between the main text, the cited literature, and the implemented metrics. Therefore, I will maintain my rating.

[1] "The Emperor’s New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination" (ICML 2025)

[2] "Jointly Measuring Diversity and Quality in Text Generation Models"(NAACL 2019 NeuralGen workshop)

Reviewer_2neEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BenchMutator, a multi-agent framework that finds benchmark-specific transformation strategies asynchronously, without requiring any form of a transformation taxonomy. The evaluation of language models generally relies on benchmark scores to understand generalized performance. In turn, benchmarks helps drive future scientific research and advances in the field. One of the most pervasive problems with benchmarks is dataset contamination, which occurs when evaluation data is present in training corpora and poses a risk of misrepresenting model capabilities. While “living” datasets, or continuously updated evals, are one counter to contamination, it is nontrivial and expensive in practice, and will eventually be swallowed by the same contamination loop they are designed to bypass. Benchmark transformation is a different means of addressing the problem, wherein one may generate variants of existing benchmark samples/processes/solutions, all the while retaining the original spirit and connection to the pre-transformed benchmark. Transformation is not perfect, however, as previous work has shown that transformations may invalidate correctness, process-level reasoning, and difficulty of the original problem specification. BenchMutator works in two phases. In the first phase, one agent identifies task constraints from a subset of the benchmark, while another proposes transformation strategies that are iterated on a calibration probe. The second phase features an agent instantiating the calibrated transformation rules, and another agent responsible for revising reference answers and tests. Across a set of benchmarks, covering a variety of domains, and a suite of models, the authors show BenchMutator has a 91.7% human acceptance on a 30-sample audit, and has lower variance than other benchmarks like self-evolving and dyval on mathematical reasoning (gsm8k).

Critical strengths
  • The authors tackle an important problem, and their argument is intuitive that benchmark score degradation after transformation is not proof that the transformation fixed contamination.
  • The proposed method is, in large part, task agnostic/generalized. The pipeline is end-to-end, unlike much prior work that covers only a single task.
  • The evaluation has breadth, as it looks beyond correctness to include diversity, human evaluation, and consistency.
  • The writing is generally clear, and the problem is well motivated (see Questions section for some presentation nits)
  • The reporting, particularly around newer and more capable models bypassing transformation as an effective measure, is reported honestly.
Critical weaknesses
  • Perhaps my biggest concern is that, in the appendix, the authors report that with DeepSeek as the target model (L767), they observed anomalous scores above 1.0 (the maximum possible) on BBC-Latest across multiple baselines. They are honest in stating that this could be a “scoring artifact rather than genuine performance”. This reads as though there is a code bug that hasn’t been addressed, but also raises a question of whether upstream results can be trusted. It appears these anomalies are being averaged into the main table

  • A core point in the framing is that BenchMutator is a transformation that helps remove advantage gained through memorization. The results (particularly Table 4) show that transformation increases the mean score in 6 cases and leaves it flat in 2 others. So in two-thirds of measured benchmarks, BenchMutator actually increases performance or stays flat. For MBPP, performance nearly doubles. This is a bit inconsistent with the human evaluation, which states there was a 0.04 difficulty shift in the audited mutated samples. So, for the audited samples, it is apparent that the difficulty remains the same, but for a case like MBPP, the benchmark performance doubles, implying that the difficulty has shifted drastically.

  • The human audit itself is framed as a headline result in the abstract, but it is fairly limited in significance due to the sample size and the stratification conducted. For instance, the limited sample size, stratified across datasets for the human eval, doesn’t empirically support the difficulty-preservation claim. Against that, the human annotators didn’t agree much, and before averaging, one group was barely above random. This, along with the benchmark results in Table 4, work to invalidate the difficulty-preservation claim.

Questions for authors
  • Can the authors clarify against W1? I am trying to understand whether the results in Table 4 are averaged over the anomalous BBC-latest scoring runs mentioned in the appendix, and why the scoring bug was not addressed in the code but instead left as a side note in the appendix.

  • Can the authors explain the variance in certain benchmark scores, like MBPP, which shows an unnaturally large performance increase on the mutated dataset? This is orthogonal to the human audit which shows that difficulty remains consistent across transformations. Given the significant increase in performance, I think additional analysis is needed to understand why MBPP experienced such a large gain (e.g., were the prompts longer and less ambiguous?).

  • Transformation is noted as preserving comparability. In that regard, if a transformation is applied to a benchmark, it would make intuitive sense for the ranking to be preserved from the original to the transformed state across a set of models. Could the others provide some form of a ranking correlation analysis here?

Presentation Concerns/Typos:

  • Table 1 is bleeding over the right margin.
  • Figure 1 is illegible (small font) at normal zoom.
  • Figure 2 x and y labels are also illegible at a normal zoom, and the density of this figure, in terms of content, is too much.
  • Tables 4 and 5 suffer from the same issue: the fonts appear too small. Table 4: it is unclear if this is reporting standard deviations or standard errors, but there is some inconsistencies with run-level error reporting in the tables, and with decimal precision.
  • The “Finding {RQ)” cards present more like a preprint, and ultimately end up taking additional space, and creating excessive amounts of white space above and below where they occur.
  • Line 471 has a typo “conde synthesis”
  • Line 775 has a typo “mentionned”
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2neE

We thank the reviewer for the positive assessment of the problem framing, the generality of the pipeline and the breadth of the evaluation.

W1 / Q1. BBC-latest scores above 1.0. The sentence in Appendix A.1 (L767–771) comes from an earlier analysis, run before we identified and fixed a scoring bug in the evaluation code. Tables 4, 5, 7 and 8 were all recomputed after the fix. No reported score exceeds 1.0, and no pre-fix run is included in any average. We have updated the appendix as well. We also corrected a formatting error in Table 8, where an entry printed as 40299000 should read 0.403.

W2 / Q2. Score increases, and MBPP in particular. Our paper does not argue that mutation should lower scores. Its central point (Section 1) is that score degradation alone is not evidence of mitigation. We measure the contamination effect separately, with CR, by comparing base and fine-tuned models (Section 5.4). We have now analyzed the MBPP increase.

  • The original prompts are underspecified. Original MBPP prompts are a one-line description with no function name or signature. Many original failures are therefore name or arity mismatches rather than reasoning errors.
  • Mutated prompts often expose the interface. Every mutated prompt is more than 1.5× the length of its original. With the DeepSeek generator, 91% of mutated prompts contain assert/>>> examples that the original lacks, and 88% name the target function (Mistral: 23% and 18%). This also explains the larger gain for DeepSeek.
  • Examples explain part of the gap. On base-model runs, the mutated pass rate exceeds the original by +0.14 when the prompt contains such examples and by +0.08 when it does not. The remaining gap is not isolated.

The human audit asks whether the underlying task is equally difficult (80% of ratings are "same"). It does not ask whether the prompt now supplies the interface the grader expects. The two results therefore measure different things, and the audit covered only 12 MBPP items.

W3. Strength of the human audit. We agree that the audit is small (30 items) and that agreement is low (Fleiss' κ=0.212\kappa = 0.212; pool A: 0.116). It supports difficulty preservation but cannot establish it on its own. At the item level, 80% of difficulty ratings are "same", and the mean shift is +0.01 for DeepSeek and +0.07 for Mistral.

Q3. Ranking preservation. We compared model rankings using the three target models that every method was evaluated on (Gemma, Llama and Qwen). For each method–benchmark cell we computed Kendall's τ\tau between the ranking on Original and the ranking on Mutated:

  • The mean τb\tau_b is 0.65. 18 cells have τb=1.00\tau_b = 1.00, 43 are positive, one is 0 and two are negative.
  • The two negative cells (τb=−0.33\tau_b = -0.33) are BenchMutator-DeepSeek on MBPP, which is the interface effect described above, and BenchMutator-Mistral on TruthfulQA, where the three Original scores lie within 0.08 of each other.
  • Over its 12 cells, BenchMutator has a mean τb\tau_b of 0.60. This is lower than identity-preserving surface edits such as back-translation, which is expected because those edits change the instances less.

We have added this analysis to the Appendix A.9.

Thank you to the authors for the rebuttal. I appreciate the brevity and concision here. That said, I will be retaining my score. Some concerns that I still retain:

  • For MBPP, "It does not ask whether the prompt now supplies the interface the grader expects.". I am a bit concerned that the mutation here made the problems inherently easier. I would argue that the semantics of the benchmark have been disrupted to an extent that leaked parts of the answer into the task.

  • The premise of W1 is something that may have been handled with more care before submission. On my initial read, I was not sure which numbers could be trusted.

  • The claims about the human audit aren't sufficiently backed, as the authors have stated here.

I appreciate the authors' work, and I encourage them to continue working in this direction, which I believe to be well motivated and important.

Reviewer_Qp24OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

BenchMutator is a task-adaptive multi-agent framework that mitigates LLM benchmark contamination by transforming existing instances. It calibrates benchmark-specific rules from sampled examples, then applies them per instance with answer updates and validation. Across six benchmarks and four model families, it shows greater diversity, high human acceptance, and lower run-to-run variance than baselines. The authors argue that performance degradation alone poorly indicates contamination mitigation, since sensitivity also reflects model capability.

Critical strengths
  1. Argues performance drops alone don't prove contamination mitigation, proposing joint assessment of validity, diversity, and consistency.
  2. Calibrates benchmark-specific rules without handcoded taxonomy, covering code, program repair, math, and QA, a broader scope than prior methods.
  3. Updates reference answers and tests, then applies task-specific checkers, reducing the silent ground-truth invalidation common in paraphrase approaches.
Critical weaknesses

1.. Stale benchmarks: I am not sure anyone is using HumanEval, MBPP, etc for experimental analysis. I would have been more interested in latest benchmarks -- DeepSWE, SWEBench family, etc. 2. Mutation doesn't seem to be preserving difficulty. See MBPP raising from 0.49 to 0.82, BBC Latest: 0.51 to 0.71, ... 3. Novelty: I didn't find the approach to be something novel or worthy of publication. This style of mutation based analysis is well understood in software engineering. 4. Table 5: the results seem to show swap character, change character case, etc outperforming Benchmutator on LLaMa

Questions for authors
  1. There are just too many data points and simplistic benchmarks that make the paper uninteresting. Doing the same on 2-3 complex benchmarks and showing value with case studies would have made it more compelling for me. What is the rationale for just adding the minor benchmarks and plethora of metrics?
  2. Has this been used for model training as a case study? What has been the outcome? In other words, how was model training impacted?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Qp24

We thank the reviewer for highlighting the joint validity–diversity–consistency assessment, the calibrated rules across four task families, and the coordinated update of reference answers and tests.

Q1. Number of benchmarks and metrics. Our central claim is task-adaptivity, so we need at least one benchmark per task family: code synthesis, program repair, math and text QA. Each metric answers one research question: diversity (RQ1), validity (RQ2), consistency (RQ3) and contamination resistance (RQ4). We agree that the current presentation is dense. In the revised paper, the main text will focus on the BenchMutator results and the most directly comparable baselines. Rule-based results is now moved to the appendix, and we will add worked case studies showing original and mutated instances, reference updates and checker outcomes for each task family.

Q2. Using mutated benchmarks for training. This is a valid use case that we did not study. Because BenchMutator produces validated variants, with updated reference answers and tests that pass a task-specific checker, its output could serve as an augmented training set. Two directions seem promising (i) training for robustness to reformulation, and (ii) replacing leaked instances during decontamination. However, this is beyond the scope of this paper. We aim to answer whether a transformed benchmark remains a valid and consistent measurement instrument. Training on the mutated variants would mix the roles of training data and evaluation data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.