acceptodds
Desk rejected at ICLR 2027

ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Benchmarks are central to measuring the capabilities of large language models and guiding model development, yet widespread data leakage from pretraining corpora undermines their validity. Models can match memorized content rather than demonstrate true generalization, which inflates scores, distorts cross-model comparisons, and misrepresents progress. We introduce ArenaBencher, a model-agnostic framework for automatic benchmark evolution that updates test cases while preserving comparability. Given an existing benchmark and a diverse pool of models to be evaluated, ArenaBencher infers the core ability of each test case, generates candidate question–answer pairs that preserve the original objective, verifies correctness and intent with an LLM as a judge, and aggregates feedback from multiple models to select candidates that expose shared weaknesses. The process runs iteratively with in-context demonstrations that steer generation toward more challenging and diagnostic cases. We apply ArenaBencher to math problem solving, commonsense reasoning, and safety domains and show that it produces verified, diverse, and fair updates that uncover new failure modes, increase difficulty while preserving test objective alignment, and improve model separability. The framework provides a scalable path to continuously evolve benchmarks in step with the rapid progress of foundation models.

Primary area:
datasets and benchmarks
Keywords:
LM Evaluation, Benchmarking, Evaluation protocols

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_H3KMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ArenaBencher, a framework for automatic benchmark evolution using feedback from multiple models. It first extracts the ability tested by each original example, generates and verifies new variants, and then selects difficult examples based on shared failures across several models. The selected examples are further used as in-context demonstrations for iterative refinement. The main idea is interesting, but the current method and experiments are too narrow to fully support the paper’s claims.

Critical strengths
  1. The paper studies an important problem: static benchmarks may become saturated or contaminated over time.
  2. Using shared failure signals from multiple models to guide benchmark evolution is a useful insight. It may reduce overfitting to the behavior of a single model.
  3. The framework has a clear pipeline, including ability extraction, candidate generation, verification, multi-model scoring, and iterative refinement.
Critical weaknesses
  1. The method is relatively simple and heuristic. It mainly combines LLM-based rewriting, LLM-based verification, and failure-based candidate selection. The iterative use of difficult examples as demonstrations is also straightforward, so the technical contribution is limited.
  2. The selected benchmarks are relatively easy. In particular, GSM8K may not be a strong test bed for benchmark evolution because many recent models already perform well on it. For mathematical reasoning, it would be more convincing to study harder benchmarks such as MATH-500, AIME, OlympiadBench, or Omni-MATH. This is important for showing whether the method can still find meaningful weaknesses when the original problems are already difficult.
  3. The evaluated models are relatively weak. The model pool mainly contains 1B–7B Llama, Qwen, and Mistral models. The results show that ArenaBencher can expose weaknesses of small models, but it remains unclear whether the method is useful for stronger and more recent models.
  4. The experimental scope is narrow. Only one benchmark is used for each domain, and there are no experiments on coding, long-context reasoning, difficult knowledge tasks, or advanced mathematics. More importantly, the generated benchmark is evaluated on the same model pool used during construction. There is no evaluation on held-out models.
Questions for authors
  1. How many examples from each benchmark were actually evolved? Will the complete generated datasets and configurations be released?

  2. Does the increase in difficulty remain when the generated examples are evaluated on stronger models that were not used during benchmark construction?

  3. Can the proposed method generalize to more challenging benchmarks, such as MATH-500, AIME, OlympiadBench, or Omni-MATH? In particular, can it generate meaningful and valid hard examples when the original test cases are already difficult?

2 comments
AuthorsOpenReview ↗

Response to Reviewer H3KM

We thank the reviewer for the detailed feedback. We appreciate the recognition of the significance of our research question and the clarity of the proposed pipeline. We address the reviewer’s concerns and questions below.

W1. Technical contribution of the pipeline. Thanks for providing the opportunity for us to highlight our contribution and novelty. The design contribution is how the stages work together: extraction specifies what should remain fixed; verification filters candidate rewrites against that objective; feedback from multiple models ranks the surviving candidates; and the selected candidates inform the next round. The distinction from difficulty-only selection is that a difficult candidate first has to pass the task-preservation check.

W2. Whether GSM8K is a meaningful test bed. GSM8K provides a setting where annotators can check both the answer and whether a rewrite still requires the same mathematical idea. In this setting the default update lowers accuracy for every evaluated model; for example, Llama-3.2-3B moves from 74.1% to 26.4%. The accompanying audit gives item-level evidence that the revisions generally retain the source objective. On advanced mathematics tasks, the same criteria would need to be checked at the level of the intended mathematical idea and the validity of the solution.

W3. Model scale and recency. The six models span Llama, Qwen, and Mistral variants from 1B to 7B. The direction of the change is consistent across these models, including those with higher source-benchmark scores. We claim that it is evidence across the tested pool, not a comparison with recent frontier models.

W4. Breadth of tasks and held-out evaluation. Arithmetic, commonsense reasoning, and safety let us examine the same pipeline under three different definitions of task success. With the given suite of model pool, advanced tasks such as coding, long-context reasoning, difficult knowledge tasks, or advanced mathematics would be way beyond the capabilities of the models. The evaluation uses the construction pool, so the results show the consistency of the difficulty increase within that pool rather than transfer to other models.


Q1. How many examples were evolved, and what is available? Thanks for the question. We evolved every item in the evaluation sets for GSM8K, CommonsenseQA, and AdvBench Harmful Behaviors. The 100 items were randomly sampled from the evolved GSM8K set for human annotation only. They were not a limit on how many items we evolved.

We will release the complete generated datasets and the pipeline used to construct them. The settings for the reported runs are documented in §4.1 and Appendix D. Because the construction procedure uses feedback from a particular model pool, the released datasets are best interpreted as tests of what those models can do on revised items that target the original skills. The pipeline is the more significant contribution since users can choose a model pool at a different capability level and use our proposed pipeline to construct an updated benchmark for that pool.

Q2. Do the results hold for stronger models outside the construction pool? Thanks for the important question. The current results show increases in difficulty for all six evaluated 1B–7B models. No stronger model entirely outside the construction pool is included in Table 1, so the results do not establish that transfer.

Q3. Can the approach produce valid updates to already-hard math problems? Thanks for the important question. ArenaBencher is most useful when the target model pool is approaching saturation on the original benchmark. If a benchmark such as MATH-500, AIME, OlympiadBench, or Omni-MATH still challenges and distinguishes the target models, making it harder is less useful. There is also a verification challenge where, for an already-difficult problem, it can be hard to establish that the rewritten problem has a correct solution and still tests the intended mathematical idea. Lower solver accuracy cannot tell us whether the rewrite is harder for the right reason or simply flawed. For those reasons, we would not recommend applying the current pipeline as-is to already-hard math benchmarks.

Thank you for the clear response. It helps clarify the scope of the current results. However, the main concerns remain. I will keep my current score.

Reviewer_FRbvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ArenaBencher. The motivation of this paper is that existing benchmarks are static and are very likely to be hacked by models due to data leakage. To tackle this challenge, the authors propose this method to dynamically generate new test samples from the original benchmarks, where the newly generated samples preserve the same evaluation objectives, but are more difficult. To demonstrate the effectiveness of the method, the authors also evaluated several aspects of the newly generated benchmark, like difficulty, separability, fairness, and alignment. They evaluated models on the new bench and showed that all models demonstrate a consistent performance drop on the new benchmark, indicating that the new benchmark is challenging for these models.

Critical strengths
  1. The motivation of this paper is clear and interesting. How to evolve existing benchmarks to elevate the impact of data leakage is a very important problem.
  2. This paper is easy to follow and understand.
Critical weaknesses
  1. The proposed method is kind of heuristic and lacks some critical details. For instance, after generating the new test cases, they used a judgment model to ensure the validity of the benchmark. How to ensure the reliability of the judgement, and the choice of the judgment model will affect the quality of the dataset.
  2. One contribution of this paper is not very clear, that is why the performance drop on the new benchmark indicates a dynamically evolved benchmark, because this is process is to find the OOD test samples for these models. The correlation between the experimental results and the motivation remains unclear.
  3. The evaluated models and benchmarks are outdated, it is unclear whether it is easy to evolve the benchmark for more advanced models and difficult benchmarks.
Questions for authors
  1. Does the updated benchmark remain challenging for models that are not used during candidate selection?
  2. For other questions, please refer to my weaknesses.
1 comment
AuthorsOpenReview ↗

Response to Reviewer FRbv

We thank the reviewer for the thoughtful feedback. We are glad the reviewer found the paper easy to follow and recognized both the importance of renewing static benchmarks and the consistent drop in performance on the revised items. We address the reviewer’s concerns and questions below.

W1. Verifier reliability and choice of judge. Thanks for raising this point. The verifier checks candidate validity and alignment before candidates are scored for difficulty. Those are separate steps. The automatic verification uses the same model snapshot as generation in a separate call, as also clarified in the response to reviewer sT6Y (Q3). The human GSM8K annotation supplies a separate check: 95 out of 100 sampled items were judged aligned and 96 valid and correct. Figure 2 documents a case that the verifier missed. Our reported results use the stated GPT-4o version; the annotation checks the quality of its resulting items rather than comparing alternative judges.

W2. Dynamic evolution, harder examples, and data leakage. The benchmark evolves because the selection feedback informs the next round of candidate generation, where selected items are reused as demonstrations, while new candidates remain anchored to the source objective. Lower accuracy or higher attack success rates measure how challenging the selected revisions are for the evaluated models. They do not distinguish ordinary difficulty, distribution shift, and reduced reliance on memorized material. Data leakage motivates renewing benchmark items; the measured contribution is an iterative way to construct more challenging revisions while checking objective preservation.

To clarify what makes the process iterative, we explain the role of selected examples in the §1 overview:

ArenaBencher checks each candidate against the source test objective and uses feedback from multiple models to select challenging candidates that pass this check. Selected candidates are retained as in-context demonstrations to guide subsequent generations.

W3. More recent models and harder tests. Thanks for the suggestion. The same procedure can accept a different set of source items or feedback models. Whether it produces useful revisions in a harder domain depends on the quality of the checks for validity and objective preservation, not just on whether the new items lower scores. The present evaluation spans three domains and six models from three families, and its empirical results concern those settings.


Q1. Are the revised benchmarks challenging for models excluded from selection? Thanks for the important question. The feedback subset varies during construction, but all six evaluated models are eligible to contribute feedback. Being unsampled for a particular update is therefore different from being held out from construction altogether. Thus we did not report results for models excluded from the pool.

To avoid conflating a model omitted from one round with a held-out model, we specify the shared pool in §4.1:

We use the same six-model pool for benchmark construction and evaluation. The pool spans three model families and model sizes from 1B to 7B parameters.

Reviewer_sT6YOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

ArenaBencher automatically evolves existing benchmarks through multi-model feedback. It extracts the ability targeted by each original item, generates candidate question–answer pairs, verifies them using an LLM, and selects candidates based on aggregated model failure signals. Selected candidates become demonstrations for subsequent iterations. Its main contribution is an integrated framework combining ability preservation, multi-model feedback, and iterative generation across domains. Evaluation covers six models on GSM8K, CommonsenseQA, and AdvBench Harmful Behaviors, supplemented by human assessment of 100 evolved GSM8K items. The results support increased difficulty for the construction-pool models, while fairness, improved separability, and contamination resilience remain insufficiently established.

Critical strengths
  1. Ability extraction, candidate verification, pooled feedback, and iterative demonstrations form a coherent automated pipeline. Under the default configuration, Table 1 shows lower reasoning accuracy and higher safety attack success rates across all six models. For example, Llama-3.2-3B’s GSM8K accuracy falls from 74.1% to 26.4%. This consistent ability to generate challenging items is the paper’s strongest empirical contribution.
  2. The framework explicitly considers ability preservation and answer validity alongside model failure. Three annotators assessed 100 randomly sampled GSM8K updates, reporting 95 aligned and 96 correct items; Figure 2 also examines a failed update. These checks provide preliminary evidence that the difficulty gains have substantive meaning, although validation remains limited in scope.
Critical weaknesses
  1. Under section 3.5, failure counts satisfy ck=N(1−ak)c_k=N(1-a_k), so fairness is exactly the complement of separability, as shown above. Consequently, higher “fairness” rewards compressed model scores; it does not independently establish freedom from model-targeted bias or measure whether performance drops are evenly distributed. A benchmark on which every model fails would achieve maximal fairness while providing no discrimination. In Table 2, default-setting separability decreases from 15.2 to 12.2 on GSM8K, 17.1 to 14.5 on safety, and 8.5 to 7.2 on CSQA. This contradicts the abstract’s improved-separability claim and materially weakens the central benchmark-quality argument.
  2. The six reported evaluation models also participate in benchmark construction, with no entirely held-out model or family. The results therefore establish effectiveness against queried models, but do not adequately demonstrate transfer to unseen systems. Moreover, the m=1m=1 versus m=3m=3 comparison fixes candidate counts and refinement rounds, giving the latter approximately three times as many feedback-model evaluations. Without budget-matched and no-feedback controls, the specific benefit of multi-model aggregation remains unclear. No direct contamination or memorization analysis is provided either. Increased error rates alone therefore do not establish more reliable, generalizable, or contamination-resilient evaluation.
Questions for authors
  1. Does the implementation use the fairness formula in section 3.5? If so, what independent evidence supports cross-model fairness and improved discrimination, given its exact complementarity with separability?
  2. Do the advantages of m=3m=3 over m=1m=1 and no-feedback generation persist on models or families excluded from construction, and under matched total feedback-evaluation budgets?
  3. Section 4.1 uses GPT-4o-2024-08-06 for both generation and verification. What precisely makes the judge independent? Are independent quality checks available for CSQA and safety, together with results after removing incorrect, ambiguous, or ability-shifting items?
2 comments
AuthorsOpenReview ↗

Response to Reviewer sT6Y 1/2

We thank the reviewer for the careful reading and constructive feedback. We appreciate the recognition that the procedure forms a coherent pipeline, that it consistently produces harder items for the evaluated models, and that the human audit and case study help interpret those results. We address the reviewer’s concerns and questions below.

W1. Fairness, separability, and the abstract. Thanks for raising this point. Under the formula in §3.5, the reported “Fairness” statistic is the complement of Separability. For a common set of NN binary-scored items, model kk has success rate aka_k and failure count ck=N(1−ak)c_k=N(1-a_k). Therefore,

\qquad \mathrm{Fairness}=1-\frac{1}{K}\sum_{k=1}^{K}\frac{|c_k-\bar c|}{N}=1-\mathrm{Sep}.$$ A benchmark on which all models fail illustrates the distinction: the scores have zero dispersion, regardless of which items became harder or by how much. In Table 2, default-setting Separability moves from 15.2 to 12.2 pp on GSM8K, 17.1 to 14.5 pp on AdvBench, and 8.5 to 7.2 pp on CommonsenseQA. Table 1, by contrast, shows increased difficulty for all six models across all three benchmarks. To address this concern about improved separability, we state the result in terms of increased difficulty and item quality in the abstract: Abstract: > We apply ArenaBencher to math problem solving, commonsense reasoning, and safety domains. **Under the default configuration, the revised benchmarks are more challenging for all six evaluated models across GSM8K, CommonsenseQA, and AdvBench Harmful Behaviors. In a human audit of 100 revised GSM8K items, 95 were judged to preserve the source objective and 96 were judged valid and correct.** We retain Separability in Table 2 and remove the duplicate Fairness column. The wording in §3.3, 3.5, and 4.1 separates the sampling rule from the metrics: To distinguish balanced sampling from an evaluation of fairness, we describe what the sampling rule controls in §3.3: > We track per-model draw counts throughout all test case updates and, at each iteration, preferentially sample under-represented models **to balance exposure during construction**. To make clear what Separability measures, we specify the scores used in each domain in §3.5: > We compute separability as the mean absolute deviation of model **success rates from their mean, using accuracy on the reasoning benchmarks and $1-\mathrm{ASR}$ on the safety benchmark**. To keep Difficulty distinct from score dispersion, we clarify its definition in §4.1: > Difficulty quantifies **the failure rate of the best-performing model in the evaluation pool: the error rate of the most accurate model on the reasoning benchmarks or the lowest attack success rate on the safety benchmark**. **W2. Model overlap, comparison budget, and contamination.** Thanks for the important concern. The same six-model pool supplies feedback and is evaluated at the end. The reanalysis above shows that the three-feedback-model configuration increased difficulty more than the one-feedback-model configuration in all 18 model–benchmark pairs. Candidate proposal and refinement-round settings were the same, but the configurations differ in the number of feedback evaluations per scored candidate. The comparison therefore describes the two configurations as run. It does not isolate the effect of aggregating distinct models at a fixed feedback budget. Lower scores on revised items measure difficulty; they are not a direct measure of memorization or contamination. To address the budget concern, we make explicit what is held fixed and what differs between the two configurations in §4.2 (“Effect of Multi-model Feedback”): > We compare the performance of ArenaBencher under two configurations, where feedback is collected from either a single model or three models during each test case update. **With candidate proposal and refinement-round settings held fixed, the three-feedback-model configuration increased difficulty more than the one-feedback-model configuration in all 18 model–benchmark pairs. Each scored candidate received three feedback-model evaluations in one configuration and one in the other. Both are evaluated on the six models in the construction pool.**
AuthorsOpenReview ↗

Response to Reviewer sT6Y 2/2

Q1. Which fairness formula is used, and is there independent evidence of fairness? Thanks for the question. The Fairness values reported in Table 2 match the §3.5 formula to rounding, consistent with the identity in W1. Balanced sampling governs how often each model contributes feedback; it is not an independent test of whether item revisions affect models evenly. We interpret the reported scores as difficulty and dispersion, rather than evidence of independent fairness or improved discrimination.

Q2. Does the difference persist for held-out models or under matched budgets? Thanks for the important question. Table 1 compares one versus three feedback models on the construction pool. It does not include an entirely held-out model or family, a matched-feedback-budget comparison, or a no-feedback condition; the observed difference applies to the configurations in W2.

Q3. How independent is the verifier, and what happens after quality filtering? Generation and verification use separate calls to the same GPT-4o-2024-08-06. Table 2 reports automatic alignment for all three domains, while the separate human audit covers GSM8K: 95 of 100 sampled revisions were judged aligned and 96 valid and correct. Figure 2 illustrates why the automatic check and the human audit are reported separately.

As the reviewer suggests, we distinguish a separate verification call from an independent verifier in the Figure 1 caption, §3.2, and §5:

A separate verification call using the same GPT-4o-2024-08-06 snapshot checks answer validity and alignment with the source objective.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading of the paper and their insightful and valuable feedback. We here answer the common question on how we assess the quality of a benchmark update.

The default ArenaBencher configuration made all three benchmarks harder for all six evaluated models. In a human audit of 100 revised GSM8K items, 95 were judged to preserve the source objective and 96 were judged valid and correct.

We also compared the one- and three-feedback-model configurations model by model based on the results reported in Table 1. The three-model configuration produced a larger change in all 18 comparisons. The table reports equal-weight averages across the six models; positive values mean the benchmark became harder.

BenchmarkMean change, 1 feedback model (pp)Mean change, 3 feedback models (pp)Models with a larger change under 3 feedback models
GSM8K (accuracy decrease)24.433.26/6
CommonsenseQA (accuracy decrease)16.623.56/6
AdvBench Harmful Behaviors (ASR increase)5.715.06/6

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.