- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ArenaBencher, a framework for automatic benchmark evolution using feedback from multiple models. It first extracts the ability tested by each original example, generates and verifies new variants, and then selects difficult examples based on shared failures across several models. The selected examples are further used as in-context demonstrations for iterative refinement. The main idea is interesting, but the current method and experiments are too narrow to fully support the paper’s claims.
Critical strengths
- The paper studies an important problem: static benchmarks may become saturated or contaminated over time.
- Using shared failure signals from multiple models to guide benchmark evolution is a useful insight. It may reduce overfitting to the behavior of a single model.
- The framework has a clear pipeline, including ability extraction, candidate generation, verification, multi-model scoring, and iterative refinement.
Critical weaknesses
- The method is relatively simple and heuristic. It mainly combines LLM-based rewriting, LLM-based verification, and failure-based candidate selection. The iterative use of difficult examples as demonstrations is also straightforward, so the technical contribution is limited.
- The selected benchmarks are relatively easy. In particular, GSM8K may not be a strong test bed for benchmark evolution because many recent models already perform well on it. For mathematical reasoning, it would be more convincing to study harder benchmarks such as MATH-500, AIME, OlympiadBench, or Omni-MATH. This is important for showing whether the method can still find meaningful weaknesses when the original problems are already difficult.
- The evaluated models are relatively weak. The model pool mainly contains 1B–7B Llama, Qwen, and Mistral models. The results show that ArenaBencher can expose weaknesses of small models, but it remains unclear whether the method is useful for stronger and more recent models.
- The experimental scope is narrow. Only one benchmark is used for each domain, and there are no experiments on coding, long-context reasoning, difficult knowledge tasks, or advanced mathematics. More importantly, the generated benchmark is evaluated on the same model pool used during construction. There is no evaluation on held-out models.
Questions for authors
-
How many examples from each benchmark were actually evolved? Will the complete generated datasets and configurations be released?
-
Does the increase in difficulty remain when the generated examples are evaluated on stronger models that were not used during benchmark construction?
-
Can the proposed method generalize to more challenging benchmarks, such as MATH-500, AIME, OlympiadBench, or Omni-MATH? In particular, can it generate meaningful and valid hard examples when the original test cases are already difficult?
2 comments
Thank you for the clear response. It helps clarify the scope of the current results. However, the main concerns remain. I will keep my current score.
Response to Reviewer H3KM
We thank the reviewer for the detailed feedback. We appreciate the recognition of the significance of our research question and the clarity of the proposed pipeline. We address the reviewer’s concerns and questions below.
W1. Technical contribution of the pipeline. Thanks for providing the opportunity for us to highlight our contribution and novelty. The design contribution is how the stages work together: extraction specifies what should remain fixed; verification filters candidate rewrites against that objective; feedback from multiple models ranks the surviving candidates; and the selected candidates inform the next round. The distinction from difficulty-only selection is that a difficult candidate first has to pass the task-preservation check.
W2. Whether GSM8K is a meaningful test bed. GSM8K provides a setting where annotators can check both the answer and whether a rewrite still requires the same mathematical idea. In this setting the default update lowers accuracy for every evaluated model; for example, Llama-3.2-3B moves from 74.1% to 26.4%. The accompanying audit gives item-level evidence that the revisions generally retain the source objective. On advanced mathematics tasks, the same criteria would need to be checked at the level of the intended mathematical idea and the validity of the solution.
W3. Model scale and recency. The six models span Llama, Qwen, and Mistral variants from 1B to 7B. The direction of the change is consistent across these models, including those with higher source-benchmark scores. We claim that it is evidence across the tested pool, not a comparison with recent frontier models.
W4. Breadth of tasks and held-out evaluation. Arithmetic, commonsense reasoning, and safety let us examine the same pipeline under three different definitions of task success. With the given suite of model pool, advanced tasks such as coding, long-context reasoning, difficult knowledge tasks, or advanced mathematics would be way beyond the capabilities of the models. The evaluation uses the construction pool, so the results show the consistency of the difficulty increase within that pool rather than transfer to other models.
Q1. How many examples were evolved, and what is available? Thanks for the question. We evolved every item in the evaluation sets for GSM8K, CommonsenseQA, and AdvBench Harmful Behaviors. The 100 items were randomly sampled from the evolved GSM8K set for human annotation only. They were not a limit on how many items we evolved.
We will release the complete generated datasets and the pipeline used to construct them. The settings for the reported runs are documented in §4.1 and Appendix D. Because the construction procedure uses feedback from a particular model pool, the released datasets are best interpreted as tests of what those models can do on revised items that target the original skills. The pipeline is the more significant contribution since users can choose a model pool at a different capability level and use our proposed pipeline to construct an updated benchmark for that pool.
Q2. Do the results hold for stronger models outside the construction pool? Thanks for the important question. The current results show increases in difficulty for all six evaluated 1B–7B models. No stronger model entirely outside the construction pool is included in Table 1, so the results do not establish that transfer.
Q3. Can the approach produce valid updates to already-hard math problems? Thanks for the important question. ArenaBencher is most useful when the target model pool is approaching saturation on the original benchmark. If a benchmark such as MATH-500, AIME, OlympiadBench, or Omni-MATH still challenges and distinguishes the target models, making it harder is less useful. There is also a verification challenge where, for an already-difficult problem, it can be hard to establish that the rewritten problem has a correct solution and still tests the intended mathematical idea. Lower solver accuracy cannot tell us whether the rewrite is harder for the right reason or simply flawed. For those reasons, we would not recommend applying the current pipeline as-is to already-hard math benchmarks.