acceptodds
Withdrawn from ICLR 2027

MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Rapid progress in Large Language Models (LLMs) is saturating static benchmarks faster than they can be designed. While existing automated evolution frameworks attempt to generate harder questions by perturbing individual tasks, they remain constrained by rigid, hard-coded generation rules. Moving beyond the evolution of isolated tasks, we propose to optimize the benchmark generation workflow itself end to end with MetaBench-Harness, a dual-loop search framework. Specifically, the inner loop utilizes a benchmark harness to generate a new benchmark in each round, while the outer meta-harness orchestration layer iteratively refines and searches over harness implementations based on historical evolution trajectories. By applying MetaBench-Harness to the competitive programming CodeContests and Olympiad mathematics AIME-2024 datasets, we demonstrate that the evolved benchmarks are challenging and discriminative for frontier models. Trajectory and quality analyses verify that MetaBench-Harness enables multi-dimensional evolution, steadily improving evolution reasonableness, benchmark competency, and evaluator robustness across successive rounds. Furthermore, case studies reveal its effective utilization of diverse difficulty levers to reframe problems and elevate required capabilities. Ultimately, this work provides a solution to the pressing challenge of benchmark saturation.

Primary area:
datasets and benchmarks
Keywords:
Benchmark Evolution, Self-Evolution, Large Language Models, Dynamic Task Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_R42nOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces a two-layer framework for generating challenging and discriminative benchmark data. The inner loop executes a benchmark generation harness, an agentic pipeline that mutates seed problems into new tasks with programmatic evaluators and validates them on two proxy models. The outer meta-harness layer has an LLM agent read the accumulated trajectory memory and propose, score, select, and apply edits to the generation harness itself. When combined, the two layers optimize for a reward that lowers average proxy-model accuracy while widening the gap between a stronger model and a weaker one. Through experiments on CodeContests and AIME-2024, the authors show that the evolved benchmarks substantially reduce the accuracy of five held-out models relative to the seed datasets.

Critical strengths
  1. The authors are focusing on the saturation problem of benchmarks, which is a real problem and well-motivated; the approach itself seems to be a natural generalization of Meta-Harness (Lee et. al., 2026) to the setting of synthetic benchmark generation.
Critical weaknesses
  1. A lot of the increased “difficulty” looks like coming from formatting and protocol requirements rather than harder reasoning. The most common difficulty lever in both domains is “Output Contract Tightening” (91% of CodeContests tasks, 70% of AIME tasks), and the evolved tasks are dominated by techniques like delimiter and JSON schema compliance. At the same time, the share of tasks that actually test domain reasoning drops sharply, and in AIME the classic mathematical skills (closed-form derivation, exact arithmetic) nearly disappear. The paper reads this as the benchmark “outgrowing basic conceptual parsing”, but it could also mean the evolved AIME is no longer really a math benchmark. The CodeContests case study shows a related problem: the seed, a constructive math problem, is evolved into a code-auditing task where the solver must find and exploit a planted bug in a Python checker. That is a completely different skill, so the “seed lineage” is preserved only nominally and the evolved benchmark might be no longer related to the seed.

  2. The search process in the outer loop is less substantial than the framing suggests, and its key step is never validated. For instance, the agent proposes several harness edits each round, and an LLM judge scores them and picks one. The chosen edit is then applied and its effect on the benchmark is measured, but the rejected ones are never run, so the system never learns whether the judge’s rankings actually track the reward it is supposed to optimize. The judge also seems to accept almost everything: Table 4 shows 87% and 83% of proposals were applied, which means little filtering is happening. These counts are also hard to reconcile with Appendix A.2.1, which describes 3 to 5 proposals per round and at most one selection per round; over 10 rounds that would imply at least 30 proposals and at most 10 selections, not 20/23 in Table 4. There is no ablation on the number of candidates, no comparison to picking a proposal at random, and no check that the judge’s scores correlate with realized reward. In practice, this procedure can be considered as an LLM editing its own prompts once per round, which is a reasonable thing to do but is a long way from the “end-to-end optimization” framing.

Questions for authors
  1. Could you compare your methdology with recent related works such as AutoCode (Zhou et. al., 2025), BenchEvolver (Wu et. al., 2026), and RST (Li et. al., 2026)? What are the relative advantages of your proposed approach compared to these prior works?

  2. As for the validity of the final benchmark, the paper’s own quality judge (Table 6) passes only about 70% of final-round tasks on all three criteria. Were these flagged tasks removed before the main results were computed? If not, some of the reported accuracy drop may be coming from tasks that are simply broken rather than harder; if they are removed, it would help to describe the filtering step and show how much the numbers change with and without it. Another related question: since humans agree with the judge on Competency only 68% of the time, the reported 70% rate of tasks passing all three quality criteria inherits that uncertainty. How robust is this validity estimate, and does it change a lot when human competency verdicts are used instead?

  3. In this work, "benchmark harness" seems to refer to the harness that's used to generate new benchmarks instead of the harness through which users evaluate a model on a benchmark (which is the more conventional impression). Is this correct? If so, maybe it'd be helpful to clarify the differences somewhere in the paper.

Reviewer_aTshOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The development of LLMs is saturating static benchmarks. This paper proposes to optimize the benchmark generation workflow end-to-end. Under this framework, the inner loop utilizes a benchmark harness to generate a new benchmark in each round; the outer meta-harness orchestration layer iteratively refines and searches over harness implementations based on historical evolution trajectories. The authors apply the method to single-step math and coding benchmarks. The results demonstrate that the evolved benchmarks are challenging and discriminative for frontier models.

Critical strengths

The idea is well-motivated. The paper shifts benchmark evolution from perturbing individual tasks to optimizing the generation workflow. The dual-loop design enables the harness to diagnose and repair its own defects.

The ablations reliably isolate the contributions of the outer loop and execution feedback. Held-out models test generalization beyond the in-loop model proxies.

Critical weaknesses

The benefit of harness evolution is not cleanly established. A sensible baseline seems to be missing: letting a general-purpose coding agent (Claude Code, Codex) freely iteratively harden the benchmark, which is typically the case how researchers develop and iterate their benchmarks nowadays. The extra compute spent by the outer loop seems to be inadequately accounted for.

Experiments are limited to single-step tasks. Agentic tasks (which may require synthesizing sandboxed environments, tools, and multi-turn interaction) are arguably the focus of frontier model development and research, yet are not included.

The proxy reward is gameable in principle. Nothing in the objective prevents wrong or degenerate questions that lower scores, or overfitting to the idiosyncratic gap between the two proxies.

Questions for authors

Benchmarks aim to assess practically relevant model capabilities through sufficiently challenging tasks. Synthesizing tasks with this pipeline seems reasonable for training. Benchmark construction, however, typically involves careful design and human review to ensure task quality and practical relevance. Given these requirements, what motivates using this pipeline to construct benchmarks rather than generate training tasks? How do you envision the community using the resulting synthetic benchmarks?

Reviewer_pdfzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MetaBench-Harness optimizes benchmark-generation workflows through a dual loop. The outer loop revises the harness using historical artifacts and execution feedback, while the inner loop generates seed-derived tasks and programmatic evaluators, validates models, and manages task lifecycles. Its main contribution is extending optimization from individual questions to the generation and evaluation workflow itself. Experiments evolve CodeContests and AIME-2024 over ten and five rounds, producing final suites of 100 and 30 tasks. Comparisons, ablations, held-out model evaluations, and quality audits show reduced pass rates, particularly in mathematics, although the validity of these reductions as capability measurements remains insufficiently established.

Critical strengths
  1. Optimizing the complete evaluation workflow is practically valuable. The framework can revise generation, scoring, execution, and feedback mechanisms, addressing problems that question mutation alone cannot resolve. Appendix B.2.2 reports formatting failures in 121/180 initial coding model–task pairs, followed by substantial score recovery after measurement correction. This concretely demonstrates the potential value of harness evolution for evaluation validity.
  2. Evaluation extends beyond the optimization models. The evaluation covers two domains and multiple held-out models, including reasoning and non-reasoning modes. For example, Table 2 shows AIME reasoning accuracy decreasing from 0.900 to 0.533 for Kimi and from 0.900 to 0.333 for DeepSeek. Thus, the observed challenge extends beyond the proxy models, supporting potential transfer.
Critical weaknesses
  1. Final benchmark validity remains insufficiently established. Table 6 reports that only 70% of final tasks in either benchmark jointly pass reasonableness, competency, and evaluator-robustness checks. Human–judge agreement for competency is only 68%, with insufficient reporting of audit sample sizes and adjudication. This does not imply that the remaining 30% are all unsolvable, but it leaves important quality concerns unresolved. Without recomputing model results on independently validated tasks, lower accuracy cannot be confidently attributed to genuine reasoning difficulty.
  2. The controlled benefit of harness optimization is not adequately demonstrated. Table 1 primarily demonstrates lower pass rates, which alone do not establish improved discrimination or benchmark quality. Section 3 does not instantiate r(s1,s2)r(s_1,s_2), while candidate selection uses multidimensional LLM judgments with unspecified weights. Total search budgets, independent run counts, and uncertainty definitions are also unclear. Consequently, the results do not adequately isolate harness improvement from additional search effort or stochastic variation.
Questions for authors
  1. Do Tables 1–3 include tasks failing the quality checks in Table 6? Please clarify the major failure categories, human-audit sample sizes, and adjudication procedure. And do difficulty, model discrimination, and improvements over the fixed harness persist on independently validated tasks?
  2. Section 4 states a uniform 32,000-token cap for held-out models, whereas Appendix B.1 gives Qwen a 96,000-token reasoning phase plus a 32,000-token extraction phase. Which protocol produced each table, and were comparison methods evaluated identically?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.