- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work introduces a two-layer framework for generating challenging and discriminative benchmark data. The inner loop executes a benchmark generation harness, an agentic pipeline that mutates seed problems into new tasks with programmatic evaluators and validates them on two proxy models. The outer meta-harness layer has an LLM agent read the accumulated trajectory memory and propose, score, select, and apply edits to the generation harness itself. When combined, the two layers optimize for a reward that lowers average proxy-model accuracy while widening the gap between a stronger model and a weaker one. Through experiments on CodeContests and AIME-2024, the authors show that the evolved benchmarks substantially reduce the accuracy of five held-out models relative to the seed datasets.
Critical strengths
- The authors are focusing on the saturation problem of benchmarks, which is a real problem and well-motivated; the approach itself seems to be a natural generalization of Meta-Harness (Lee et. al., 2026) to the setting of synthetic benchmark generation.
Critical weaknesses
-
A lot of the increased “difficulty” looks like coming from formatting and protocol requirements rather than harder reasoning. The most common difficulty lever in both domains is “Output Contract Tightening” (91% of CodeContests tasks, 70% of AIME tasks), and the evolved tasks are dominated by techniques like delimiter and JSON schema compliance. At the same time, the share of tasks that actually test domain reasoning drops sharply, and in AIME the classic mathematical skills (closed-form derivation, exact arithmetic) nearly disappear. The paper reads this as the benchmark “outgrowing basic conceptual parsing”, but it could also mean the evolved AIME is no longer really a math benchmark. The CodeContests case study shows a related problem: the seed, a constructive math problem, is evolved into a code-auditing task where the solver must find and exploit a planted bug in a Python checker. That is a completely different skill, so the “seed lineage” is preserved only nominally and the evolved benchmark might be no longer related to the seed.
-
The search process in the outer loop is less substantial than the framing suggests, and its key step is never validated. For instance, the agent proposes several harness edits each round, and an LLM judge scores them and picks one. The chosen edit is then applied and its effect on the benchmark is measured, but the rejected ones are never run, so the system never learns whether the judge’s rankings actually track the reward it is supposed to optimize. The judge also seems to accept almost everything: Table 4 shows 87% and 83% of proposals were applied, which means little filtering is happening. These counts are also hard to reconcile with Appendix A.2.1, which describes 3 to 5 proposals per round and at most one selection per round; over 10 rounds that would imply at least 30 proposals and at most 10 selections, not 20/23 in Table 4. There is no ablation on the number of candidates, no comparison to picking a proposal at random, and no check that the judge’s scores correlate with realized reward. In practice, this procedure can be considered as an LLM editing its own prompts once per round, which is a reasonable thing to do but is a long way from the “end-to-end optimization” framing.
Questions for authors
-
Could you compare your methdology with recent related works such as AutoCode (Zhou et. al., 2025), BenchEvolver (Wu et. al., 2026), and RST (Li et. al., 2026)? What are the relative advantages of your proposed approach compared to these prior works?
-
As for the validity of the final benchmark, the paper’s own quality judge (Table 6) passes only about 70% of final-round tasks on all three criteria. Were these flagged tasks removed before the main results were computed? If not, some of the reported accuracy drop may be coming from tasks that are simply broken rather than harder; if they are removed, it would help to describe the filtering step and show how much the numbers change with and without it. Another related question: since humans agree with the judge on Competency only 68% of the time, the reported 70% rate of tasks passing all three quality criteria inherits that uncertainty. How robust is this validity estimate, and does it change a lot when human competency verdicts are used instead?
-
In this work, "benchmark harness" seems to refer to the harness that's used to generate new benchmarks instead of the harness through which users evaluate a model on a benchmark (which is the more conventional impression). Is this correct? If so, maybe it'd be helpful to clarify the differences somewhere in the paper.