acceptodds
Under review as a conference paper at ICLR 2027

EvalBuilder: Can LLMs Build Efficient Benchmarks?

Abstract

Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the conclusions of costly reference evaluations? We study this capability by formulating automatic benchmark development as a meta-evaluation problem, where an agent is given a task and an expensive reference benchmark and requested to construct a lower-cost proxy benchmark that faithfully distinguishes among candidate models and solutions. We introduce \method, a framework for evaluating such benchmark-generation capabilities along three key dimensions: fidelity, efficiency, and robustness. Specifically, we assess whether generated benchmarks preserve model rankings, generalize to held-out models and freshly collected responses, and provide well-calibrated and discriminative scores while substantially reducing evaluation cost. We further provide an iterative environment in which agents can refine their benchmarks under a fixed time budget. Our framework provides a systematic testbed and benchmark for understanding when current LLM agents can develop useful evaluators and how benchmark construction itself can be increasingly automated.

Primary area:
datasets and benchmarks
Keywords:
Long-horizon Agent, Benchmark, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_D2Y1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces EvalBuilder, a framework for evaluating agents’ ability to construct efficient proxy benchmarks from costly reference benchmarks. Within a 16-hour budget, an agent iteratively construct new tasks and uses a verifier. The resulting benchmark is independently evaluated using both visible and held-out models. Experiments cover two reference benchmarks: SWE-bench Verified and Terminal-Bench 2.1.

Critical strengths

An important and practically motivated research question. Constructing inexpensive evaluations that preserve the conclusions of costly benchmarks could substantially reduce model-development costs. Studying whether LLM agents can automate this process is meaningful.

Critical weaknesses
  1. Insufficient discussion of relevant prior work. The paper does not discuss AutoBencher [1] or BenchMaker [2]. AutoBencher already formulates benchmark construction as an optimization problem and uses LLMs to iteratively generate benchmarks based on model-performance. BenchMaker introduces a multidimensional framework for evaluating benchmark generators, including agreement with reference evaluations, robustness, and efficiency. Although EvalBuilder focuses on inexpensive proxies for costly agentic benchmarks, without a substantive discussion and comparison with these prior works, the novelty of its construction procedure and evaluation framework is insufficiently established.

  2. Lack of informative construction baselines. The experiments compare different construction agents but do not include simpler alternatives, such as task selection from existing benchmarks. Consequently, it is unclear whether iterative task–verifier refinement provides meaningful benefits over simpler construction strategies.

  3. Insufficient justification of benchmark size and the fidelity–cost tradeoff. The framework specifies a target of 50 tasks without explaining this choice or systematically varying the task budget. It remains unclear how the number of tasks affects ranking fidelity, held-out generalization, and evaluation cost, or whether the observed fidelity limitations could be alleviated by slightly increasing the task budget. Since constructing efficient proxies is a central objective, a controlled analysis across task budgets is needed to establish the tradeoff between evaluation cost and reliability.

[1] LI, XIANG, et al. "Autobencher: Towards declarative benchmark construction." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Yuan, Peiwen, et al. "Llm-powered benchmark factory: Reliable, generic, and efficient." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Questions for authors
  1. How does EvalBuilder differ from prior work on automatic benchmark construction? Is there evidence that jointly optimizing tasks and verifiers improves reference fidelity compared with a feedback-driven dataset-generation approach adapted to the same objective?

  2. Could you provide comparisons with simpler benchmark construction methods, such as random or strategic task selection from existing benchmarks?

Reviewer_XNpsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether LLM agents can construct efficient proxy benchmarks for expensive agent evaluations. Given a reference benchmark and reference results for visible models, an agent generates new tasks and an executable verifier under a fixed time budget. The generated benchmark is evaluated by ranking fidelity, candidate generalization, score calibration, score resolution, and evaluation cost. The authors introduce EVALBUILDER, which supports iterative benchmark construction and independent evaluation using fresh responses from visible and held-out models. Experiments on SWE-bench Verified and Terminal-Bench 2.1 show that some frontier models can construct benchmarks that partially preserve the reference ranking.

Critical strengths
  1. The paper provides a concrete evaluation framework for automatic benchmark construction. It separates construction from final evaluation, hides held-out models during construction, and collects fresh responses after submission. This setup can expose overfitting to visible models and saved responses, which is important for evaluating whether a generated benchmark measures a target capability.

  2. The evaluation is more comprehensive than ranking agreement alone. The paper separately measures pairwise accuracy, Spearman correlation, MAE, bias, and score spread. The experiments show that a benchmark can preserve some rankings while compressing or exaggerating score differences. These results support the need to evaluate ranking fidelity, calibration, resolution, and candidate generalization separately.

Critical weaknesses
  1. The motivation is unclear. When an agent can genuinely evaluate its own capabilities, why is it necessary to evaluate the agent by asking it to design a benchmark? The reason for human-designed benchmarks is precisely that agents do not have sufficient evaluation ability. However, the paper then evaluates the agent's ability to simplify the benchmark. This seems contradictory. How about using a different approach, such as directly analyzing the error trajectories? Existing studies suggest that LLMs lack a meta-ability, or what could be called taste, compared with humans [1]. I want to see more insightful analysis in your paper.

  2. This method has a weakness similar to other methods that use LLMs for evaluation. Does this process introduce the model's own inductive bias?

  3. Lines 116-125 subjectively dismiss a category of related work without experimental support. From the perspective of reducing cost, what is the difference between asking an LLM to design a benchmark and compressing a benchmark, such as SERPANT [2]?

  4. To display parentheses correctly in the references, please use \citep instead of \cite consistently.

[1] Anthropic TASTE benchmark: models judge research at chance [2] https://openreview.net/forum?id=7Y8xRnGQ47

Questions for authors

See Weaknesses.

Reviewer_fxJfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces EvalBuilder, a framework for evaluating the ability of LLM agents to automatically construct low-cost proxy benchmarks. Given reference benchmark scores and execution trajectories, agents generate evaluation tasks and verifiers, then iteratively refine them based on evaluation feedback. The authors conduct experiments on SWE-bench Verified and Terminal-Bench 2.1, assessing the generated benchmarks in terms of ranking fidelity, efficiency, and generalization. The results show that LLM agents have some potential for automated benchmark construction, but the generated benchmarks still exhibit limited ranking accuracy and generalization.

Critical strengths

The paper addresses the practical problem of automatically constructing low-cost proxy benchmarks using LLM agents, with the potential to reduce evaluation costs and human effort.

Critical weaknesses
  1. Insufficient Benchmark Validity: The paper mainly evaluates generated benchmarks based on ranking and score consistency, without sufficiently verifying whether they preserve the key capabilities and failure modes of the reference benchmarks.
  2. Limited Novelty: Task generation, verifier construction, and iterative refinement largely rely on existing LLM agents. Similar workflows have already been explored in BenchAgents and Benchmark Agent. The paper lacks a clearly novel technical mechanism.
  3. Insufficient Validation of the Optimization Process: The authors do not adequately explain why jointly refining tasks and verifiers improves benchmark quality. The paper also lacks sufficient theoretical analysis and key ablation studies to validate this design.
  4. Lack of Baseline Comparisons: The experiments mainly compare different LLM agents within the EvalBuilder framework, without direct comparisons to existing proxy benchmark methods such as PACE. Therefore, it is difficult to assess whether EvalBuilder offers meaningful advantages over existing approaches.
  5. Limited Generalization: Some generated benchmarks show substantial performance degradation on held-out models. For example, DeepSeek-V4-Pro achieves a Visible PAcc of 70.0% but only a Held-out PAcc of 13.6% on SWE-bench Verified, suggesting that the generated benchmark may overfit to the visible models.
Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.