- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces EvalBuilder, a framework for evaluating agents’ ability to construct efficient proxy benchmarks from costly reference benchmarks. Within a 16-hour budget, an agent iteratively construct new tasks and uses a verifier. The resulting benchmark is independently evaluated using both visible and held-out models. Experiments cover two reference benchmarks: SWE-bench Verified and Terminal-Bench 2.1.
Critical strengths
An important and practically motivated research question. Constructing inexpensive evaluations that preserve the conclusions of costly benchmarks could substantially reduce model-development costs. Studying whether LLM agents can automate this process is meaningful.
Critical weaknesses
-
Insufficient discussion of relevant prior work. The paper does not discuss AutoBencher [1] or BenchMaker [2]. AutoBencher already formulates benchmark construction as an optimization problem and uses LLMs to iteratively generate benchmarks based on model-performance. BenchMaker introduces a multidimensional framework for evaluating benchmark generators, including agreement with reference evaluations, robustness, and efficiency. Although EvalBuilder focuses on inexpensive proxies for costly agentic benchmarks, without a substantive discussion and comparison with these prior works, the novelty of its construction procedure and evaluation framework is insufficiently established.
-
Lack of informative construction baselines. The experiments compare different construction agents but do not include simpler alternatives, such as task selection from existing benchmarks. Consequently, it is unclear whether iterative task–verifier refinement provides meaningful benefits over simpler construction strategies.
-
Insufficient justification of benchmark size and the fidelity–cost tradeoff. The framework specifies a target of 50 tasks without explaining this choice or systematically varying the task budget. It remains unclear how the number of tasks affects ranking fidelity, held-out generalization, and evaluation cost, or whether the observed fidelity limitations could be alleviated by slightly increasing the task budget. Since constructing efficient proxies is a central objective, a controlled analysis across task budgets is needed to establish the tradeoff between evaluation cost and reliability.
[1] LI, XIANG, et al. "Autobencher: Towards declarative benchmark construction." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Yuan, Peiwen, et al. "Llm-powered benchmark factory: Reliable, generic, and efficient." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
Questions for authors
-
How does EvalBuilder differ from prior work on automatic benchmark construction? Is there evidence that jointly optimizing tasks and verifiers improves reference fidelity compared with a feedback-driven dataset-generation approach adapted to the same objective?
-
Could you provide comparisons with simpler benchmark construction methods, such as random or strategic task selection from existing benchmarks?