acceptodds
Withdrawn from ICLR 2027

EvalScientist: Automated Benchmarking and Hypothesis-Driven Diagnosis for Frontier LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As frontier large language models (LLMs) advance, they need to be evaluated on more capabilities and in more complex settings, but expert-curated benchmarks are costly to build and quickly saturate. Automated methods reduce this cost, but they are largely limited to what available data or domain-specific formats support, and they show where models fail without diagnosing why. We introduce **EvalScientist**, an automated framework that handles natural-language evaluation requests end to end. It designs tasks using research on external sources, constructs them in a unified task representation that covers all task types, and evaluates target models on them. It then performs hypothesis-driven diagnosis, using targeted probes to test candidate weaknesses that may explain observed task failures. On 8 conventional requests covering widely benchmarked capabilities such as knowledge, reasoning, and coding, EvalScientist outperforms existing automated frameworks and GPT-6-Astra in correctness, alignment with the request, diversity, and difficulty. We further identify 15 new evaluation requests from authoritative sources such as AI safety policies of governments and frontier AI developers. These requests target frontier agentic and safety-related behaviors that no existing benchmark fully covers. EvalScientist constructs correct and challenging benchmarks for all of them, whereas existing automated frameworks either fail to accommodate these requests or generate tasks that are incorrect, misaligned with request, or overly easy. In blind pairwise human evaluation, judges prefer EvalScientist's tasks in 62.5–87.5% of comparisons against other automated frameworks and in 55–60% against expert-curated benchmarks. Hypothesis-driven diagnosis yields findings supported by correct and aligned tasks, whereas simply generating more tasks like those the target models fail drifts toward ambiguous tasks, such that low scores increasingly reflect task defects rather than model weaknesses.

Primary area:
datasets and benchmarks
Keywords:
LLM Evaluation, Multi-agent System, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_LNjqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvalScientist, a framework that integrates evaluation task design, automatic construction, and model weakness diagnosis. Through a unified task representation and hypothesis-driven iterative testing, it enables the automatic generation of complex benchmarks along with analyses of model weaknesses. Experiments show that the method overall outperforms existing approaches on both conventional and frontier evaluation tasks.

Critical strengths
  1. Experiments show that EvalScientist's evaluation quality is superior to existing methods, suggesting that EvalScientist has some practical value.
  2. The paper is well-written and easy to follow.
Critical weaknesses

My main concern is that the paper reads more like a technical optimization than a valuable research finding. The authors identify shortcomings in existing automated evaluation systems but do not analyze the causes of these shortcomings. Moreover, the design ideas behind Planner, Builder, and Analyzer have already been proposed in many similar works. The system design itself lacks conceptual novelty and feels more like a combination of existing technical routes.

Questions for authors
  1. I suppose that the ethics statement and reproducibility statement should be placed in the appendix, and I wonder whether this conflicts with the page limit requirement.
  2. Data construction is inherently harder than data evaluation. EvalScientist generates evaluation data directly from requirements using LLMs, yet it does not provide a fine-grained analysis of the reliability of the generated data.
  3. The authors use relatively few models in their experiments, which to some extent weakens the reliability of their conclusions.
  4. The benchmark is not released during review, which makes the quality of the benchmark should be further evaluated.
1 comment

Novelty of the design

We thank the reviewer for the candid assessment. We agree that the submitted version did not analyze why automatically built tasks fall short, and that its design offered limited conceptual novelty over existing systems. Since submission we have studied these shortcomings and substantially redesigned the Builder accordingly, for example by introducing new agent roles and a new agent topology. To our knowledge, these designs and the insights behind them go beyond prior automated benchmark builders, and we will present them in a future version of this work.

Placement

Following the guidelines, we placed the Ethics Statement and Reproducibility Statement at the end of the main text, before the references. The guidelines state that the optional ethics statement “will not count toward the page limit, but should not be more than 1 page” and that the reproducibility statement “is not part of the main text and therefore will not count toward the page limit”, so the placement is consistent with the page limit.

Reviewer_VvnLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvalScientist, an automated framework for constructing benchmarks and diagnosing LLM weaknesses from natural-language evaluation requests. The framework consists of three components: a Planner that designs evaluation tasks using external research, a Builder that constructs and validates tasks through a unified harness, and an Analyzer that identifies potential model weaknesses through hypothesis-driven diagnostic probing. The authors evaluate EvalScientist on 23 requests, including 8 conventional and 15 frontier evaluation scenarios, and compare it against several automated benchmark generation frameworks and direct generation with GPT-6-Astra. Results demonstrate improvements in benchmark correctness, alignment, diversity, and difficulty, supported by human evaluations. The paper further shows that hypothesis-driven diagnosis produces more reliable diagnostic evidence than simply generating additional tasks based on model failures.

Critical strengths

S1. Comprehensive evaluation with human validation.

The paper evaluates benchmark quality using multiple metrics, including correctness, alignment, diversity, difficulty, and contamination resistance, complemented by human evaluations.

S2. Strong empirical results and extensive baselines.

The paper compares against several relevant baselines and demonstrates consistent improvements across multiple evaluation metrics.

Critical weaknesses

W1. Insufficient downstream validation.

Although the paper uses LLM judges, human preferences, and target-model scores, it does not sufficiently demonstrate whether the generated benchmarks reliably identify model weaknesses. Further experiments on model discriminability, ranking consistency, and generalization of diagnosed weaknesses would strengthen the claims.

W2. Limited evaluation request coverage.

The experiments cover only 23 evaluation requests, limiting the generalizability of the proposed framework to broader domains and evaluation scenarios.

W3. Unclear computational budget and comparison fairness.

Although the methods generate similar numbers of tasks, the paper does not report comparable computational costs. Differences in model calls, token usage, and execution time may confound the performance comparisons.

Questions for authors

Q1. Can the authors demonstrate that the generated benchmarks reliably expose model weaknesses that generalize to independent tasks?

Q2. What are the computational budgets of EvalScientist and each baseline? How would their performance compare under matched inference budgets?

Q3. How well does the framework generalize to evaluation requests beyond the 23 selected cases?

1 comment

We thank the reviewer for the careful reading and the constructive questions. Testing whether the weaknesses diagnosed by EvalScientist generalize to independent tasks, and comparing EvalScientist with the baselines under matched computational budgets, are our future work. Thank you for your time and for the helpful feedback.

Reviewer_8FLrOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed EvalScientist, an automated framework that can accept a natural language evaluation request as input and construct benchmarks that grade target models performance based on different dimensions based on the request while also generating explanations for these failures. Specifically, it does web search on relevant external resources and constructs the task designs based on a unified task representation format. This unified task representation covers diverse task types, from multiple-choice questions to agentic tasks in executable environments. The framework also generates diagnostic explanations for the models' observed failures, specifically by generating hypothetical candidates on the model's failures, then constructing targeted probes to verify these hypotheses. The authors test the framework on 8 conventional evaluation requests used in existing literature, while also adding 15 new requests that they claim to be “frontier” and derived from authoritative sources. EvalScientist is reported to outperform other baselines on task correctness, diversity, difficulty; while also being preferred by human judges.

Critical strengths
  1. Well-motivated and logical framework: The paper unifies multiple necessary processes of benchmarking into a single autonomous framework: deriving benchmark objectives, researching relevant external resources, curating an unified set of tasks and metrics, observing results and generating potential hypotheses, iteratively testing these hypotheses. The hypothesis-driven diagnosis in particular is the most interesting one, allowing the framework to offer explanations for their failures.
  2. Diverse task support with clear constraint: The integration of different task types, from typical QA questions to complex agentic tasks in simulated environments, all unified under a representation is appreciated. The QC component helps ensure the task's executability, demonstrated by EvalScientist's 100% yield.
Critical weaknesses
  1. Missing evaluation for the diagnostics: The hypothesis-driven diagnostic component is one of the paper's core contributions and novelty points over previous works, however the quality of this component is not actually evaluated and verified properly. Table 2 reports the quality of the evidence tasks, but this does not reflect whether the actual diagnostic findings are correct and useful. There are no gold diagnostic labels to compare them with, no metrics to judge whether these explanations are actually correct or useful. The evaluated metrics (yield, difficulty, diversity, correctness, non-contamination) only evaluates on the output of the task generated by the builder, so the quality of the diagnosis parts have no influence on these scores either. One of the paper's most crucial components and selling points are not demonstrated to be accurate or necessary. Without this contribution, the paper seems like an incremental work to other automated benchmark generation methods that also offer benchmarked results, so this component has to be tested rigorously and demonstrate its usefulness.

  2. Biased and unreliable experimental results: The reported results and comparisons on frontier requests in Table 1 are not apples-to-apples. Other baselines like Gym-Anything, Petri… are all only run on a subset of tasks, with the figures presented in Table 1 are from those chosen tasks only. This makes the comparisons unreliable. The paper asserts that these methods only work for these certain tasks, but doesn't offer any demonstrations or results proving this, and just decide to leave them out without attempted runs or results. Are there any reasons for the authors not to run the baselines on the tasks they deemed not compatible, and just report a 0 score for all metrics if they fail catastrophically? Additionally, the experiments are done on rather small sample sizes: 40 tasks for 8 conventional requests (so 40x8=320 samples) and 10 for 15 frontier requests (10x15=150), and other methods only run a subset of these requests. All the reported figures are only run once, graded by a single non-deterministic API judge GPT-6 Astra, without any confidence interval. This raises concerns on the statistical significance and reproducibility of the results From interpreting the Table 1’s reported figures, EvalScientist’s only significant consistent improvements across both sets of requests is Diversity. Petri, AutoControl Arena, Direct already got near-perfect yield results; Direct are close to EvalScientist on Conventional Requests in terms of Correctness (90.55 vs 91.33) Alignment (85.94 vs 87.66) on a small sample size without any confidence intervals, while Petri outperforms it on Alignment and Non-cont. All the other baselines' outputs are converted to EvalScientist's native task representation, which might potentially favor its native method, EvalScientist, more during grading. The Difficulty metrics are susceptible to broken tasks as the authors stated. A method can generate many broken, unaccomplishable tasks and still obtain a very high score, demonstrated by Gym-Anything For table 2, the paper scores EvalScientist only on tasks the Analyzer kept as evidence, while dropping the broken ones. Meanwhile, AutoBencher and Gym-Anything are scored on every single task generated by these baselines without any filtering, which can make this result biased.

Questions for authors

Q1 How to evaluate the correctness and quality of EvalScientist's generated diagnostics Are there any ways the authors can provide this assessment, either by constructing a gold dataset with accurate diagnosis labels, or through expert judges' validation? Or maybe through some simulated experiments that can reliably demonstrate whether these diagnostics are accurate and useful?

Q2 Why didn't the authors report the baselines' results for all requests? The authors ran each baseline on a subset of benchmarks only based on their perceived fitness of that baseline. However, I believe this approach is not justified, as that leads to uneven and unreliable comparisons in the paper’s headline results. Why don't they just run all requests, and assign a 0 score for the ones that each baseline fails? Or potentially, could the authors report EvalScientist’s metrics restricted to each baseline’s subset for a like-for-like comparison, and/or report attempted runs of each baseline on the remaining requests, with Yield and failure modes?

Q3 How to ensure statistical significance of the reported results: The obtained results are done on a single run over a small set of samples, with a single LLM Judge (GPT 6 Astra). Can the authors run each sample with different seeds/or different backbone models, potentially combining different powerful LLM judges (Claude, Gemini 4, kimi…) and report the confidence intervals and p-values?

Q4 Experiments and analysis on the effect of task budgets The paper set a fixed task budget for EvalScientist on all experiments. What are the effects of different task budgets (10, 20, 50, 100, 200, or further) on the evaluation metrics? How do the computing costs and latency scale in parallel with task budgets and their performance?

Q5 Compute cost of EvalScientist and compute-equivalent comparisons: EvalScientist contains many components and steps in its pipelines with complex interactions. What are EvalScientist's compute cost (number of tokens, memory, latency), as well as its different components (Planner, Builder, Analyzer…) and steps (research, building benchmark, QC repair, iterative diagnostic rounds)? How do EvalScientist’s compute cost compare with other baselines? Do EvalScientist's improvements actually come from a better design, and not simply extra token spending?

Q6 EvalScientist's generalizability for other LLMs : The framework mainly uses DeepSeek-V4.1-Flash for most of its components. Does the method generalize to other classes of models like Qwen, Kimi… with good performance? Any issues with the task's format and executability when switching to other models?

Q7 Are the LLM Judge (GPT 6 Astra) reliable? While GPT 6 Astra is a powerful state-of-the-art model, benchmark and diagnosis generation is a very challenging task requiring expert level knowledge and skills. Are the judgements by GPT 6 Astra proven to be reliable and accurate? Can the human judges evaluate GPT 6 Astra's judgement in terms of correctness? Or alternatively, do human judges have high agreement rate with the LLM judge?

Q8 Frontier task validity: The authors curated 15 requests, which they claim to be from authoritative sources that deal with frontier demands. Are these requests’ usefulness validated by any experts, or have anything to ensure they are actually representative of these authoritative sources’ demands?

Q9 Unified Task Representation Novelty and Comparison METR [1] and Inspect [2] already introduced a unified representation for LLM evaluation. How are their formats different from these standards? Do they offer any improvements or novelty over these popular works?

Q10 Novelty - Missing references AutoDetect [4] also has the similar idea of generating hypotheses of the target model's potential issues, even though it does not test and confirm them. ACD [3] also identifies potential failure modes of the model, but these come from clustering thousands of generated tasks into broader categories instead of natural language hypotheses, and they also don't test these theories either. Please discuss and position the work against these.

[1] METR Task Standard, 2024b. URL https://github.com/METR/task-standard/blob/main/STANDARD.md. Version 0.5.0. [2] UK AI Security Institute. Inspect AI: Framework for large language model evaluations. Software, 2024. URL https://inspect.aisi.org.uk/. Released May 10, 2024. [3] Lu, Cong, Shengran Hu, and Jeff Clune. "Automated capability discovery via foundation model self-exploration." arXiv preprint arXiv:2502.07577 (2025). [4] Jiale Cheng, Yida Lu, Xiaotao Gu, Pei Ke, Xiao Liu, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. 2024. AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6786–6803, Miami, Florida, USA. Association for Computational Linguistics.

1 comment

We thank the reviewer for the detailed review and the concrete suggestions. Regarding the frontier comparison, the subset for each baseline was determined by what that framework is able to construct, and the assignment was fixed before any results were obtained, rather than cherry-picked.

  • Gym-Anything turns software applications into desktop operation tasks, so it was assigned the requests about work in software environments (D1–D4 and D6).
  • Petri audits a target through conversations in which an auditor model simulates the other roles, so it was assigned the multi-role interaction requests (D7–D9 and D13).
  • AutoControl Arena generates stateful environments for behavioral assessment, so it was assigned the remaining behavioral requests (D5, D10–D12, D14 and D15).

Experiments with more underlying models and agent harnesses over repeated runs, and a direct check of whether the Analyzer’s diagnoses are correct, are part of our future work. We appreciate the effort you devoted to reviewing our paper.

Reviewer_5RcTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents EvalScientist, an agentic framework that turns a natural-language evaluation request into a benchmark and then into diagnostic findings. A Planner decomposes the request into evaluation dimensions and task designs, using web research to find resources; a Builder writes each task in a unified, executable task specification, which can include containers or virtual machines, other agents and scripted events, with quality control that runs scripted correct and incorrect solutions; and an Analyzer separates model failures from task defects, proposes hypotheses about model weaknesses, and requests targeted probes over several rounds. The evaluation covers 8 conventional and 15 frontier requests, with an LLM judge (GPT-6-Astra) scoring correctness, alignment and diversity, three target models determining difficulty, a contamination search, a blind pairwise human study, and a comparison of the tasks behind the diagnostic findings with tasks from two baselines that keep generating what the target model fails.

Critical strengths

The problem is timely and the ambition is right. Evaluation is now a bottleneck for frontier models, and the requests that matter most, involving agentic, multi-agent and safety-related behavior, need executable environments that most automated benchmark builders cannot produce; the framework handles these within one task representation. The comparison is also broad: five existing frameworks and direct generation by a frontier model, a contamination measure, a human study, and ablations that isolate the research and harness components (frontier correctness falls from 85.83 to 43.33 without the harness, Table 3). The observation that repeatedly generating tasks a model fails drifts toward ambiguous ones (p. 38) is useful in its own right.

Critical weaknesses

The pipeline is validated only at its output. The Planner, quality control and the Analyzer have no metrics of their own; the number of tasks that quality control repaired or excluded is not reported (p. 5), and the ablation removes the harness tools and quality control together (Table 3). The Analyzer's decision of whether a failure reflects a model weakness or a task defect determines which results count as evidence (p. 6), yet its accuracy is never measured. The diagnostic findings are not tested for correctness either. The only quantitative evidence, Table 2, compares the judged quality of the tasks cited as evidence with the baselines' third-round tasks; this measures task quality rather than whether a finding is true, and the two sides are not comparable, since EvalScientist's tasks have already been filtered by the Analyzer. Beyond that there are three qualitative cases, one with no finding and none naming the target model (pp. 35-37). For a framework presented as an evaluation scientist, its central capability therefore remains untested, and the claims that evaluations can be "established quickly" and that the framework helps improve models (p. 9) are not measured.

The quality of the generated tasks is established almost entirely by an LLM judge and a preference study rather than by direct verification. The judge scores 40 of the 200 tasks per conventional request and 10 of the 20 per frontier request (pp. 6-7), with no reason given for sampling, and the paper itself states that quality control "cannot ensure that it is unambiguous or correctly answered" (p. 9). The human study does not close this gap. Judges give one overall preference that folds correctness together with alignment and difficulty (p. 27), the protocol does not include running the executable tasks, and agreement and calibration are not reported. Because EvalScientist's tasks are written from the very request shown to the judges, the comparison favors them by construction, particularly against expert-curated benchmarks that were not written for that request. Difficulty is defined from the target models' failures, which rewards defective tasks as much as hard ones; the paper's own Table 2 shows this for Gym-Anything, with the highest difficulty and the lowest correctness. The trade-off between correctness and difficulty is not quantified, and with three target models it cannot be checked whether the generated benchmarks rank models the way established benchmarks do.

The robustness of the results is not established. All frameworks share one underlying model, DeepSeek-V4.1-Flash, which is also one of the three target models (pp. 6-7); every experiment is run once (only a seed of 42 is mentioned, p. 24); and the stability of construction and diagnosis across repeated runs is not reported. The underlying model is never changed, the target models always run in one agent harness (OpenClaw, p. 7), and the direct-generation baseline differs from EvalScientist in both model and harness, so the effect of the workflow cannot be separated from that of the model. Against this baseline, the margin on the conventional requests is small for correctness (91.33 vs. 90.55) and alignment (87.66 vs. 85.94). Contamination is checked only through verbatim web matches, where direct generation scores better (91.02 vs. 82.58), and how quickly the benchmarks become outdated is not addressed. The limitations section names the single model and the reliance on the judge (p. 9) but offers no evidence on either.

Beyond these points, the contribution is mainly the design of an agent workflow, and the relation of its unified task representation to existing standards such as Inspect and the METR Task Standard is not sufficiently analyzed (p. 3, p. 19). The frontier comparison aggregates EvalScientist over all 15 requests and each baseline over its assigned subset (p. 27); on the requests assigned to Gym-Anything, that baseline's tasks are harder by the paper's own measure (47.21 vs. 26.45, computed from Table 14). There is no statement on code release, the prompts of the components are not given, and no cost or time is reported.

Questions for authors

How accurate is the Analyzer in separating model failures from task defects, and can the diagnosis recover a weakness that is known in advance?

How stable are the constructed benchmarks and the diagnostic findings across repeated runs, different underlying models and different agent harnesses?

What do constructing a benchmark and running the diagnosis cost, and will the code, prompts and generated tasks be released?

1 comment

We thank the reviewer for the thorough and constructive review. Evaluating EvalScientist across additional underlying models, agent harnesses, and repeated runs, together with a direct validation of the Analyzer’s diagnostic ability, is our future work. Thank you again for the time and care you put into this review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.