Q1 How to evaluate the correctness and quality of EvalScientist's generated diagnostics Are there any ways the authors can provide this assessment, either by constructing a gold dataset with accurate diagnosis labels, or through expert judges' validation? Or maybe through some simulated experiments that can reliably demonstrate whether these diagnostics are accurate and useful?
Q2 Why didn't the authors report the baselines' results for all requests? The authors ran each baseline on a subset of benchmarks only based on their perceived fitness of that baseline. However, I believe this approach is not justified, as that leads to uneven and unreliable comparisons in the paper’s headline results. Why don't they just run all requests, and assign a 0 score for the ones that each baseline fails? Or potentially, could the authors report EvalScientist’s metrics restricted to each baseline’s subset for a like-for-like comparison, and/or report attempted runs of each baseline on the remaining requests, with Yield and failure modes?
Q3 How to ensure statistical significance of the reported results: The obtained results are done on a single run over a small set of samples, with a single LLM Judge (GPT 6 Astra). Can the authors run each sample with different seeds/or different backbone models, potentially combining different powerful LLM judges (Claude, Gemini 4, kimi…) and report the confidence intervals and p-values?
Q4 Experiments and analysis on the effect of task budgets The paper set a fixed task budget for EvalScientist on all experiments. What are the effects of different task budgets (10, 20, 50, 100, 200, or further) on the evaluation metrics? How do the computing costs and latency scale in parallel with task budgets and their performance?
Q5 Compute cost of EvalScientist and compute-equivalent comparisons: EvalScientist contains many components and steps in its pipelines with complex interactions. What are EvalScientist's compute cost (number of tokens, memory, latency), as well as its different components (Planner, Builder, Analyzer…) and steps (research, building benchmark, QC repair, iterative diagnostic rounds)? How do EvalScientist’s compute cost compare with other baselines? Do EvalScientist's improvements actually come from a better design, and not simply extra token spending?
Q6 EvalScientist's generalizability for other LLMs : The framework mainly uses DeepSeek-V4.1-Flash for most of its components. Does the method generalize to other classes of models like Qwen, Kimi… with good performance? Any issues with the task's format and executability when switching to other models?
Q7 Are the LLM Judge (GPT 6 Astra) reliable? While GPT 6 Astra is a powerful state-of-the-art model, benchmark and diagnosis generation is a very challenging task requiring expert level knowledge and skills. Are the judgements by GPT 6 Astra proven to be reliable and accurate? Can the human judges evaluate GPT 6 Astra's judgement in terms of correctness? Or alternatively, do human judges have high agreement rate with the LLM judge?
Q8 Frontier task validity: The authors curated 15 requests, which they claim to be from authoritative sources that deal with frontier demands. Are these requests’ usefulness validated by any experts, or have anything to ensure they are actually representative of these authoritative sources’ demands?
Q9 Unified Task Representation Novelty and Comparison METR [1] and Inspect [2] already introduced a unified representation for LLM evaluation. How are their formats different from these standards? Do they offer any improvements or novelty over these popular works?
Q10 Novelty - Missing references AutoDetect [4] also has the similar idea of generating hypotheses of the target model's potential issues, even though it does not test and confirm them. ACD [3] also identifies potential failure modes of the model, but these come from clustering thousands of generated tasks into broader categories instead of natural language hypotheses, and they also don't test these theories either. Please discuss and position the work against these.
[1] METR Task Standard, 2024b. URL https://github.com/METR/task-standard/blob/main/STANDARD.md. Version 0.5.0.
[2] UK AI Security Institute. Inspect AI: Framework for large language model evaluations. Software, 2024. URL https://inspect.aisi.org.uk/. Released May 10, 2024.
[3] Lu, Cong, Shengran Hu, and Jeff Clune. "Automated capability discovery via foundation model self-exploration." arXiv preprint arXiv:2502.07577 (2025).
[4] Jiale Cheng, Yida Lu, Xiaotao Gu, Pei Ke, Xiao Liu, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. 2024. AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6786–6803, Miami, Florida, USA. Association for Computational Linguistics.
Novelty of the design
We thank the reviewer for the candid assessment. We agree that the submitted version did not analyze why automatically built tasks fall short, and that its design offered limited conceptual novelty over existing systems. Since submission we have studied these shortcomings and substantially redesigned the Builder accordingly, for example by introducing new agent roles and a new agent topology. To our knowledge, these designs and the insights behind them go beyond prior automated benchmark builders, and we will present them in a future version of this work.
Placement
Following the guidelines, we placed the Ethics Statement and Reproducibility Statement at the end of the main text, before the references. The guidelines state that the optional ethics statement “will not count toward the page limit, but should not be more than 1 page” and that the reproducibility statement “is not part of the main text and therefore will not count toward the page limit”, so the placement is consistent with the page limit.