acceptodds
Under review as a conference paper at ICLR 2027

Building the Reliable Evaluation Layer of Auto Research with On-Demand Benchmarks

Abstract

Auto Research is emerging as a new paradigm that extends Large Language Model (LLM)-Agent assistance to the autonomous exploration and discovery of scientific experiments. However, research on novel topics may lack reliable benchmarks for supporting its claims. While Auto Research systems generate test cases to fill this gap, the resulting evaluations may favor their own methods, raising concerns about fair comparison. To this end, we introduce Idea2AutoEval, a general framework that constructs reliable evaluations on demand from research intents for Auto Research. The framework iteratively organizes prior benchmarks into domain taxonomies to identify established evaluations and clarify the scope of new ones. Building on the in-domain benchmark library, it combines research claims and benchmark coverage to define the evaluation scope and scoring criteria. To support evaluation integrity, we adopt the information separation mechanism that exposes only task inputs to evaluated models while reserving reference data exclusively for the evaluator. Guided by these, our framework couples benchmark generation with iterative quality assurance to support the validity of model comparisons through semantic verification and alignment with research goals. Experiments span eight benchmark settings across four task categories, with datasets produced by two generation models. Spearman correlations with public model rankings exceed 0.8 in 14 of 16 comparisons, while rankings from the two generators correlate above 0.85 in seven of eight settings. On four representative tasks, Idea2AutoEval outperforms BenchMaker and BenchBench in macro-average Spearman correlation with public rankings, reaching 0.8810. Further analyses show that feedback can improve agreement with public rankings, while calibration of the intended difficulty remains task dependent. These findings support generated benchmarks for model comparison and suggest their potential as evaluation feedback for Auto Research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Auto Research, Large Language Model, Data synthesis, Benchmark, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_uSk8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Idea2AutoEval, an external evaluation component for Auto Research systems. It builds benchmarks on demand from a natural-language research requirement, keeping evaluation design separate from method development. The pipeline has three stages:

  1. Benchmark knowledge base. Prior benchmarks are standardized as "Benchmark Cards" and routed into a capability–domain–task taxonomy. The taxonomy is iteratively revised using a routing-gap rate and a feedback set of low-confidence or contract-violating routes.
  2. Requirement-driven construction. The research intent is matched against the knowledge base, and the system selects a REUSE / ADAPT / CONSTRUCT strategy. It then writes an Evaluation Specification (task contract, coverage plan, evaluation contract). Instances are generated per coverage-quota slot. Each instance must pass structural, domain-specific and semantic gates; rejected instances are regenerated for the same slot.
  3. Release packaging. Model-visible inputs are separated from evaluator-only references and hidden tests.

The authors reconstruct eight existing benchmarks with two generators. The reported rank correlation between synthetic-benchmark and public-leaderboard scores over 5–8 matched models surpasses frontier baselines.

Critical strengths
  1. Timely and well-motivated problem framing. Auto Research systems that write both the method and its benchmark risk self-favoring evaluation and answer retrieval. Separating evaluation design and scoring from method iteration is a sensible principle, and making it a reusable component is a useful direction. The design choices follow from concrete failure modes: an explicit Evaluation Specification fixed before generation, quota-based coverage control so that filtering does not silently narrow coverage, and public/private packaging.

  2. Comprehensive empirical study. The evaluation spans four task families with different scoring mechanisms (multiple-choice accuracy, executable tests, ranking metrics, verifiable constraints). It uses two independent generators, reports Spearman, Kendall and Pearson correlations, and gives full per-model tables (App. A.4). The authors openly report failure cases: GPQA with Luna (ρ = 0.30), non-monotone refinement, and difficulty calibration failing in 6 of 16 variants. This makes the results easier to interpret.

Critical weaknesses
  1. The experiments do not evaluate the claimed contribution: on-demand evaluation for novel research intents. The paper motivates Idea2AutoEval for cases where existing benchmarks do not cover the needed capability (ADAPT/CONSTRUCT), and as protection against self-favoring evaluation. Yet every experiment asks the system to reproduce an existing public benchmark. It "preserves the target capability, task format, evaluation protocol, and primary metric" (l. 347–348) and is judged by agreement with that benchmark's leaderboard. This tests whether an LLM can regenerate an MMLU-Pro/MBPP/IFEval-like test set, not whether the framework produces reliable evaluations where no reference exists. Several consequences follow:

    • No ablations. The knowledge base, taxonomy routing and REUSE/ADAPT/CONSTRUCT selection (Sec. 3.1–3.2) are not evaluated. It is not reported which strategy was selected in each setting, and in the baseline comparison all methods receive "the same fixed domain-specific task specification" (l. 954). This seems to bypass these components entirely. There are also no ablations of the coverage controller, the semantic gate or targeted refill, so the source of the gains over BenchMaker/BenchBench is unknown.
    • No test of the integrity claim. There is no experiment in an Auto Research loop, and no measurement of whether self-preference or method-favoring bias is reduced, which is the stated motivation. Separating public inputs from private references (hidden tests/labels) is standard benchmark practice and is not shown to address the bias the paper describes.
    • Contamination is not controlled. The generators very likely memorized the targeted public benchmarks (MMLU-Pro, GPQA, MBPP, IFEval), and many evaluated models were trained on or tuned toward them. High correlation could therefore reflect near-duplicate regeneration rather than valid new evaluation. No overlap or similarity analysis against the public test sets is reported.

    Because the paper's thesis is about reliability beyond existing coverage, I consider this the main reason for rejection.

  2. The statistical evidence behind the headline numbers is weak.

    • Small model sets. All correlations use only 5–8 models (5 for MMLU-Pro), and these models are mostly older 0.5B–14B checkpoints with large capability gaps (e.g., Qwen2.5-0.5B vs. 14B; BERT/average-word embeddings vs. BGE). Rank agreement on such spread-out models is a low bar, and with n = 8 a single adjacent swap moves ρ by about 0.02–0.05.
    • No uncertainty estimates. There are no confidence intervals, permutation p-values or repeated generations: each setting is a single generated benchmark per generator. The "8 of 12 wins" and macro-average ρ = 0.881 vs. baselines (Fig. 2, 4 tasks × 8 models) therefore cannot be separated from noise.
    • Discriminative power is not examined. Several synthetic benchmarks hit ceiling or floor effects. On Luna MMLU-Pro all models score 90.75–98.75 (vs. 28–65 publicly), with ties at the top. On Luna MBPP, Mistral scores 0.00 and Gemma/Zephyr tie at 8.04. Spearman correlation hides these effects, even though they matter for a benchmark meant to give feedback to research iterations.
    • No direct quality audit. There is no human audit of instance correctness, such as label-error rates or test-case validity, which would be the most direct check of "reliable" evaluation.
Questions for authors
  1. In Sec. 4.4, what information does the "feedback from the preceding evaluation round" contain? If it includes public-benchmark scores or correlations, refinement directly optimizes the evaluation target. That information would also be unavailable in the intended Auto Research setting, where no public reference exists. Please specify the feedback content and stopping criterion.
  2. Strategy selection and knowledge base. In each of the eight settings, which strategy (REUSE / ADAPT / CONSTRUCT) was selected? How large is the benchmark library, and how many routing/revision rounds were run? Was the knowledge base used at all in the reported experiments, including the baseline comparison under a "fixed task specification"?
  3. Data Contamination. Can you report n-gram or embedding-similarity overlap between generated instances and the public test sets (and their training splits)? Can you also report how alignment changes after removing near-duplicates?
  4. Can you report bootstrap or permutation confidence intervals for ρ and τ, and results over multiple independent generations per setting? Can you state whether Idea2AutoEval's advantage over BenchMaker and BenchBench is significant?
  5. Baseline fairness. Did all three methods produce the same number of instances, with the same validator models and comparable generation/validation budget (API calls, cost)?
  6. Pipeline statistics. What are the benchmark sizes, per-gate rejection rates and validator models? Was any human verification of instance correctness performed?
6 comments

We sincerely thank the reviewer for the positive recognition of our research motivation, methodological design, and experimental work. We are pleased that the reviewer acknowledges the necessity of decoupling evaluation design from method iteration, as well as the rationale behind our design choices, including explicit evaluation specifications, coverage constraints, and public–private information separation. The reviewer's positive assessment of our experimental coverage, diversity of evaluation metrics, and transparent reporting of failure cases is highly meaningful to our work.

Reply to W1 & Q3:

We thank the reviewer for the detailed comments regarding experimental scope, ablation studies, integrity, and data contamination. First, for genuinely novel research tasks with no prior studies, if no existing benchmarks, public leaderboards, or reference evaluation results are available, there is no reliable external ranking reference against which to directly compare generated benchmarks. Therefore, we choose to use existing public leaderboards to validate the effectiveness and reliability of our proposed framework, with the goal of further extending it to genuinely novel research tasks and previously unexplored domains based on the established reliability of the framework.

W1-1: We propose Idea2AutoEval as a complete automated benchmark construction framework with explicit procedural dependencies, rather than a simple combination of multiple independently executable modules. Taxonomy Construction, Semantic Routing, REUSE/ADAPT/CONSTRUCT strategy selection, Coverage Control, and Semantic Validation jointly form the complete construction pipeline. These stages have explicit dependencies, where the output of each preceding stage provides the foundation for the subsequent stages. Therefore, directly removing a key component may disrupt the integrity of the entire pipeline, making it difficult to independently measure its contribution while preserving the original construction objectives.

W1-2: We thank the reviewer for the concern regarding our claim of evaluation integrity. We would like to clarify that Idea2AutoEval is not a complete Auto Research system, but an independent and reusable external evaluation component. Its objective is to separate benchmark design and scoring from the research iteration of the methods being evaluated. Therefore, our integrity claim primarily refers to the structural decoupling mechanism provided by the framework, rather than claiming to have quantitatively measured the reduction of self-favoring bias within a complete Auto Research loop.

Specifically, Idea2AutoEval freezes the Evaluation Specification before instance generation, separates the evaluation construction component from the method iteration process, and isolates model-visible inputs from evaluator-private references, labels, and hidden tests. These design choices prevent the evaluated methods from directly accessing private evaluation references or modifying task definitions and scoring rules based on their own outputs, thereby reducing the risk of evaluation design being influenced by method iteration at the mechanism level.

Therefore, the experiments in this paper validate whether the external evaluation component can independently construct benchmarks with stable model-discrimination capabilities and whether the evaluation results of generated benchmarks remain consistent with established external evaluation systems. We do not interpret our current results as a direct causal measurement of the reduction in self-preference or method-favoring bias. The latter would require integrating the evaluation component into a complete Auto Research system and comparing research iteration outcomes before and after decoupling, which constitutes a system-level research question.

W1-3 & Q3:

We thank the reviewer for raising this important issue. We fully acknowledge the concern regarding data contamination. To address this concern, we have added a data overlap analysis between the generated instances and the available train, validation, and test splits of the corresponding public benchmarks. We examine potential contamination at three levels:

  1. Normalized exact matching, to identify instances with identical content.

  2. Word 5-gram overlap. We consider an instance a textual near-duplicate if its Jaccard similarity is at least 0.80, or if its containment is at least 0.80 with at least 20 shared 5-grams.

  3. Embedding cosine similarity of the main task text, computed using the frozen BAAI/bge-small-en-v1.5 model. Following prior work that uses a cosine similarity threshold of 0.95 to screen for semantic matches or near-duplicate instances, we adopt 0.95 as a high-precision screening threshold.

Exact matching and n-gram checks cover all relevant fields, including questions, candidate answers, code, retrieval queries, and candidate documents. Embedding similarity primarily compares text that carries the task semantics. For the MTEB dataset, which contains 8,453,892 public query and document components, we first perform full-scale exact matching and n-gram retrieval, followed by BGE-based reranking of the retrieved candidates.

Among the 5,628 released instances across 16 generated versions, the primary threshold flags 10 instances, accounting for 0.18% of all instances. All 10 instances come from the DeepSeek-generated Retrieval-MTEB-Reranking benchmark: 10 out of 400 instances (2.50%). These include one query that exactly matches an instance in the public training split, one n-gram near-duplicate instance, and 10 instances with embedding similarity scores of at least 0.95. The detection categories overlap. No near-duplicate instances are detected in the remaining 15 generated versions under the primary threshold.

BenchmarkGeneratorInstancesExact DuplicatesN-gram Near-DuplicatesSemantic Similarity ≥ 0.95Final Flagged
QA-MMLU-ProDeepSeek4000000
QA-MMLU-ProLuna4000000
QA-GPQADeepSeek1790000
QA-GPQALuna1790000
Coding-MBPPDeepSeek2860000
Coding-MBPPLuna2860000
Coding-Aider133DeepSeek1330000
Coding-Aider133Luna1330000
Retrieval-MTEBDeepSeek400111010 (2.50%)
Retrieval-MTEBLuna4000000
Retrieval-SciFactDeepSeek4160000
Retrieval-SciFactLuna4160000
Instruction-IFEvalDeepSeek5000000
Instruction-IFEvalLuna5000000
Instruction-IFBench-OODDeepSeek5000000
Instruction-IFBench-OODLuna5000000
Total—5,628111010 (0.18%)

We further removed these 10 instances and recomputed the per-instance MAP scores for all eight models using the remaining 390 instances. After deduplication, the relative rankings of all models remained unchanged. The changes in MAP scores for individual models ranged from +0.0010+0.0010 to +0.0045+0.0045. The correlations between the public benchmark and the deduplicated benchmark remained:

ρ=1.0000,τ=1.0000.\rho = 1.0000,\qquad \tau = 1.0000.

ModelOriginal MAPDeduplicated MAPChangeDeduplicated Rank
bge-large-en-v1.50.73580.7381+0.00231
all-mpnet-base-v20.73440.7361+0.00172
bge-base-en-v1.50.72300.7271+0.00413
gtr-t5-base0.71240.7136+0.00134
gte-small0.71170.7127+0.00105
contriever-base-msmarco0.66300.6672+0.00426
average-word-embeddings0.63820.6419+0.00387
bert-base-uncased0.54920.5537+0.00458

Across all 5,628 released instances, only 10 were flagged under the primary detection criteria, accounting for 0.18% of the total, and all were concentrated in a single generated version. No near-duplicate instances were detected in the remaining 15 versions. More importantly, after removing all flagged instances, the relative rankings of all eight evaluated models remained exactly unchanged. The Spearman ρ\rho and Kendall τ\tau correlations between the public benchmark and the deduplicated benchmark both remained at 1.00001.0000, while the changes in individual models' MAP scores were limited to +0.0010+0.0010 to +0.0045+0.0045. These results demonstrate that the high ranking consistency observed between the public and generated benchmarks is not driven by a small number of duplicated or highly similar instances.

Reply to W2&Q4:

We thank the reviewer for the detailed suggestions regarding statistical uncertainty and model discrimination capability.

W2-1: We would like to clarify that the reviewer's observation that each task uses 5–8 models does not mean that our entire experimental study involves only 5–8 models. In fact, we independently selected different evaluation model sets according to the characteristics of each task. Across eight tasks, we conducted a total of 122 model–dataset evaluations. These models come from different providers and cover diverse model types and capability levels. The specific models are listed below:

BenchmarkModelsEvaluated Models
QA–MMLU-Pro5Phi-3-mini, Llama-3-8B, OpenChat-3.5, MAP-Neo-7B-Instruct, Zephyr-7B. (Abdin et al., 2024; Grattafiori et al., 2024; Wang et al., 2024a; Zhang et al., 2024; Tunstall et al., 2023)
QA–GPQA8Gemma-2-9B, Llama-3.1-8B, Gemma-7B, Meta-Llama-3-8B, Mistral-v0.3, StableLM, Llama-2-7B-Chat, Mistral-v0.1. (Grattafiori et al., 2024; Jiang et al., 2023; Bellagente et al., 2024; Touvron et al., 2023)
Coding–MBPP8OpenCoder, Magicoder, Llama-3.1, Meta-Llama, WizardCoder, Mistral, Gemma, Zephyr. (Huang et al., 2025; Wei et al., 2023; Grattafiori et al., 2024; Jiang et al., 2023; Gemma Team, 2024; Tunstall et al., 2023)
Coding–Aider1338Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-0.5B-Instruct, Yi-Coder-9B-Chat, Llama-3.1-8B-Instruct, OpenCodeInterpreter-DS-6.7B, Hermes-3-Llama-3.1-8B, Llama-3.2-3B-Instruct, Granite-3.0-8B-Instruct. (Hui et al., 2024; Young et al., 2024; Zheng et al., 2024; Quesnelle & Guang, 2024; Granite Team, 2024)
Retrieval–MTEB-Reranking8bge-large-en-v1.5, all-mpnet-base-v2, bge-base-en-v1.5, all-MiniLM-L12-v2, gtr-t5-base, contriever-base-msmarco, average-word, bert-base-uncased. (Xiao et al., 2024; Song et al., 2020; Wang et al., 2020; Ni et al., 2022; Izacard et al., 2021; Devlin et al., 2019)
Retrieval–SciFact8gte-base, bge-large, bge-small, gte-tiny, contriever, gtr-t5, multi-qa-MiniLM, average-word. (Li et al., 2023; Xiao et al., 2024; Izacard et al., 2021; Ni et al., 2022; Wang et al., 2020)
Instruction–IFEval8Qwen2.5-14B, Qwen2.5-7B, Qwen2.5-3B, Mistral, Aya-23, Qwen2.5-1.5B, DeepSeek-LLM, Qwen2.5-0.5B. (Yang et al., 2024; Jiang et al., 2023; Aryabumi et al., 2024; Bi et al., 2024)
Instruction–IFBench-OOD8RAIF-Qwen2.5-7B, Qwen2.5-7B, RAIF-DeepSeek-Qwen-7B, Ministral-8B, Qwen2.5-1.5B, RAIF-DeepSeek-Qwen-1.5B, RAIF-DeepScaleR-1.5B, DeepSeek-R1-Distill. (Qin et al., 2026; Yang et al., 2024; Jiang et al., 2023; Guo et al., 2025)

Overall, our experiments cover a diverse set of models far beyond the number used in any individual task. Rather than repeatedly evaluating the framework using the same fixed model set, we independently examine its effectiveness across different tasks and different model combinations. This cross-task and cross-model experimental design further examines whether the framework depends on any particular model set and provides empirical evidence of its generalization capability.

Reviewer_AWCZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Idea2AutoEval, a framework for automatically generating benchmarks for autonomous AI research. It combines existing benchmark knowledge, coverage-controlled generation, and multi-stage validation to construct evaluation datasets. Experiments across eight tasks show strong correlations with public benchmark rankings and improvements over existing benchmark generation methods.

Critical strengths
  1. The proposed framework integrates several useful components, including hierarchical benchmark organization, research-intent-based benchmark retrieval, explicit evaluation specifications, coverage-constrained instance generation and multi-stage quality validation. The mechanism provides a structured way to leverage existing benchmark knowledge while allowing new evaluation tasks to be introduced.
  2. The comparisons against BenchMaker and BenchBench also demonstrate improvements in preserving model rankings on several representative tasks.
Critical weaknesses
  1. The paper is motivated by the need to evaluate novel research ideas for which existing benchmarks may be insufficient or unavailable. However, all eight experimental settings are derived from well-established benchmarks with predefined task formats and evaluation metrics. The experiments primarily demonstrate that the framework can generate new instances for existing task types while preserving model rankings. This is substantially different from constructing reliable evaluations for genuinely novel research problems. In particular, the paper does not provide convincing empirical evidence for its construct strategy when no suitable benchmark or evaluation protocol exists. This limits the extent to which the experiments support the claimed applicability to open-ended Auto Research.
  2. The primary evaluation measures agreement between model rankings on generated benchmarks and existing public benchmarks. While high correlation suggests that generated benchmarks capture similar performance differences, it does not necessarily establish the correctness of individual reference answers, the validity of the measured capability, or the fairness of the evaluation.
Questions for authors
  1. Since all experimental tasks are based on existing benchmarks with established evaluation protocols, how would Idea2AutoEval handle genuinely novel research problems where no suitable benchmark or ground truth exists?
  2. Why is correlation with public model rankings sufficient to establish benchmark reliability? Could the authors provide more direct evidence of benchmark validity, such as human-verified instance correctness or analysis of evaluation bias?
2 comments

We sincerely thank the reviewer for the positive recognition of our methodology and experimental results. We are pleased that the reviewer acknowledges the systematic design of Idea2AutoEval in integrating existing benchmark knowledge, constructing new evaluation tasks, and performing multi-stage quality validation. We greatly appreciate these positive comments.

Reply to W1:

We thank the reviewer for pointing out the distinction between existing public tasks and genuinely novel research problems. First, for genuinely novel research tasks with no prior studies, if no existing benchmarks are available, there is no reliable external ranking reference against which to directly compare the generated benchmarks. Therefore, we choose to use existing public leaderboards to validate the effectiveness and reliability of the Idea2AutoEval framework, thereby providing an empirical basis for its reasonable generalization to novel tasks.

Furthermore, our framework is not limited to reconstructing existing benchmarks, but incorporates three construction strategies: REUSE, ADAPT, and CONSTRUCT. In particular, CONSTRUCT is specifically designed for research intents for which no suitable existing benchmark can be retrieved. The CONSTRUCT strategy directly generates an Evaluation Specification defining the task structure, input and output formats, coverage constraints, and scoring rules, followed by multi-stage validation to generate and filter instances.

Reply to W2:

We thank the reviewer for pointing out that model ranking consistency and instance-level correctness represent two distinct evaluation dimensions. We believe that instance-level quality validation and overall evaluation results provide complementary evidence. If generated instances contain substantial errors, or if the internal validation mechanism fails to identify critical defects, these problems may further affect the actual performance of different models on the generated benchmark and be reflected in the final model rankings and score distributions. However, our experimental results show that generated benchmarks maintain high ranking consistency with public benchmarks across different tasks and model combinations. This provides further support for the soundness of our internal quality validation mechanism and overall framework design from the perspective of overall evaluation results.

We do not rely solely on ranking correlations to determine the correctness of generated instances. Before being included in the final dataset, each candidate instance must pass three internal quality validation stages: structural validation, domain-specific validation, and semantic validation. These stages respectively examine data formats and information boundaries, task executability and scoring validity, as well as content relevance and consistency. Through these validation mechanisms, the system can identify and filter out formatting errors, execution issues, and semantic defects in individual instances. Our design focuses on ensuring instance quality through automated internal validation, rather than relying on continuous human expert participation in reviewing every sample in a semi-automated system.

Therefore, our work does not use ranking correlation as a substitute for instance-level correctness verification. Instead, it combines three-stage instance-level quality validation with benchmark-level external ranking consistency to jointly establish the reliability of automatically generated benchmarks from both local sample quality and overall evaluation capability.

Reply to Q1:

We thank the reviewer for asking us to further clarify how Idea2AutoEval handles genuinely novel research tasks. The design of Idea2AutoEval does not depend on existing mature benchmarks or predefined evaluation protocols. Existing benchmarks primarily serve as reusable sources of knowledge, helping the framework understand the organizational structures and evaluation principles of existing tasks, rather than serving as prerequisites for constructing new tasks. When the framework cannot retrieve a suitable existing benchmark, it can directly construct a new evaluation task based on the research intent provided by the user through the CONSTRUCT strategy.

The local benchmark library adopts an extensible Taxonomy organized around capability, domain, and task. When existing categories or routing contracts cannot accommodate a particular task, the system records the corresponding gap and extends the knowledge structure by introducing a new task entry or revising the routing contract, rather than forcing the task into an inappropriate category. Furthermore, Benchmark Analysis assesses the compatibility between the research intent and existing tasks in terms of objectives, structures, and evaluation protocols. When no suitable match can be found, this serves as a signal for selecting the CONSTRUCT strategy. Under CONSTRUCT, existing benchmark task templates are no longer required as input. Instead, the system generates a new Evaluation Specification directly from the user's research intent and transforms it into an executable benchmark through pilot instances, coverage control, and three-stage validation.

We also note that, among the strengths of our work, the reviewer acknowledges research-intent-driven benchmark retrieval, explicit Evaluation Specifications, coverage-constrained generation, and multi-stage quality validation, and recognizes that the framework provides a structured mechanism for leveraging existing benchmark knowledge while allowing new evaluation tasks to be introduced. We appreciate this accurate summary, as these components are precisely designed to address research requirements beyond the coverage of existing benchmarks. Our experiments on public benchmarks select tasks with available external references to objectively examine the model-discrimination capability of generated benchmarks. This experimental choice should not be interpreted as indicating that our framework can only reconstruct existing tasks.

Reply to Q2:

We thank the reviewer for asking us to further clarify the relationship between ranking correlation and benchmark reliability. The public benchmarks we use originate from existing research studies, where they were designed and curated by researchers with corresponding evaluation protocols. Their publicly available model rankings reflect the evaluations of different model capabilities under these human-designed evaluation tasks. By comparing model rankings on our independently generated benchmarks with those on existing public benchmarks, we are effectively examining whether automatically generated evaluation results are consistent with established human-designed evaluation systems. In our experiments, most generated benchmarks achieve high ranking correlations, providing empirical evidence of consistency between automatically generated evaluation results and existing human-designed evaluation systems.

Reviewer_pWKrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Idea2AutoEval, a framework for automatically constructing benchmarks from research requirements for Auto Research systems. The framework organizes existing benchmarks into a hierarchical taxonomy, translates research requirements into evaluation specifications, and generates new evaluation instances with coverage control and validation.

The authors evaluate the framework across eight benchmark settings in four task categories, using two generation models. The generated benchmarks generally show high correlations with public model rankings, and the framework outperforms existing baselines on several comparisons. The paper also examines iterative refinement through feedback and difficulty calibration.

Critical strengths

Strong results compared to existing baselines - Idea2AutoEval achieves the highest macro-average Spearman correlation across the four evaluated domains and performs best in 8 out of 12 domain-metric comparisons against BenchMaker and BenchBench. These results suggest that the framework can generate benchmarks that preserve model rankings better than the compared methods.

Potential for iterative improvement - Since benchmark generation and validation are automated, the framework has the potential for continuous improvement through feedback and regeneration. The experimental results support this potential, showing that a refinement round can improve agreement with public rankings. Although the improvements are not consistent across all tasks and generation models, the results suggest a promising direction. If the feedback process can be adapted to each task and generator to achieve more consistent improvements, the framework could become a more reliable and effective tool for automated evaluation.

Critical weaknesses

Gap between the motivation and the experiments - The paper motivates the framework as a way to evaluate new research ideas that are not covered by existing benchmarks. However, the experiments focus on established benchmarks such as MMLU-Pro, GPQA, and MBPP. While the results show that the framework can reproduce rankings from existing evaluations, they do not demonstrate that it can reliably construct benchmarks for genuinely new research tasks. An experiment requiring the CONSTRUCT strategy, where no suitable benchmark already exists, would better support the main motivation.

Heavy reliance on LLMs with limited human validation - The framework relies heavily on language models throughout the pipeline, including taxonomy construction, question generation, and semantic validation of the generated instances. However, the paper does not report systematic human expert evaluation of these stages. This raises concerns about errors that may go undetected when LLMs are involved in both generating and validating evaluation data. Human evaluation of a sample of generated benchmarks, as well as expert review of the taxonomy, would provide stronger evidence of their quality.

Difficulty identifying unreliable generated benchmarks - Most generated benchmarks show strong correlations with public rankings, but there are notable failures. For example, the Luna-generated GPQA benchmark achieves a Spearman correlation of only 0.299. This raises an important practical question: if the framework is used for a new research task without a public benchmark, how can we know whether the generated evaluation is reliable? The paper would benefit from a way to detect such failures without requiring an existing benchmark for comparison.

Missing ablation studies - The framework includes several components, such as taxonomy construction, semantic routing, coverage control, and semantic validation. However, the paper does not isolate their individual contributions. It is therefore difficult to tell which components are responsible for the improvements over the baselines, or whether all of them are necessary.

Limited analysis across generation models - The experiments use only two benchmark generators, DeepSeek and GPT-5.6 Luna. Evaluating additional generators would help establish how general the results are. It would also be useful to analyze cases where the two generators produce substantially different results under the same evaluation specification, particularly on GPQA, to understand what causes one generated benchmark to perform much worse than the other.

Questions for authors
  1. Is the same LLM used for both generating benchmark instances and validating them? Have you experimented with using different models for these two stages? Using the same model for both tasks could lead to repeated errors or shared biases, where the validator fails to detect mistakes made during generation.

  2. How much time and computational cost does it take to generate a complete benchmark? Could you report the average generation and validation costs, the percentage of generated instances that are rejected during validation, and the total time required to construct each benchmark?

  3. How are stopping criteria for feedback-driven refinement determined without relying on access to public benchmark rankings?

4 comments

We sincerely thank the reviewer for recognizing the experimental performance and iterative improvement potential of Idea2AutoEval. We appreciate the reviewer's acknowledgment that our framework achieves the highest macro-averaged Spearman correlation across four evaluation domains and performs best in 12 domain-metric comparisons against BenchMaker and BenchBench. We are also pleased that the reviewer recognizes the potential of automated feedback and regeneration mechanisms to further improve benchmark quality and consistency with public leaderboards. Below, we address the relevant concerns point by point.

Reply to W1 & W3:

We thank the reviewer for highlighting the important relationship between constructing benchmarks for novel tasks and assessing their reliability. We would like to point out that the evaluation requirements raised in W1 and W3 present an inherent tension: W1 asks us to validate the CONSTRUCT strategy on novel research tasks without existing benchmarks, while W3 correctly points out that, in the absence of public benchmarks, leaderboards, or reference evaluation results, it is difficult to establish the reliability of newly generated benchmarks through external evidence.

For genuinely novel research tasks, if no public benchmarks are available, there is no reliable external ranking reference against which to directly compare generated benchmarks. Therefore, we choose to conduct experiments on tasks with existing public references to validate the effectiveness and reliability of our framework. Our goal is to establish the reliability of Idea2AutoEval, thereby providing an empirical basis for its reasonable generalization to novel research tasks.

Furthermore, our framework is not limited to reconstructing existing benchmarks. It incorporates three construction strategies: REUSE, ADAPT, and CONSTRUCT. Among them, CONSTRUCT is specifically designed for research intents for which no suitable existing benchmark can be retrieved.

We acknowledge that the ranking correlation between the generated benchmark and the public benchmark on GPQA is relatively low. We would like to explain the particular characteristics of the GPQA dataset. GPQA is a challenging benchmark targeting graduate-level scientific question answering, designed to evaluate different models on highly difficult scientific questions. For extreme benchmarks such as Humanity's Last Exam (HLE), which are designed to test the limits of model capabilities, the questions themselves are designed and written by human experts. Requiring current automated generation frameworks to independently construct such extreme tasks beyond the cognitive capabilities of existing models does not reflect the current state of practical technology. This represents an objective limitation of current automated task-generation capabilities rather than a deficiency in our framework design.

However, we believe that this experimental result actually highlights the importance of using existing public benchmarks as external references. For tasks with public rankings, we can identify biases and shortcomings in generated benchmarks by comparing their ranking results with public references, thereby assessing the reliability of the current construction results. This is precisely why we validate our framework on existing benchmarks first: if generated results cannot be effectively validated even on tasks with established public references, it would be even more difficult to establish an objective assessment of reliability for entirely novel tasks without any external references.

Reply to W2:

We thank the reviewer for the important comments regarding LLM dependence and human validation. We would like to clarify that LLM dependence is an inherent characteristic of the Auto Research environment. Auto Research systems themselves rely on LLMs as their core components. Our goal is to achieve automated benchmark construction and reliable evaluation, rather than introducing a semi-automated workflow that requires continuous participation from human experts.

Our framework does not rely on a single LLM-based judgment. Instead, it employs three complementary validation stages: structural validation checks data structures and information separation; domain-specific validation checks task execution and scoring requirements; and semantic validation examines content relevance and consistency. Importantly, domain-specific validation can employ executable tests and rule-based checks, providing verification mechanisms beyond model-based judgments.

We compare the relative rankings of different models on two types of benchmarks to examine the consistency between automatically generated evaluation results and existing evaluation systems. Public benchmarks are evaluation standards that have already been designed and validated by human researchers, providing empirical external references for assessing the reliability of our framework.

Reply to W4:

We thank the reviewer for raising the question regarding ablation studies. We would like to clarify that Idea2AutoEval is not simply a combination of several independent model modules, but rather an automated benchmark construction framework with explicit procedural dependencies.

Its components, including Taxonomy Construction, Semantic Routing, Coverage Control, and Semantic Validation, serve different functions and jointly form a complete benchmark construction pipeline. These stages have explicit dependencies, where the output of each preceding stage provides the foundation for the subsequent stages.

Our contribution lies in the coordination of these components to form a complete automated workflow, rather than in any single independent module. Removing an individual component would alter the task definition and disrupt the integrity of the overall pipeline.

Reply to W5:

We thank the reviewer for the suggestion regarding the coverage of benchmark generators. We understand the concern that using only two generators may limit the generalizability of our experiments. However, we would like to emphasize that the generalizability of our experiments is reflected not only in the number of generators but also in the independent evaluations conducted across different tasks and different evaluation-model sets.

We use two independent generators, DeepSeek and GPT-5.6 Luna, to construct benchmarks across eight different tasks, resulting in 16 generated benchmark variants. Rather than simply testing two generators using the same fixed set of models, we conduct extensive independent evaluations across different tasks and model combinations. We use different evaluation-model sets for all eight tasks, independently selected according to the characteristics of each task and covering diverse models from multiple providers. In total, we conduct 122 independent model–dataset evaluation runs on our generated benchmarks.

This experimental design allows us to examine the applicability of our framework across different tasks, generators, and evaluation-model combinations, thereby assessing whether its effectiveness depends on any particular task or evaluation-model set. The results show that, among the 16 independent comparisons between public and generated benchmarks, 14 achieve Spearman correlations above 0.8. This indicates that our framework is not effective only under a particular task or model combination, but demonstrates good reliability and consistency across diverse experimental conditions, providing empirical support for its generalization across tasks and model combinations.

Regarding the different results on GPQA, we would like to explain the particular characteristics of this dataset. GPQA targets graduate-level scientific question answering and requires substantial domain knowledge and complex reasoning capabilities. Due to the limitations of the generation models themselves, they may struggle to consistently construct sufficiently challenging questions with accurately calibrated difficulty, potentially resulting in reversed difficulty trends. This result primarily reflects the capability boundaries of current generation models on highly challenging scientific tasks, rather than a structural deficiency of the Idea2AutoEval framework.

Reply to Q1:

We thank the reviewer for raising the concern regarding potential shared biases between generation models and validation models. We would like to explain that using the same LLM for benchmark instance generation and internal quality validation within each experimental setting is a deliberate design choice. Using different models within the same generation pipeline may introduce differences in generation capabilities, comprehension abilities, and judgment criteria, causing the generation and validation stages to be influenced by model switching. Therefore, we use the same model for both stages to maintain consistent model conditions within each construction process and reduce interference caused by differences in model capabilities.

Furthermore, we do not rely solely on LLMs for validation. Instead, we design a three-stage validation mechanism to ensure the reliability of generated data. Our experimental results demonstrate strong correlations between public and self-constructed benchmark rankings across a large number of experiments, providing empirical support for the effectiveness of our multi-stage validation mechanism.

Reply to Q2:

We thank the reviewer for asking about the time and computational costs of benchmark generation. To provide a more concrete assessment, we report below the computational statistics of Idea2AutoEval and the two reproduced baselines, BenchMaker and BenchBench.

MethodTaskAPI CallsGeneratedAcceptedRejected / RemovedAcceptance RateNet Construction Time
Idea2AutoEvalQA-GPQA2592511797271.31%01:06:29
BenchBenchQA-GPQA36936915021940.65%02:02:24
BenchMakerQA-GPQA54154015039027.78%00:44:37
Idea2AutoEvalCoding-MBPP8813002861495.33%02:29:26
BenchBenchCoding-MBPP26826815011855.97%00:25:44
BenchMakerCoding-MBPP26926815011855.97%00:24:18
Idea2AutoEvalRetrieval-MTEB4934854008582.47%00:42:57
BenchBenchRetrieval-MTEB47447415032431.65%01:12:21
BenchMakerRetrieval-MTEB44043915028934.17%01:09:36
Idea2AutoEvalInstruction-IFEval505005000100.00%00:03:17
BenchBenchInstruction-IFEval2032031505373.89%00:03:57
BenchMakerInstruction-IFEval28728615013652.45%00:10:07

Idea2AutoEval achieves higher instance acceptance rates than both baselines across all four tasks. Aggregating the generated instances across all tasks, Idea2AutoEval generates 1,536 candidates and accepts 1,365, achieving an overall acceptance rate of 88.87%, substantially higher than 45.66% for BenchBench and 39.14% for BenchMaker. These results indicate that the Evaluation Specification, coverage constraints, and multi-stage quality validation help establish explicit instance requirements before generation, reducing invalid generations and subsequent rejections, and enabling the system to maintain high and consistent instance acceptance rates across different task types.

Reply to Q3:

We thank the reviewer for asking us to clarify the feedback information and stopping criteria. We would like to clarify that the internal feedback mechanism of Idea2AutoEval is completely independent of public benchmark rankings and correlation results. System feedback is derived exclusively from internal quality diagnostics, including missing coverage, structural or execution failures, semantic conflicts, near-duplicates, difficulty or quota deviations, and scorer or release-gate errors. None of these feedback signals are related to model scores or ranking correlations on public benchmarks. Instead, they are used to guide the system in correcting issues identified during generation and validation. Public benchmark rankings and correlations are used exclusively for external comparative analysis in the paper's experiments to evaluate the consistency between generated benchmarks and existing public evaluation results.

Regarding the stopping criteria, benchmark construction terminates when the target sample size and coverage quotas are satisfied, all instances pass the release gate, and no unresolved high-severity errors remain. If a coverage slot still fails validation after at most KK attempts, the system returns an Incomplete Report rather than lowering the validation requirements to force publication.

Thank you to the authors for the detailed responses and for addressing the points raised. Below are my follow-up comments.

Response to W1 & W3:

I do not agree that there is an inherent tension between W1 and W3. In my view, both concerns reflect the same underlying problem: the framework has not demonstrated that it can reliably generate benchmarks for novel tasks without existing reference evaluations. An important first step would be to develop a benchmark reliability metric that does not depend on existing benchmarks or public rankings, even if the metric is imperfect. The authors could then validate this metric on tasks with public reference rankings and examine whether it can successfully distinguish reliable from unreliable generated benchmarks, particularly in cases such as GPQA.

Additionally, I agree that generating challenging benchmarks such as GPQA and HLE is difficult. However, given the rapid improvement in model capabilities, benchmarks increasingly need to evaluate the limits of these capabilities. Therefore, the ability to generate challenging tasks is an essential requirement for an automated benchmark construction framework, and the limitations of current generation models do not fully address this concern. If the framework cannot replace or at least substantially reduce the need for human experts in constructing such evaluations, its practical usefulness remains limited.

Response to W2:

Thank you for the clarification. I would like to emphasize that my suggestion is not to turn benchmark construction into a semi-automated process or require human involvement during deployment. Rather, I suggest conducting systematic human evaluation during the development and validation of the framework to ensure that the LLMs used as judges achieve sufficient agreement with human experts. Such an evaluation would help establish the reliability of the validation mechanism before the framework is deployed for fully automated use, without compromising its goal of complete automation.

Response to W4:

Thank you for the clarification. I agree that there are explicit dependencies between the framework's components, but I am not convinced that this prevents conducting ablation studies. For example, could the Coverage Control component be removed, allowing question generation without coverage constraints, to evaluate its contribution to the framework's performance? Such an experiment could help clarify whether all components are necessary and how much each contributes to the improvements over the baselines.

Response to W5:

Thank you for the response. I believe that the fact that two generators, following the same generation process and relying on the same benchmark knowledge base, produce substantially different results requires deeper analysis. For example, on GPQA, DeepSeek achieves a Spearman correlation of 0.714, compared to only 0.299 for Luna. Furthermore, the authors themselves acknowledge that some language models struggle to generate reliable benchmarks for challenging tasks. In my view, this is not merely a limitation of the generation models but also a fundamental concern regarding the framework's current practical usability. If benchmark reliability depends significantly on the choice of generator, a mechanism is needed to determine which models are suitable for use within the framework and under what conditions. Otherwise, it is difficult to rely on its results in practice.

Finally, I would like to thank the authors for addressing all my questions and providing additional clarifications. I appreciate the effort invested in the response, although my main concerns regarding the framework's reliability and generalizability remain unresolved.

Reviewer_3hzKOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a critical bottleneck in the emerging paradigm of "Auto Research" (autonomous LLM agents conducting scientific discovery): the lack of reliable, unbiased evaluation benchmarks for novel research topics. When Auto Research systems generate their own test cases, they risk introducing self-serving bias. To solve this, the authors propose Idea2AutoEval, an external, on-demand benchmark construction framework that decouples evaluation design from method development.

Critical strengths

Highly Relevant and Timely Motivation: As Auto Research systems become more capable, the risk of evaluation bias (where an agent overfits to its own self-generated tests) is a major concern. Proposing a decoupled, external evaluation layer is a strong conceptual contribution that addresses a real pain point in autonomous AI research.

Well-Designed and Comprehensive Methodology: The pipeline is thoughtfully constructed. The use of a semantic routing taxonomy to leverage existing benchmark knowledge, combined with coverage-constrained generation and strict public/private information separation, provides a robust mechanism to ensure fairness and prevent data leakage.

Rigorous and Extensive Empirical Evaluation: The experimental design is impressive. Testing across 4 distinct domains, 8 specific tasks, and 2 different generators provides strong evidence of the framework's generalizability. The use of multiple correlation metrics (Spearman, Kendall, Pearson) against public leaderboards is appropriate and thorough.

Critical weaknesses

Lack of Evaluation on Truly "Novel" Tasks: The core motivation of the paper is to evaluate novel research topics where reliable benchmarks do not exist. However, all 8 experimental settings involve reconstructing existing public benchmarks (MMLU-Pro, GPQA, MBPP, etc.). There is no experiment demonstrating the framework's ability to handle a genuinely novel research intent that has no prior benchmark equivalent. Evaluating on existing benchmarks only proves it can replicate known distributions, not that it can invent and validate valid new ones.

Computational Overhead Not Discussed: The framework involves multiple LLM calls for semantic routing, generation, and multi-level validation. For an "on-demand" component meant to be integrated into iterative Auto Research loops, latency and API costs are critical factors. The paper lacks an analysis of the computational overhead, time required per benchmark, and cost compared to the baselines.

Vulnerability of LLM-based Validators: The semantic validation and routing rely heavily on LLMs. If the Auto Research system and the evaluator use similar LLM families, there is a risk of shared biases or blind spots. The robustness of these validation gates against subtly flawed or adversarial generated instances is not deeply explored.

Difficulty Calibration Limitations: The paper acknowledges that translating specified difficulty into empirical performance is "task-dependent" and sometimes fails (e.g., the Luna generator on GPQA showed reversed difficulty trends). If an Auto Research system specifically targets a certain difficulty level to test the limits of a new model, the inability to reliably calibrate difficulty could render the generated benchmark less useful for that specific intent.

Questions for authors

please see weakness

2 comments

We sincerely thank the reviewer for the positive evaluation of our work. We appreciate the recognition of the timeliness of our research motivation, the comprehensiveness of our methodology, and the rigor of our empirical evaluation. In particular, we are grateful for the reviewer’s recognition of the importance of an independent evaluation layer for Auto Research, as well as the effectiveness of our framework across diverse domains and tasks. Below, we address the reviewer’s comments point by point.

Reply to W1:

We thank the reviewer for pointing out the relationship between our main experiments and genuinely novel research tasks. First, for genuinely novel research tasks, if no existing benchmarks or public leaderboards are available, there is no reliable external ranking reference against which to compare the generated benchmarks. Therefore, we conduct experiments on tasks with existing public references to establish the effectiveness and reliability of our framework. Our goal is to validate the reliability of Idea2AutoEval, thereby providing an empirical basis for its potential generalization to novel research tasks.

Idea2AutoEval does not depend on existing benchmarks to operate. Its local library adopts an open-ended semantic structure organized around capability, domain, and task. When a research intent cannot be matched to an existing task, the system identifies the knowledge gap, and the Benchmark Analysis module selects the CONSTRUCT strategy to directly generate an Evaluation Specification defining the task structure and scoring rules. It then generates and filters instances through pilot examples, coverage control, and multi-stage validation.

Based on this design, we use public benchmark scores and rankings from MMLU-Pro, MBPP, MTEB, SciFact, and IFEval as external references to examine whether our generated benchmarks preserve the relative distinctions among different models. For each task, we independently select a corresponding set of evaluation models, evaluate them on both public and generated benchmarks, and use Spearman's rank correlation coefficient (ρ\rho), Kendall's rank correlation coefficient (τ\tau), and Pearson's correlation coefficient (rr) to measure the consistency of the results.

Using two generators across eight tasks, we construct 16 benchmark variants and conduct 122 model–dataset evaluation runs. Among the 16 comparisons between public and generated benchmarks, 14 achieve Spearman correlations above 0.8. These results demonstrate that Idea2AutoEval can effectively preserve model discrimination patterns across different tasks and model sets, providing empirical support for its further application to novel research tasks.

Reply to W2:

We thank the reviewer for pointing out the importance of computational overhead for the practical applicability of on-demand benchmark construction. To provide a more concrete assessment, we compare the computational costs of Idea2AutoEval with those of the reproduced baselines, BenchMaker and BenchBench. The detailed statistics are reported below.

MethodTaskAPI CallsGen.Acc.Reject.Accept. RateBuild Time
Idea2AutoEvalQA-GPQA2592511797271.31%01:06:29
BenchBenchQA-GPQA36936915021940.65%02:02:24
BenchMakerQA-GPQA54154015039027.78%00:44:37
Idea2AutoEvalCoding-MBPP8813002861495.33%02:29:26
BenchBenchCoding-MBPP26826815011855.97%00:25:44
BenchMakerCoding-MBPP26926815011855.97%00:24:18
Idea2AutoEvalRetrieval-MTEB4934854008582.47%00:42:57
BenchBenchRetrieval-MTEB47447415032431.65%01:12:21
BenchMakerRetrieval-MTEB44043915028934.17%01:09:36
Idea2AutoEvalInstruction-IFEval505005000100.00%00:03:17
BenchBenchInstruction-IFEval2032031505373.89%00:03:57
BenchMakerInstruction-IFEval28728615013652.45%00:10:07

Idea2AutoEval achieves higher instance acceptance rates than both baselines across all four tasks. Aggregating the generated instances across all tasks, Idea2AutoEval generates 1,536 candidates and accepts 1,365, achieving an overall acceptance rate of 88.87%, substantially higher than 45.66% for BenchBench and 39.14% for BenchMaker. These results indicate that the Evaluation Specification, coverage constraints, and multi-stage quality validation help establish explicit instance requirements and reduce invalid generations and subsequent rejections, enabling the framework to maintain high and consistent instance acceptance rates across different task types.

Reply to W3:

We thank the reviewer for raising the important concern regarding potential shared biases between generators and validators. We would like to clarify that the Evaluator in Idea2AutoEval primarily serves as an internal quality validation component for generated benchmarks. It checks the task relevance, answer consistency, and potential duplication of generated instances, rather than directly evaluating the final performance of different models. The actual model performance evaluation is conducted by running the generated benchmarks on the target models and calculating their resulting scores and relative rankings.

Our experiments provide empirical evidence of evaluation stability from both the generator and evaluation-model perspectives. On the one hand, we independently use DeepSeek and GPT-5.6 Luna as benchmark generators. Across the eight task settings, we obtain 16 comparisons between public and generated benchmarks, of which 14 achieve Spearman correlations above 0.8. On the other hand, we employ different evaluation-model sets for different tasks, covering diverse models from multiple providers rather than relying on a fixed model set. In total, we conduct 122 model–dataset evaluation runs, thereby examining the effectiveness of our framework across diverse model combinations and task types.

The experimental results show that our generated benchmarks maintain high ranking consistency with public benchmarks across different generators, task types, and evaluation-model combinations. This provides further empirical support for the soundness of our multi-stage validation mechanism and the reliability of the overall evaluation pipeline.

Reply to W4:

We thank the reviewer for this question. First, we would like to explain the particular characteristics of the GPQA dataset. GPQA is a challenging question-answering benchmark designed for graduate-level scientific reasoning, aiming to evaluate model performance on highly difficult questions. Since such questions require substantial domain knowledge and reasoning capabilities, current generation models may be limited by their own capabilities in understanding and constructing sufficiently challenging questions, potentially resulting in reversed difficulty trends in the generated datasets.

In our experiments, 10 synthetic benchmark variants achieve the intended difficulty ordering, indicating that difficulty control is effective in most settings, although its reliability indeed depends on the task characteristics and generator capabilities. We would like to clarify that difficulty control in Idea2AutoEval does not imply the ability to generate tasks of arbitrary difficulty. In particular, we do not claim that our framework can generate extremely challenging questions beyond the knowledge and reasoning capabilities of its underlying generation models. For extreme benchmarks such as Humanity's Last Exam (HLE), which are designed to test the limits of model capabilities, the questions themselves are designed and written by human experts. Requiring current automated generation frameworks to independently construct such extreme tasks beyond the cognitive capabilities of existing models does not reflect the current state of practical technology.

Our work focuses on constructing reasonable, executable, and verifiable evaluation tasks according to specific research intents, rather than generating challenges that exceed the limits of human or model capabilities. Generating tasks beyond the knowledge and reasoning capabilities of current generators may require stronger generation models or additional constraints defined by domain experts in the future. This represents an objective limitation of current automated generation capabilities rather than a deficiency in the design of our framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.