- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Idea2AutoEval, an external evaluation component for Auto Research systems. It builds benchmarks on demand from a natural-language research requirement, keeping evaluation design separate from method development. The pipeline has three stages:
- Benchmark knowledge base. Prior benchmarks are standardized as "Benchmark Cards" and routed into a capability–domain–task taxonomy. The taxonomy is iteratively revised using a routing-gap rate and a feedback set of low-confidence or contract-violating routes.
- Requirement-driven construction. The research intent is matched against the knowledge base, and the system selects a REUSE / ADAPT / CONSTRUCT strategy. It then writes an Evaluation Specification (task contract, coverage plan, evaluation contract). Instances are generated per coverage-quota slot. Each instance must pass structural, domain-specific and semantic gates; rejected instances are regenerated for the same slot.
- Release packaging. Model-visible inputs are separated from evaluator-only references and hidden tests.
The authors reconstruct eight existing benchmarks with two generators. The reported rank correlation between synthetic-benchmark and public-leaderboard scores over 5–8 matched models surpasses frontier baselines.
Critical strengths
-
Timely and well-motivated problem framing. Auto Research systems that write both the method and its benchmark risk self-favoring evaluation and answer retrieval. Separating evaluation design and scoring from method iteration is a sensible principle, and making it a reusable component is a useful direction. The design choices follow from concrete failure modes: an explicit Evaluation Specification fixed before generation, quota-based coverage control so that filtering does not silently narrow coverage, and public/private packaging.
-
Comprehensive empirical study. The evaluation spans four task families with different scoring mechanisms (multiple-choice accuracy, executable tests, ranking metrics, verifiable constraints). It uses two independent generators, reports Spearman, Kendall and Pearson correlations, and gives full per-model tables (App. A.4). The authors openly report failure cases: GPQA with Luna (ρ = 0.30), non-monotone refinement, and difficulty calibration failing in 6 of 16 variants. This makes the results easier to interpret.
Critical weaknesses
-
The experiments do not evaluate the claimed contribution: on-demand evaluation for novel research intents. The paper motivates Idea2AutoEval for cases where existing benchmarks do not cover the needed capability (ADAPT/CONSTRUCT), and as protection against self-favoring evaluation. Yet every experiment asks the system to reproduce an existing public benchmark. It "preserves the target capability, task format, evaluation protocol, and primary metric" (l. 347–348) and is judged by agreement with that benchmark's leaderboard. This tests whether an LLM can regenerate an MMLU-Pro/MBPP/IFEval-like test set, not whether the framework produces reliable evaluations where no reference exists. Several consequences follow:
- No ablations. The knowledge base, taxonomy routing and REUSE/ADAPT/CONSTRUCT selection (Sec. 3.1–3.2) are not evaluated. It is not reported which strategy was selected in each setting, and in the baseline comparison all methods receive "the same fixed domain-specific task specification" (l. 954). This seems to bypass these components entirely. There are also no ablations of the coverage controller, the semantic gate or targeted refill, so the source of the gains over BenchMaker/BenchBench is unknown.
- No test of the integrity claim. There is no experiment in an Auto Research loop, and no measurement of whether self-preference or method-favoring bias is reduced, which is the stated motivation. Separating public inputs from private references (hidden tests/labels) is standard benchmark practice and is not shown to address the bias the paper describes.
- Contamination is not controlled. The generators very likely memorized the targeted public benchmarks (MMLU-Pro, GPQA, MBPP, IFEval), and many evaluated models were trained on or tuned toward them. High correlation could therefore reflect near-duplicate regeneration rather than valid new evaluation. No overlap or similarity analysis against the public test sets is reported.
Because the paper's thesis is about reliability beyond existing coverage, I consider this the main reason for rejection.
-
The statistical evidence behind the headline numbers is weak.
- Small model sets. All correlations use only 5–8 models (5 for MMLU-Pro), and these models are mostly older 0.5B–14B checkpoints with large capability gaps (e.g., Qwen2.5-0.5B vs. 14B; BERT/average-word embeddings vs. BGE). Rank agreement on such spread-out models is a low bar, and with n = 8 a single adjacent swap moves ρ by about 0.02–0.05.
- No uncertainty estimates. There are no confidence intervals, permutation p-values or repeated generations: each setting is a single generated benchmark per generator. The "8 of 12 wins" and macro-average ρ = 0.881 vs. baselines (Fig. 2, 4 tasks × 8 models) therefore cannot be separated from noise.
- Discriminative power is not examined. Several synthetic benchmarks hit ceiling or floor effects. On Luna MMLU-Pro all models score 90.75–98.75 (vs. 28–65 publicly), with ties at the top. On Luna MBPP, Mistral scores 0.00 and Gemma/Zephyr tie at 8.04. Spearman correlation hides these effects, even though they matter for a benchmark meant to give feedback to research iterations.
- No direct quality audit. There is no human audit of instance correctness, such as label-error rates or test-case validity, which would be the most direct check of "reliable" evaluation.
Questions for authors
- In Sec. 4.4, what information does the "feedback from the preceding evaluation round" contain? If it includes public-benchmark scores or correlations, refinement directly optimizes the evaluation target. That information would also be unavailable in the intended Auto Research setting, where no public reference exists. Please specify the feedback content and stopping criterion.
- Strategy selection and knowledge base. In each of the eight settings, which strategy (REUSE / ADAPT / CONSTRUCT) was selected? How large is the benchmark library, and how many routing/revision rounds were run? Was the knowledge base used at all in the reported experiments, including the baseline comparison under a "fixed task specification"?
- Data Contamination. Can you report n-gram or embedding-similarity overlap between generated instances and the public test sets (and their training splits)? Can you also report how alignment changes after removing near-duplicates?
- Can you report bootstrap or permutation confidence intervals for ρ and τ, and results over multiple independent generations per setting? Can you state whether Idea2AutoEval's advantage over BenchMaker and BenchBench is significant?
- Baseline fairness. Did all three methods produce the same number of instances, with the same validator models and comparable generation/validation budget (API calls, cost)?
- Pipeline statistics. What are the benchmark sizes, per-gate rejection rates and validator models? Was any human verification of instance correctness performed?
6 comments
W1-3 & Q3:
We thank the reviewer for raising this important issue. We fully acknowledge the concern regarding data contamination. To address this concern, we have added a data overlap analysis between the generated instances and the available train, validation, and test splits of the corresponding public benchmarks. We examine potential contamination at three levels:
-
Normalized exact matching, to identify instances with identical content.
-
Word 5-gram overlap. We consider an instance a textual near-duplicate if its Jaccard similarity is at least 0.80, or if its containment is at least 0.80 with at least 20 shared 5-grams.
-
Embedding cosine similarity of the main task text, computed using the frozen
BAAI/bge-small-en-v1.5model. Following prior work that uses a cosine similarity threshold of 0.95 to screen for semantic matches or near-duplicate instances, we adopt 0.95 as a high-precision screening threshold.
Exact matching and n-gram checks cover all relevant fields, including questions, candidate answers, code, retrieval queries, and candidate documents. Embedding similarity primarily compares text that carries the task semantics. For the MTEB dataset, which contains 8,453,892 public query and document components, we first perform full-scale exact matching and n-gram retrieval, followed by BGE-based reranking of the retrieved candidates.
Among the 5,628 released instances across 16 generated versions, the primary threshold flags 10 instances, accounting for 0.18% of all instances. All 10 instances come from the DeepSeek-generated Retrieval-MTEB-Reranking benchmark: 10 out of 400 instances (2.50%). These include one query that exactly matches an instance in the public training split, one n-gram near-duplicate instance, and 10 instances with embedding similarity scores of at least 0.95. The detection categories overlap. No near-duplicate instances are detected in the remaining 15 generated versions under the primary threshold.
| Benchmark | Generator | Instances | Exact Duplicates | N-gram Near-Duplicates | Semantic Similarity ≥ 0.95 | Final Flagged |
|---|---|---|---|---|---|---|
| QA-MMLU-Pro | DeepSeek | 400 | 0 | 0 | 0 | 0 |
| QA-MMLU-Pro | Luna | 400 | 0 | 0 | 0 | 0 |
| QA-GPQA | DeepSeek | 179 | 0 | 0 | 0 | 0 |
| QA-GPQA | Luna | 179 | 0 | 0 | 0 | 0 |
| Coding-MBPP | DeepSeek | 286 | 0 | 0 | 0 | 0 |
| Coding-MBPP | Luna | 286 | 0 | 0 | 0 | 0 |
| Coding-Aider133 | DeepSeek | 133 | 0 | 0 | 0 | 0 |
| Coding-Aider133 | Luna | 133 | 0 | 0 | 0 | 0 |
| Retrieval-MTEB | DeepSeek | 400 | 1 | 1 | 10 | 10 (2.50%) |
| Retrieval-MTEB | Luna | 400 | 0 | 0 | 0 | 0 |
| Retrieval-SciFact | DeepSeek | 416 | 0 | 0 | 0 | 0 |
| Retrieval-SciFact | Luna | 416 | 0 | 0 | 0 | 0 |
| Instruction-IFEval | DeepSeek | 500 | 0 | 0 | 0 | 0 |
| Instruction-IFEval | Luna | 500 | 0 | 0 | 0 | 0 |
| Instruction-IFBench-OOD | DeepSeek | 500 | 0 | 0 | 0 | 0 |
| Instruction-IFBench-OOD | Luna | 500 | 0 | 0 | 0 | 0 |
| Total | — | 5,628 | 1 | 1 | 10 | 10 (0.18%) |
We further removed these 10 instances and recomputed the per-instance MAP scores for all eight models using the remaining 390 instances. After deduplication, the relative rankings of all models remained unchanged. The changes in MAP scores for individual models ranged from to . The correlations between the public benchmark and the deduplicated benchmark remained:
| Model | Original MAP | Deduplicated MAP | Change | Deduplicated Rank |
|---|---|---|---|---|
| bge-large-en-v1.5 | 0.7358 | 0.7381 | +0.0023 | 1 |
| all-mpnet-base-v2 | 0.7344 | 0.7361 | +0.0017 | 2 |
| bge-base-en-v1.5 | 0.7230 | 0.7271 | +0.0041 | 3 |
| gtr-t5-base | 0.7124 | 0.7136 | +0.0013 | 4 |
| gte-small | 0.7117 | 0.7127 | +0.0010 | 5 |
| contriever-base-msmarco | 0.6630 | 0.6672 | +0.0042 | 6 |
| average-word-embeddings | 0.6382 | 0.6419 | +0.0038 | 7 |
| bert-base-uncased | 0.5492 | 0.5537 | +0.0045 | 8 |
Across all 5,628 released instances, only 10 were flagged under the primary detection criteria, accounting for 0.18% of the total, and all were concentrated in a single generated version. No near-duplicate instances were detected in the remaining 15 versions. More importantly, after removing all flagged instances, the relative rankings of all eight evaluated models remained exactly unchanged. The Spearman and Kendall correlations between the public benchmark and the deduplicated benchmark both remained at , while the changes in individual models' MAP scores were limited to to . These results demonstrate that the high ranking consistency observed between the public and generated benchmarks is not driven by a small number of duplicated or highly similar instances.
Reply to W2&Q4:
We thank the reviewer for the detailed suggestions regarding statistical uncertainty and model discrimination capability.
W2-1: We would like to clarify that the reviewer's observation that each task uses 5–8 models does not mean that our entire experimental study involves only 5–8 models. In fact, we independently selected different evaluation model sets according to the characteristics of each task. Across eight tasks, we conducted a total of 122 model–dataset evaluations. These models come from different providers and cover diverse model types and capability levels. The specific models are listed below:
| Benchmark | Models | Evaluated Models |
|---|---|---|
| QA–MMLU-Pro | 5 | Phi-3-mini, Llama-3-8B, OpenChat-3.5, MAP-Neo-7B-Instruct, Zephyr-7B. (Abdin et al., 2024; Grattafiori et al., 2024; Wang et al., 2024a; Zhang et al., 2024; Tunstall et al., 2023) |
| QA–GPQA | 8 | Gemma-2-9B, Llama-3.1-8B, Gemma-7B, Meta-Llama-3-8B, Mistral-v0.3, StableLM, Llama-2-7B-Chat, Mistral-v0.1. (Grattafiori et al., 2024; Jiang et al., 2023; Bellagente et al., 2024; Touvron et al., 2023) |
| Coding–MBPP | 8 | OpenCoder, Magicoder, Llama-3.1, Meta-Llama, WizardCoder, Mistral, Gemma, Zephyr. (Huang et al., 2025; Wei et al., 2023; Grattafiori et al., 2024; Jiang et al., 2023; Gemma Team, 2024; Tunstall et al., 2023) |
| Coding–Aider133 | 8 | Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-0.5B-Instruct, Yi-Coder-9B-Chat, Llama-3.1-8B-Instruct, OpenCodeInterpreter-DS-6.7B, Hermes-3-Llama-3.1-8B, Llama-3.2-3B-Instruct, Granite-3.0-8B-Instruct. (Hui et al., 2024; Young et al., 2024; Zheng et al., 2024; Quesnelle & Guang, 2024; Granite Team, 2024) |
| Retrieval–MTEB-Reranking | 8 | bge-large-en-v1.5, all-mpnet-base-v2, bge-base-en-v1.5, all-MiniLM-L12-v2, gtr-t5-base, contriever-base-msmarco, average-word, bert-base-uncased. (Xiao et al., 2024; Song et al., 2020; Wang et al., 2020; Ni et al., 2022; Izacard et al., 2021; Devlin et al., 2019) |
| Retrieval–SciFact | 8 | gte-base, bge-large, bge-small, gte-tiny, contriever, gtr-t5, multi-qa-MiniLM, average-word. (Li et al., 2023; Xiao et al., 2024; Izacard et al., 2021; Ni et al., 2022; Wang et al., 2020) |
| Instruction–IFEval | 8 | Qwen2.5-14B, Qwen2.5-7B, Qwen2.5-3B, Mistral, Aya-23, Qwen2.5-1.5B, DeepSeek-LLM, Qwen2.5-0.5B. (Yang et al., 2024; Jiang et al., 2023; Aryabumi et al., 2024; Bi et al., 2024) |
| Instruction–IFBench-OOD | 8 | RAIF-Qwen2.5-7B, Qwen2.5-7B, RAIF-DeepSeek-Qwen-7B, Ministral-8B, Qwen2.5-1.5B, RAIF-DeepSeek-Qwen-1.5B, RAIF-DeepScaleR-1.5B, DeepSeek-R1-Distill. (Qin et al., 2026; Yang et al., 2024; Jiang et al., 2023; Guo et al., 2025) |
Overall, our experiments cover a diverse set of models far beyond the number used in any individual task. Rather than repeatedly evaluating the framework using the same fixed model set, we independently examine its effectiveness across different tasks and different model combinations. This cross-task and cross-model experimental design further examines whether the framework depends on any particular model set and provides empirical evidence of its generalization capability.
We sincerely thank the reviewer for the positive recognition of our research motivation, methodological design, and experimental work. We are pleased that the reviewer acknowledges the necessity of decoupling evaluation design from method iteration, as well as the rationale behind our design choices, including explicit evaluation specifications, coverage constraints, and public–private information separation. The reviewer's positive assessment of our experimental coverage, diversity of evaluation metrics, and transparent reporting of failure cases is highly meaningful to our work.
Reply to W1 & Q3:
We thank the reviewer for the detailed comments regarding experimental scope, ablation studies, integrity, and data contamination. First, for genuinely novel research tasks with no prior studies, if no existing benchmarks, public leaderboards, or reference evaluation results are available, there is no reliable external ranking reference against which to directly compare generated benchmarks. Therefore, we choose to use existing public leaderboards to validate the effectiveness and reliability of our proposed framework, with the goal of further extending it to genuinely novel research tasks and previously unexplored domains based on the established reliability of the framework.
W1-1: We propose Idea2AutoEval as a complete automated benchmark construction framework with explicit procedural dependencies, rather than a simple combination of multiple independently executable modules. Taxonomy Construction, Semantic Routing, REUSE/ADAPT/CONSTRUCT strategy selection, Coverage Control, and Semantic Validation jointly form the complete construction pipeline. These stages have explicit dependencies, where the output of each preceding stage provides the foundation for the subsequent stages. Therefore, directly removing a key component may disrupt the integrity of the entire pipeline, making it difficult to independently measure its contribution while preserving the original construction objectives.
W1-2: We thank the reviewer for the concern regarding our claim of evaluation integrity. We would like to clarify that Idea2AutoEval is not a complete Auto Research system, but an independent and reusable external evaluation component. Its objective is to separate benchmark design and scoring from the research iteration of the methods being evaluated. Therefore, our integrity claim primarily refers to the structural decoupling mechanism provided by the framework, rather than claiming to have quantitatively measured the reduction of self-favoring bias within a complete Auto Research loop.
Specifically, Idea2AutoEval freezes the Evaluation Specification before instance generation, separates the evaluation construction component from the method iteration process, and isolates model-visible inputs from evaluator-private references, labels, and hidden tests. These design choices prevent the evaluated methods from directly accessing private evaluation references or modifying task definitions and scoring rules based on their own outputs, thereby reducing the risk of evaluation design being influenced by method iteration at the mechanism level.
Therefore, the experiments in this paper validate whether the external evaluation component can independently construct benchmarks with stable model-discrimination capabilities and whether the evaluation results of generated benchmarks remain consistent with established external evaluation systems. We do not interpret our current results as a direct causal measurement of the reduction in self-preference or method-favoring bias. The latter would require integrating the evaluation component into a complete Auto Research system and comparing research iteration outcomes before and after decoupling, which constitutes a system-level research question.