acceptodds
Under review as a conference paper at ICLR 2027

EvalResearchBench: Can AI Agents Design Their Own Evaluations?

10 views

Abstract

Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.

Primary area:
datasets and benchmarks
Keywords:
llm, agent, RSI, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Luj6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce EvalResearchBench (ERB), which is a benchmark for autonomous evaluation research. The core idea is rather than asking agents to solve a given benchmark, ERB asks them to design the evaluator component itself. Specifically, to select or synthesize questions, implement the graders, run pilots and revise the suite.

The authors evaluate nine model-harness configurations, each through one research episode against a panel of 13 candidate models and 14 target benchmarks. Quality is evaluated throught a score concordance and pairwise agreement with target rankings. Overall, the results show that GPT-6-Astra achieves the highest concordance score while Grok achieves the highest pairwise agreement. The authors also show that a human-designed baseline remains competitive, and run additional analyses looking into each task contributions, question-count sensitivity, weighting, execution costs, and research trajectories.

Critical strengths
  1. The ERB benchmark captures a meaningful, underexplored portion of a research task rather than only looking at the question generation. I personally find the executable-artifact requirement particularly valuable, in which the research must make decisions about task selection, grading, aggregation, and execution constraints, and then submit something that works under independent assessment. This provides a concrete setting for studying evaluation design as an agent capability.
  2. The additional analyses shown in the paper helps readers understand the failure modes of the evaluator. The paper shows analysis on component deletions, matched-panel comparisons, and frozen-response reweighting to help pinpoint where the measurement signal comes from. These analyses make the paper useful even where agent-designed evaluators do not clearly beat the baseline.
Critical weaknesses
  1. The experimental design does not establish how reliably a researcher configuration produces a good evaluator. Since each configuration contributes only one research episode, and the models are evaluated in different harnesses, the observed differences in the paper may rather reflect specific episodes or harness behavior rather than actual differences in the evaluation research capability. The candidate-resampling intervals and question-subset analyses are useful, but they hold the submitted artifacts fixed; they do not measure variation in the research process itself. The paper lacks documentations on the evaluator-level performance overall.
  2. The primary concordance metric mixes evaluator quality with incompatible target-score scales. It seems that every evaluator returns one unchanged score per domain, which is then compared against several benchmarks with different difficulty levels and scoring conventions. Despite the authors acknowledge this and provide compatibility analyses, the main concordance ranking seems to remain sensitive to target composition.
Questions for authors
  • Have any configurations been run more than once? Even repeated runs of a few representative configurations would help understand the capability and variance coming from individual trajectories.
  • What is the intended meaning of a single domain score across targets with different score distributions?
  • How much positive scoring weight is completed for each candidate and domain? Since Equation 1 show that it renormalizes over completed items, different candidates can receive different tests when some budget stops occur.
  • Do the authors have a sense of how the generated questions and graders look like, especially the highest-weight items?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Luj6 (1/2)

We thank you for the careful review and for recognizing the value of executable evaluators and the artifact analyses.

W1 & Q1: On research reliability and the role of the harness.

Thank you for suggesting repeated research episodes and raising the question of harness effects. Each ERB entry is a researcher configuration consisting of a model and its harness. ERB assesses the evaluation research capability of this combined system. To measure variation across research episodes, we ran two additional independent Astra episodes with the same model, harness, inputs, and budget ceilings.

Astra iterative episodeScorePA (%)Construction cost ($)
Submitted22.7969.5476.9
New 131.8671.6261.3
New 227.6370.8962.5
Mean ± SD27.43 ± 4.5470.68 ± 1.0666.9 ± 8.7

Across the three independent episodes, PA ranges from 69.54% to 71.62%, with a standard deviation of 1.06 percentage points. Both new episodes exceed the submitted episode on Score and PA. These results support consistent agreement with target rankings across evaluators independently constructed by the Astra configuration.

W2 & Q2: On the meaning of a domain score across targets.

Thank you for asking us to clarify the meaning of a domain score. A domain score is the evaluator's single, reusable estimate of a candidate's capability in that domain. Comparing this score unchanged with multiple benchmarks tests how well one frozen evaluator captures capability across different tasks in the domain, without target-specific recalibration. PA depends only on candidate order and is therefore unaffected by differences in target score scales. On the full 14-target panel and the scale-compatible 8-target subset, PA remains stable for the evaluators below, changing by at most 2.5 percentage points. Grok leads on both, with PA of 74.87% and 75.06%.

EvaluatorPA (%), all 14 targetsPA (%), compatible 8 targets
Astra69.5467.08
Grok 4.774.8775.06
Fable 5.167.8069.57

Score, based on Lin's CCC, complements PA by additionally measuring numerical agreement with the target references. Appendix A reports it for both target sets, and Appendix B decomposes it into correlation, location, and spread.

Q3: On completed positive scoring weight.

Thank you for asking us to report completed scoring weight for each candidate and domain. Coverage means the share of a candidate's scoring weight in a domain that comes from items that were actually run. Before the rerun, 12 of the 312 candidate-domain cells across 8 evaluators, 13 candidates, and 3 domains were incomplete, mostly for Qwen3.8 Max and Kimi K3. Overall, 98.6% of the scoring weight had been run. Fable 5.1, Opus 5, and Kimi K3 were complete in every cell.

For the eight main-table evaluators, all missing items resulted from infrastructure issues, including API overload, quota errors, and budget held by failed requests. The separately reported ninth submission, Fable 5, stopped at its assessment budget cap. We reran exactly the missing items with the frozen programs, keeping every existing answer unchanged. Coverage is now 100% for every candidate and domain on the main-table panel. Within each evaluator, all candidates now receive the same complete set of items with the same frozen scoring weights. Score changes by at most 0.92 and PA by at most 0.51 percentage points. The full results appear in the table in our response to Reviewer MwHQ, W2.

AuthorsOpenReview ↗

Response to Reviewer Luj6 (2/2)

Q4: On generated questions, graders, and high-weight items.

Thank you for asking for concrete examples of the generated questions and graders, especially the highest-weight items. The suites contain different forms of evaluation. Astra's repository-tracing tasks ask candidates to predict cached outputs and recomputation counts after in-place mutation versus replacement of a dependency object. Executing the fixture produces the expected answer. Its professional tasks use rubrics, as illustrated by the memo repair in our response to Reviewer HsKc.

Grok's suite uses short functions, numeric answers, and format checks. Six co-work items carry 15.645% of the co-work weight each, totaling 93.87%. Five of them are short calculations with one exact numeric answer, such as FIFO inventory cost. The sixth is a formatting task. Here is the formatting task, w_if_hold, in full. The candidate receives the system message "Be concise and exact. Follow the requested output format. Do not explain inside answer tags or code fences." and this prompt:

Write a hold note.
- Exactly 3 lines
- Line 1 is exactly STATUS: HOLD
- Line 3 is exactly Owner: Priya
- Line 2 begins with Reason:
- Line 2 contains the digit 9
- No exclamation mark
- Line 2 has 4 to 8 words inclusive
Put the note inside <answer></answer>.

The answer may use at most 768 output tokens, including reasoning. The grader takes the text inside <answer> and runs 7 checks. The note must have exactly 3 lines, with line 1 equal to STATUS: HOLD and line 3 equal to Owner: Priya. Line 2 must start with Reason:, contain 9, and have 4 to 8 words. The note must contain no !. The score is the number of passed checks divided by 7.

Nine candidates pass all 7 checks. The examples below use / to indicate line breaks:

  • Kimi K3: STATUS: HOLD / Reason: awaiting 9 approvals from compliance / Owner: Priya
  • DeepSeek V4 Flash: STATUS: HOLD / Reason: 9 items blocked / Owner: Priya
  • Nemotron 3 Ultra: STATUS: HOLD / Reason: awaiting approval 9 / Owner: Priya

The other four candidates, DeepSeek-V3.2, Gemma 4 31B, Qwen 3.5, and GPT-OSS-120B, spend all 768 tokens on reasoning and never write the note, so they score 0. Gemma 4 31B, for example, uses its 768 tokens to re-check the rules line by line: "Line 1: Exactly STATUS: HOLD … Line 2: Must start with Reason: . Must contai…". It stops without producing the note.

Across all six items and all 13 candidates, every completed final answer receives full marks. All 11 zeros among the 78 responses come from answers cut off at the token limit. These six items, which carry 93.87% of Grok's co-work weight, separate candidates only by whether they finish within the limit. Reweighting the full suite uniformly within each domain raises Grok's matched-panel Score from 22.18 to 37.03 using the same saved responses and grades. This analysis shows how ERB can identify specific design choices that limit an evaluator's measurement quality and reveal useful information already present in its responses.

Reviewer_Luj6OpenReview ↗

Response to the Authors

Thank you for the clarification and additional results! I will maintain my original positive assessment of the paper.

Reviewer_HsKcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

With RSI in consideration, this paper asks question whether a model can select its own evaluation benchmarks. The authors assume access to target materials, development references, candidate APIs, and fixed time and API budgets. Given this, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests. They then create an executable evaluator for domains such as coding, co-work, and reasoning.

Critical strengths

I see the following strengths in the paper:

1/ The problem seem novel and easy to understand. Evaluation will be a critical bottleneck before agents/models can evaluate themselves and the paper sets this problem very well. 2/ The design seem to be intuitive and follows general ML principles. The definition of the eval is on a held out eval set and the final evaluations are done on a separate set of evaluations.

Critical weaknesses

I see the following weaknesses:

1/ Methodological novelty: While the paper proposes a new problem, the technical novelty aspect is a bit lighter. Each technical building block is an existing tool cobbled together.

2/ Unclear whether the approach will generalize: As the use cases and capabilities of agents improve, it is unclear whether the approach will stand the test of generalization to new domains. Particularly where grader construction is the hard part. There is some evidence in the paper where fitting on visible references does not generalize to out-of-scope or held out candidates.

3/

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer HsKc

We thank you for recognizing the importance of the problem and the held-out design.

W1: On the methodological contribution.

Thank you for asking us to clarify the methodological contribution. ERB introduces a task and an independent assessment protocol for autonomous evaluation research. The agent is responsible for building the measurement instrument. It chooses what evidence to collect, selects or writes tasks, implements graders, and uses a limited pilot budget to test and revise its design. The resulting program is frozen and assessed on withheld candidates and target references. ERB tests whether these research decisions yield informative, affordable measurements beyond the development feedback available to the agent. The experiments reveal concrete consequences of these decisions. Grok's saved responses contain more signal than its chosen weights use. Uniform reweighting raises its matched-panel Score from 22.18 to 37.03. Pilot trajectories also show how agents identify and repair rubric defects before submission. Random120, a human-designed public-task sampler, provides a competitive baseline in pairwise agreement.

W2: On new domains without reference benchmarks, and difficult graders.

Thank you for raising the questions of generalization and grader construction. References serve to score the researcher. The evaluator itself never uses them at run time. ERB measures the ability to design tasks and graders from a domain description and revise them through pilots. To examine this ability without benchmark-specific guidance, we ran a no-reference episode. The researcher receives only the broad domain names of coding, co-work, and reasoning, with no target benchmark names, task materials, trajectories, reference scores, or Internet access. It may still pilot development models and see their raw scores. Its frozen evaluator is assessed exactly like the others.

Astra conditionScorePA (%)
Original protocol (visible targets and references)22.7969.54
No visible benchmarks or references5.5666.38

Without visible references, Astra still achieves 66.38% PA, only 3.2 points below the original protocol. Its Score is 5.56 compared with 22.79 under the original protocol. The agent retains useful candidate ordering, while numerical agreement with target scores is harder to achieve from a domain description alone.

Grader construction is part of what ERB tests, and co-work tasks are rubric-graded open-ended work. In the saved research trajectories, Astra repaired a memo task whose rubric required "mean close_hours" that the brief never asked for. Before freezing, it also relaxed a cloud-migration rubric that prescribed one architecture to criteria that "accept technically coherent managed alternatives".

We also assessed the submitted evaluators on the nine sealed candidates to examine transfer to unseen models. Ten targets retain at least six matched candidates, and both columns below use these same targets.

EvaluatorScore / PA, all 13 candidatesScore / PA, sealed candidates only
Astra30.16 / 74.0935.66 / 78.90
Grok 4.721.50 / 76.7317.90 / 75.41
Fable 5.124.57 / 70.1129.67 / 75.28
Opus 530.23 / 67.9737.57 / 74.76
Kimi K322.85 / 74.2624.95 / 78.58
Sol18.88 / 63.7823.63 / 64.40
GLM-5.320.75 / 68.0425.39 / 75.24
DeepSeek V4 Pro10.83 / 63.6215.21 / 70.50

On these ten targets, seven of the eight evaluators improve on both Score and PA when assessment is restricted to sealed candidates.

The no-reference experiment tests evaluator construction without access to an existing benchmark, while retaining hidden references for independent assessment. In a domain with no established benchmark, the same research workflow could support evaluator design, but its quality would need to be validated using independent expert judgments or domain-specific task outcomes.

Dear Reviewer HsKc,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would greatly appreciate it if you could review our responses and let us know whether they address your concerns.

If any points remain unclear or unresolved, please let us know, and we would be happy to provide further clarification before the discussion ends. We would also appreciate it if you could reconsider your assessment after reviewing our responses.

Thank you for your time and consideration.

Sincerely, The Authors

Reviewer_MwHQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EvalResearchBench, a benchmark for evaluating agents’ ability to design inexpensive evaluations that reproduce the scores and rankings of more costly benchmarks. Under fixed time and API budgets, agents select or generate tasks, implement graders, conduct pilot experiments, and freeze their evaluators. The study evaluates nine researcher configurations across coding, co-work, and reasoning, using held-out candidates and target references. It also analyzes task selection, aggregation weights, execution costs, and research trajectories.

Critical strengths
  1. The paper addresses an important aspect of autonomous research: constructing useful evaluation feedback. The task includes substantive decisions about experimental design, grading, and resource allocation.
  2. The protocol includes independent assessment, held-out candidates and references, and a competitive public-task baseline.
  3. The artifact analyses are informative. In particular, the weighting counterfactual and grader-repair examples reveal concrete successes and limitations beyond aggregate rankings.
Critical weaknesses
  1. Each researcher configuration contributes only one research episode, limiting conclusions about stable differences between systems. Repeated episodes and a non-iterative design baseline would better establish the added value of autonomous experimentation.
  2. Some results depend on execution conditions. Renormalizing over completed items may make candidate scores less comparable, while substantial answer truncation complicates the interpretation of high-value components. Broader coverage reporting and targeted answer-budget sensitivity checks would strengthen the findings.
Questions for authors
  1. How stable are the results across repeated research episodes, and how much does iteration improve over a single-pass design?
  2. Do the main findings hold on commonly completed items and under modest changes to answer budgets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer MwHQ (1/2)

We thank you for the positive assessment and the concrete suggestions. We address each of your points below.

W1 & Q1: On repeated research episodes and the value of iteration.

Thank you for suggesting repeated research episodes and a comparison with a single-pass design. The submission contains one research episode per configuration, and its resampling analyses hold each evaluator fixed. To measure the research process itself, we ran two additional independent Astra episodes under the original protocol and a no-pilot control. The control receives the same materials, references, deadline, and budget ceilings but cannot call any development model or receive pilot feedback. It designs once and submits. Every submitted evaluator undergoes the same independent assessment. The iterative results below report the mean and standard deviation across the three episodes.

Astra research conditionEpisodesScorePA (%)Construction cost ($)
No pilots112.1763.8331.6
Iterative research327.43 ± 4.5470.68 ± 1.0666.9 ± 8.7

With pilots, each of the three independent episodes exceeds the no-pilot control on both Score and PA. Score ranges from 22.79 to 31.86 against 12.17 for the control, and PA ranges from 69.54% to 71.62% against 63.83%, at about twice the construction cost. The saved trajectories in Appendix E show how feedback changes a design. Astra found that a memo rubric required a statistic absent from the task brief and added the requirement to the brief. In a cloud-migration task, it relaxed rubric criteria that prescribed one architecture despite an open-ended brief.

AuthorsOpenReview ↗

Response to Reviewer MwHQ (2/2)

W2 & Q2: Unfinished items and answer length.

Thank you for suggesting broader coverage reporting and checks on answer-budget sensitivity. We examined unfinished items and reran Astra's spreadsheet tasks under the original and doubled answer limits.

Equation 1 averages only over items that were actually run. Wrong, empty and cut-off answers still count and keep their grade. Only items that were never run are left out. We checked every such item. For the eight main-table evaluators, all missing items resulted from infrastructure issues on our side, including API overload, quota errors, and budget held by failed requests. The separately reported ninth submission, Fable 5, stopped at its assessment budget cap. We reran exactly the missing items with the same frozen programs, prompts, length limits and graders, keeping every existing answer unchanged. All eight main-table evaluators now have results for every item on all 13 candidates. The table reports results on the main-table panel of 13 candidates and 14 targets.

EvaluatorCoverage before rerun (%)Score after rerun (change)PA (%) after rerun (change)
Astra99.223.07 (+0.28)69.91 (+0.38)
Grok 4.796.122.56 (−0.03)74.68 (−0.19)
Fable 5.1100.021.96 (0.00)67.80 (0.00)
Opus 5100.021.79 (0.00)64.88 (0.00)
Kimi K3100.018.03 (0.00)74.58 (0.00)
Sol95.717.46 (+0.47)63.83 (+0.51)
GLM-5.397.715.19 (+0.92)64.98 (+0.27)
DeepSeek V4 Pro99.910.55 (+0.06)63.50 (0.00)

Coverage is the share of scoring weight that comes from items that were run, averaged over the 39 candidate-domain cells of each evaluator. Before the rerun, 12 of all 312 candidate-domain cells were incomplete, mostly for Qwen3.8 Max and Kimi K3. Overall, 98.6% of the scoring weight had been run. After the rerun, every candidate answers the same complete set of items within each evaluator, so these are also the results on commonly completed items. Score changes by at most 0.92 and PA by at most 0.51 points. The Score ranking of the eight evaluators does not change. In PA, GLM-5.3 and Opus 5 swap places, as do Sol and DeepSeek V4 Pro. Both pairs were separated by less than 0.5 points. Averaging over finished items therefore did not affect our results.

For answer length, the researcher sets the maximum as part of its evaluator, and ERB keeps it fixed. This choice affects evaluator performance and is one of the design decisions ERB tests. To examine its effect, we reran Astra's 8 spreadsheet tasks, where answers were cut off most often, with the original limit and with twice the limit. The three OpenAI candidates retain their original answers because their API route did not allow us to set this limit.

RunAnswers cut off (out of 104)Co-work PA (%) with these 8 tasksCo-work PA (%) without them
Original results4677.9865.61
Rerun, same limit3375.6765.61
Rerun, 2× limit1974.6865.61

The panel contains 104 answers from 8 tasks and 13 candidates. With twice the limit, far fewer answers are cut off, and these 8 tasks still raise co-work PA from 65.61% to 74.68%, a gain of 9.1 points. Removing 8 randomly chosen co-work tasks changes PA by at most about 3 points, with 95% of 1,000 random removals falling between −2.6 and +3.2. The two reruns differ by only 1.0 point, less than the 2.3-point change from simply rerunning with the same limit. So the length limit has some effect, but most of the signal comes from the tasks themselves.

Thank you for the detailed response. It largely addresses my concerns, and I am happy to maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.