- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors introduce EvalResearchBench (ERB), which is a benchmark for autonomous evaluation research. The core idea is rather than asking agents to solve a given benchmark, ERB asks them to design the evaluator component itself. Specifically, to select or synthesize questions, implement the graders, run pilots and revise the suite.
The authors evaluate nine model-harness configurations, each through one research episode against a panel of 13 candidate models and 14 target benchmarks. Quality is evaluated throught a score concordance and pairwise agreement with target rankings. Overall, the results show that GPT-6-Astra achieves the highest concordance score while Grok achieves the highest pairwise agreement. The authors also show that a human-designed baseline remains competitive, and run additional analyses looking into each task contributions, question-count sensitivity, weighting, execution costs, and research trajectories.
Critical strengths
- The ERB benchmark captures a meaningful, underexplored portion of a research task rather than only looking at the question generation. I personally find the executable-artifact requirement particularly valuable, in which the research must make decisions about task selection, grading, aggregation, and execution constraints, and then submit something that works under independent assessment. This provides a concrete setting for studying evaluation design as an agent capability.
- The additional analyses shown in the paper helps readers understand the failure modes of the evaluator. The paper shows analysis on component deletions, matched-panel comparisons, and frozen-response reweighting to help pinpoint where the measurement signal comes from. These analyses make the paper useful even where agent-designed evaluators do not clearly beat the baseline.
Critical weaknesses
- The experimental design does not establish how reliably a researcher configuration produces a good evaluator. Since each configuration contributes only one research episode, and the models are evaluated in different harnesses, the observed differences in the paper may rather reflect specific episodes or harness behavior rather than actual differences in the evaluation research capability. The candidate-resampling intervals and question-subset analyses are useful, but they hold the submitted artifacts fixed; they do not measure variation in the research process itself. The paper lacks documentations on the evaluator-level performance overall.
- The primary concordance metric mixes evaluator quality with incompatible target-score scales. It seems that every evaluator returns one unchanged score per domain, which is then compared against several benchmarks with different difficulty levels and scoring conventions. Despite the authors acknowledge this and provide compatibility analyses, the main concordance ranking seems to remain sensitive to target composition.
Questions for authors
- Have any configurations been run more than once? Even repeated runs of a few representative configurations would help understand the capability and variance coming from individual trajectories.
- What is the intended meaning of a single domain score across targets with different score distributions?
- How much positive scoring weight is completed for each candidate and domain? Since Equation 1 show that it renormalizes over completed items, different candidates can receive different tests when some budget stops occur.
- Do the authors have a sense of how the generated questions and graders look like, especially the highest-weight items?
3 comments
Response to Reviewer Luj6 (2/2)
Q4: On generated questions, graders, and high-weight items.
Thank you for asking for concrete examples of the generated questions and graders, especially the highest-weight items. The suites contain different forms of evaluation. Astra's repository-tracing tasks ask candidates to predict cached outputs and recomputation counts after in-place mutation versus replacement of a dependency object. Executing the fixture produces the expected answer. Its professional tasks use rubrics, as illustrated by the memo repair in our response to Reviewer HsKc.
Grok's suite uses short functions, numeric answers, and format checks. Six co-work items carry 15.645% of the co-work weight each, totaling 93.87%. Five of them are short calculations with one exact numeric answer, such as FIFO inventory cost. The sixth is a formatting task. Here is the formatting task, w_if_hold, in full. The candidate receives the system message "Be concise and exact. Follow the requested output format. Do not explain inside answer tags or code fences." and this prompt:
Write a hold note.
- Exactly 3 lines
- Line 1 is exactly STATUS: HOLD
- Line 3 is exactly Owner: Priya
- Line 2 begins with Reason:
- Line 2 contains the digit 9
- No exclamation mark
- Line 2 has 4 to 8 words inclusive
Put the note inside <answer></answer>.
The answer may use at most 768 output tokens, including reasoning. The grader takes the text inside <answer> and runs 7 checks. The note must have exactly 3 lines, with line 1 equal to STATUS: HOLD and line 3 equal to Owner: Priya. Line 2 must start with Reason:, contain 9, and have 4 to 8 words. The note must contain no !. The score is the number of passed checks divided by 7.
Nine candidates pass all 7 checks. The examples below use / to indicate line breaks:
- Kimi K3:
STATUS: HOLD / Reason: awaiting 9 approvals from compliance / Owner: Priya - DeepSeek V4 Flash:
STATUS: HOLD / Reason: 9 items blocked / Owner: Priya - Nemotron 3 Ultra:
STATUS: HOLD / Reason: awaiting approval 9 / Owner: Priya
The other four candidates, DeepSeek-V3.2, Gemma 4 31B, Qwen 3.5, and GPT-OSS-120B, spend all 768 tokens on reasoning and never write the note, so they score 0. Gemma 4 31B, for example, uses its 768 tokens to re-check the rules line by line: "Line 1: Exactly STATUS: HOLD … Line 2: Must start with Reason: . Must contai…". It stops without producing the note.
Across all six items and all 13 candidates, every completed final answer receives full marks. All 11 zeros among the 78 responses come from answers cut off at the token limit. These six items, which carry 93.87% of Grok's co-work weight, separate candidates only by whether they finish within the limit. Reweighting the full suite uniformly within each domain raises Grok's matched-panel Score from 22.18 to 37.03 using the same saved responses and grades. This analysis shows how ERB can identify specific design choices that limit an evaluator's measurement quality and reveal useful information already present in its responses.
Response to the Authors
Thank you for the clarification and additional results! I will maintain my original positive assessment of the paper.
Response to Reviewer Luj6 (1/2)
We thank you for the careful review and for recognizing the value of executable evaluators and the artifact analyses.
W1 & Q1: On research reliability and the role of the harness.
Thank you for suggesting repeated research episodes and raising the question of harness effects. Each ERB entry is a researcher configuration consisting of a model and its harness. ERB assesses the evaluation research capability of this combined system. To measure variation across research episodes, we ran two additional independent Astra episodes with the same model, harness, inputs, and budget ceilings.
Across the three independent episodes, PA ranges from 69.54% to 71.62%, with a standard deviation of 1.06 percentage points. Both new episodes exceed the submitted episode on Score and PA. These results support consistent agreement with target rankings across evaluators independently constructed by the Astra configuration.
W2 & Q2: On the meaning of a domain score across targets.
Thank you for asking us to clarify the meaning of a domain score. A domain score is the evaluator's single, reusable estimate of a candidate's capability in that domain. Comparing this score unchanged with multiple benchmarks tests how well one frozen evaluator captures capability across different tasks in the domain, without target-specific recalibration. PA depends only on candidate order and is therefore unaffected by differences in target score scales. On the full 14-target panel and the scale-compatible 8-target subset, PA remains stable for the evaluators below, changing by at most 2.5 percentage points. Grok leads on both, with PA of 74.87% and 75.06%.
Score, based on Lin's CCC, complements PA by additionally measuring numerical agreement with the target references. Appendix A reports it for both target sets, and Appendix B decomposes it into correlation, location, and spread.
Q3: On completed positive scoring weight.
Thank you for asking us to report completed scoring weight for each candidate and domain. Coverage means the share of a candidate's scoring weight in a domain that comes from items that were actually run. Before the rerun, 12 of the 312 candidate-domain cells across 8 evaluators, 13 candidates, and 3 domains were incomplete, mostly for Qwen3.8 Max and Kimi K3. Overall, 98.6% of the scoring weight had been run. Fable 5.1, Opus 5, and Kimi K3 were complete in every cell.
For the eight main-table evaluators, all missing items resulted from infrastructure issues, including API overload, quota errors, and budget held by failed requests. The separately reported ninth submission, Fable 5, stopped at its assessment budget cap. We reran exactly the missing items with the frozen programs, keeping every existing answer unchanged. Coverage is now 100% for every candidate and domain on the main-table panel. Within each evaluator, all candidates now receive the same complete set of items with the same frozen scoring weights. Score changes by at most 0.92 and PA by at most 0.51 percentage points. The full results appear in the table in our response to Reviewer MwHQ, W2.