acceptodds
Under review as a conference paper at ICLR 2027

SE-Eval: Verifiable Benchmark Evolution for Adjacent Model Releases

Abstract

Static benchmarks can lose resolution precisely where rapid model development needs it most: between adjacent releases near ceiling. We introduce SE-Eval, a programmatically grounded loop that generates candidate items, verifies unique solutions, measures an ordered model-version sequence, and selects a bounded item pool for the next round. On three pre-specified Qwen seeds, SE-Eval increases adjacent-release disagreement over a frozen pool by +0.0451 [+0.0251, +0.0660], a repeatable but sub-minimum-detectable-effect gain. A fresh-seed replication analyzed under a protocol fixed before accepted outcomes reveals a family-dependent boundary: +0.0396 on Qwen, -0.0893 on DeepSeek, and +0.1673 on Kimi with a family-disjoint generator. In a saturation diagnostic, the three newest Qwen releases score 100% on frozen items but only 83-93% on generated items, showing retained frontier headroom. A scoring-fault diagnostic shows why resolution alone is insufficient: placeholder labels reduce true-gold boundary disagreement from 0.56 to 0.28 while still producing smooth trajectories. Together, these results establish self-evolving benchmarks as longitudinal measurement instruments whose value depends on transition structure, verifiable grounding, and validation across model families.

Primary area:
datasets and benchmarks
Keywords:
self-evolving evaluation, dynamic benchmarks, large language models, benchmark validity, programmatic verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_riLfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SE-Eval maintains an evolving benchmark to distinguish adjacent language model releases. Candidates come from a failure-driven generator and pass an executable solution check. The controller selects a bounded pool using model disagreement and retains three reference items. Fresh-seed puzzle experiments find positive gains on Qwen and Kimi and a decline on DeepSeek. Logged streams allow selection to be compared with matched random retirement. Further analyses audit item quality and scoring errors and explore transfer to Python synthesis.

Critical strengths
  1. A relevant evaluation objective with an auditable selection analysis. Static benchmarks can obscure differences among recent releases even when their answers remain valid. SE-Eval makes adjacent-release resolution an explicit objective and implements an interesting evaluation scheme.
Critical weaknesses

The results are inconclusive. You need better-designed experiments, not just more experiments. Your underlying approach is reasonable. However, you must establish that it measures meaningful model differences and that repeated adaptation adds value. See remarks in questions section

Questions for authors

Explain why the iterative loop is needed. Can you compare the full method with one-shot generation and selection, and with random refresh, using matched pool sizes and budgets Do experiments that justify either the existing method, a revised selection objective, or a simpler one-shot method. The present evidence does not decide among them.

1 comment

Dear Reviewer riLf,

Thank you for your thoughtful feedback. We appreciate that you found the goal of distinguishing adjacent model releases relevant and the item selection analysis auditable. Your questions about reproducibility and the gains from iteration have helped us refine our interpretation of the results, and we will reflect these changes in the revised manuscript.


  1. W: For the third Qwen seed, 3.5-plus and 3.7-max agreed on which items they answered correctly in the final static pool. The final evolved pool contained three entries (two distinct items) that only the newer release answered correctly. We will add this example and outline a follow-up evaluation using independent responses.
  2. Q1: Iteration lets errors exposed in one round guide item generation in the next. Across five Qwen seeds, about 97% of the mean gain in discrimination came in the first round. We will clarify that further rounds can be used as needed to probe newly exposed errors.
  3. Q2: We will outline how to compare SE-Eval, one-shot generation and selection, and random refresh at matched budgets and final pool sizes, using independent responses for evaluation. The design will also match pool sizes round by round for the multi-round methods and compare selection rules separately on the same candidate items.

Thank you again for your review.

Reviewer_qyyoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SE-Eval, a programmatically grounded generate-verify-measure-select loop for the purpose of maintaining adjacent-version resolution as static benchmarks approach ceiling. It generates candidate items, verifies unique solutions, measures disagreement across ordered model releases, and updates a bounded item pool. Results show that the performance is family-dependent. A placeholder-label diagnostic shows that smooth trajectories can still be valid and therefore a need for verifiable grounding is needed.

Critical strengths
  • The paper addresses a very important and practical problem on static benchmarks that they can lose resolution between adjacent model releases while performance approaches ceiling.
  • Clear methodology design that combines essential aspects from generation, verification, measurement to selection.
Critical weaknesses
  • The evaluation concerns mainly one domain which is knights-and-knaves puzzles. It limits the generalizability to broader settings in real world.
  • Some generated items are still ambiguous to human despite passing the programmatic unique-solution check. And human evaluation is offline, the items can still enter the pool.
Questions for authors
  • Can you provide multiple-seed results in another task domain?
  • Would filtering out ambiguous items change the adjacent-version disagreement results?
1 comment

Dear Reviewer qyyo,

Thank you for taking the time to review our work. We appreciate your recognition of the practical significance of the problem and the clarity of the evaluation procedure. In response to your suggestions, we have clarified our discussion of the cross-domain evidence and ambiguity in item wording and will incorporate these changes into the revised manuscript.


  1. W1 & Q1: The single-seed Python function synthesis experiment in Appendix E.11 shows that SE-Eval can use a unit-test verifier. We will clarify the scope of this evidence and outline a matched-budget, multi-seed comparison as a follow-up.
  2. W2 & Q2: The automated check verifies that the formalized item has a unique solution; human ratings did not determine which items entered the pool. We will add an offline analysis of one Qwen seed: excluding the four items flagged by human raters increased normalized disagreement AUC from 0.2110 to 0.2138. We will also describe how to conduct a blinded review of item wording before admission.

Thank you again for your review.

Reviewer_TGCYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SE-Eval, a framework that evolves benchmark items to preserve discrimination between adjacent model releases. It combines failure-driven item generation, executable unique-solution verification, and behavior-guided item selection while retaining fixed reference items. Experiments primarily on knights-and-knaves puzzles show modest positive gains for Qwen, negative results for DeepSeek, and larger gains for Kimi. Additional validation includes a matched random-retirement replay, human assessments of item quality, and an exploratory code-domain experiment. A scoring-fault diagnostic further illustrates how incorrect labels can produce plausible but misleading evaluation trajectories.

Critical strengths
  • The paper addresses a meaningful limitation of static evaluation. SE-Eval explicitly targets discrimination between adjacent model releases when fixed benchmarks approach the ceiling.

  • The empirical analysis is transparent about both gains and failures. The paper reports negative cross-family results, distinguishes pre-specified analyses from exploratory findings, and acknowledges scoring and baseline confounds.

Critical weaknesses
  • The evaluation does not establish that the selected items provide reliable discrimination beyond the responses used to select them. SE-Eval optimizes adjacent-version disagreement and reports gains on the same model sequence and observed responses. The matched-retirement replay controls candidate supply but retains this dependence. Without held-out releases or independent response samples, the gains may reflect adaptation to observed behavior rather than durable measurement quality, particularly for Kimi’s stochastic evaluations.

  • The experiments do not establish the benefit of iterative evolution over simpler alternatives. Approximately 97% of the reported Qwen gain occurs in the first round, yet the paper lacks a budget- and capacity-matched one-shot generation-and-selection baseline. The original random-refresh baseline is confounded by different pool sizes and gating, while the retrospective replay isolates retirement only. Consequently, the evidence does not clearly justify the proposed closed-loop framework over a simpler benchmark construction procedure.

Questions for authors
  • Were any reported gains evaluated using model responses independent of those used for item selection, or on held-out model releases? If so, please identify those results. This would help determine whether the gains persist beyond the observations optimized by the controller, particularly for Kimi’s stochastic evaluations.

  • Do any completed experiments compare SE-Eval against one-shot generation and selection with matched total generation budget, verification, pool capacity, and evaluation budget?

1 comment

Dear Reviewer TGCY,

Thank you for taking the time to provide detailed feedback. We appreciate your recognition of the importance of the problem and the transparency of our analysis. We have carefully considered your concerns and questions and prepared changes to incorporate into the revised manuscript.


To provide more direct experimental evidence, we plan to:

  1. W1 & Q1: Test whether the gains in discrimination hold for model releases that were not used to construct the item pool.
  2. W2 & Q2: Compare SE-Eval with one-shot generation and selection under the same budget, verification procedure, and item pool capacity.

Thank you again for your review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.