acceptodds
Under review as a conference paper at ICLR 2027

Answer-Key Drift: Which Benchmark Conclusions Survive a Revised Key

Abstract

When a benchmark is revised, maintainers must decide which published model comparisons still hold. We study this on fixed model outputs, separating changes to answers, scored items, scoring rules and reference actions across twelve archived revision series, a text-to-SQL release and a 2026 tool-use benchmark revision. Headline results can stay put while specific conclusions move. On fixed arithmetic responses from 60 models, one answer-and-item update reverses 10, 9, 10 and 8 of 1,770 comparisons under four scoring rules, and 1,748 comparisons reverse under none of them; yet the four rules yield three distinct reversal sets, with no pair common to all four. On WorkBench, with scorer, tools and state fixed, updated reference actions for 9 of 679 tasks leave the champion and the top-five set unchanged but rewrite 179 of the 216 affected success judgments and reverse 4 of 276 comparisons. The revision's footprint also says what can be reused: a budget certificate that uses only the number of changed tasks certifies 252 of the 276 comparisons from the old scores alone, so all four reversals lie among the other 24. Replacing text-to-SQL references reverses at least six of 120 comparisons under set-equality scoring for every completion of failed evaluations. Across the archived series we express displacement in item-sampling units, locate where edits land, and compare budget rules with same-information baselines.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, answer-key revision, label errors, dataset curation, scoring rules, robustness of model rankings, certified comparisons, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pcAMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an approach to decide how conlcusions from a benchmark change or loose validity when the benchmark is updated. However, it is written quite badly and I cannot understand the exact problem statement and proposed solution.

Critical strengths

I cannot find any critical strength in this paper. The contribution seems marginal.

Critical weaknesses

The paper is not readable despite considerable effort. It does not formulate the problem clearly. The paper would thus need a full revision in order to be eligible for ICLR or similar top tier conferences.

Questions for authors

N/A

2 comments

Thanks for the comments.

You could not identify the problem or the proposed solution. We rewrote the introduction around one decision: after a benchmark is revised, can its maintainer keep the published top five, or the winner, without rescoring every stored answer?

The inputs are the old scores, the unchanged model outputs and a restriction on what the update may change. Compare the old lead with the most it could lose under that restriction. If the lead is larger, its sign cannot change. Otherwise the rule leaves the comparison open; it does not predict a reversal. Section 2 gives a five-question example.

On WorkBench, nine tasks have revised reference actions; the scorer, tools, initial state and unaffected judgments stay fixed. Each changed binary task could cost the old leader at most two points relative to a rival, hence at most 18 in total. The fifth model led the sixth by 19, so no model outside the top five could enter. The winner led by only 7. It did survive, but confirming that required rescoring.

The contribution is empirical, not a new algorithm. On the same saved arithmetic answers, changing extraction reversed 126 of the 1,634 comparisons above the original margin screen; correcting answers, removing items or doing both reversed none. The winner stayed put while the top-three and top-five sets changed. We also measured the work behind a reuse decision: with the changed WorkBench tasks identified and the same caches, resolving all pair relations took 92 action-list executions versus 132 executions when scoring every affected model–task cell. Generic intervals with cached version differences gave the same plan (Section 5, Table 4).

Dear authors, thank you very much for the rebuttal. I do appreciate the effort. After reading the other reviews, I am convinced though that my judgement of the paper will not change and I unfortunately do not have the capacity to give the paper another full read (I would do this if it was borderline or going to be accepted).

Reviewer_HWQJOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper studies which model comparisons remain valid after benchmark revisions while model outputs remain fixed. It measures comparison displacement in item-sampling units and proposes completed-revision checks and pre-revision budget certificates. Results show that headline rankings can remain stable while specific pairwise comparisons reverse.

Critical strengths
  • The paper raises a practical problem related to benchmark and determines which published model comparisons remain valid after a revision.
  • The paper proposes completed-version checks and pre-revision budget certificates and evaluate them from twelve archived revision series, a text-to-SQL release, and WorkBench.
Critical weaknesses
  • Most evidence are from retrospective replay of revisions, so the changed items and edit counts were known and we don't know how prospective performance on unseen revisions would look like.
  • The pooled results are specific to the archived revisions and may not generalize to other benchmarks.
Questions for authors
  • How should maintainers choose a revision budget before the update is known?
  • Do you expect the results to generalize to other benchmark revisions?
1 comment

Thanks for the comments.

Your first concern was that the evidence was retrospective. On September 4 we froze 1,300 TIME comparisons, their budgets and the decision rule. The November 16 current-vintage checkpoint was scored once on November 19; an independent recomputation agreed. Only one revision supplied evidence published after registration: the September 30 Vehicle Supply vintage. None of the 209 comparisons certified at the 90th-percentile budget reversed; six of the 116 uncertified ones did. The registered cell-level hypothesis failed: 38 of 300 cells (12.7%) moved beyond their budgets, above the 10% ceiling. Post hoc, keeping the same number of largest old margins also kept no reversal. Job Claims and US Labor had no revised registered cells. Housing repeated a change published before registration and remains descriptive. The separate November 2 fixed-date check has no result (Section 5, Appendix H).

How to choose a budget before the update? Start with the decision you want to preserve. Old scores tell you how much change it can tolerate, not what change will occur. For binary success with fixed execution, one changed task can cost the old leader at most two points against a rival. The fifth WorkBench model led the sixth by 19, so nine edits can be tolerated; the winner led by seven, so this argument protects it only up to three. A maintainer must enforce the chosen cap or use the announced task identities before rescoring. For an uncontrolled external source we would report sensitivity, not promise coverage. In a separate retrospective TIME replay, a historical 90th-percentile budget marked 1,068 of 1,300 comparisons safe, yet seven reversed; the historical maximum missed the same seven.

What transfers? The conditional rule applies when the scoring assumptions hold and the edits really stay within the stated bounds. The observed frequencies and costs need not transfer. The arithmetic extraction effect did not recur in a separate MathArena parser repair, which changed none of 1,860 scores (Appendix Y.3). The checkpoint adds one revision on one task, not an estimate of coverage for unseen revisions.

Reviewer_naZAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate how model comparisons change when benchmarks are updated. They examine several benchmarks to determine which model comparisons are reversed following a benchmark update. Unfortunately, despite conducting a large number of experiments, the authors do not identify any noteworthy patterns beyond the observation that some comparisons change while others remain unchanged. From this work it is hard to conclude anything more than the fact that comparisons that remain unchanged can be retained, while the others should be recomputed.

Critical strengths

None.

Critical weaknesses
  1. The manuscript currently lacks sufficient coherence for readers to understand what exactly was done in the study. The text lacks a clear structure, and definitions, numbers, and tables appear throughout the paper in a seemingly arbitrary order. Moreover, the paper does not present any meaningful observations, conclusions, suggestions for applying the findings, or directions for further research. Instead, it contains an abundance of new definitions and literal repetitions of experimental results, often repeated multiple times throughout the paper. For example, the numbers "10, 9, 10 and 8 of 1,770 comparisons under four scoring rules" appear repeatedly, while the corresponding scoring rules are not explicitly identified in most instances where these numbers are mentioned.

  2. Even after careful reading and attempts to understand what the authors actually did, the motivation behind the work remains unclear due to the poor presentation. The authors themselves do not establish a clear objective or research problem that they aim to address. Furthermore, despite conducting numerous experiments, they do not appear to consider it necessary to draw meaningful conclusions from the work they have performed.

Questions for authors
  1. Could the authors briefly explain what they intended to investigate through these experiments and whether they were able to obtain any significant findings that could be further developed and investigated in future work?
1 comment

Thanks for the comments.

You asked what we wanted to learn and what the experiments found. We wanted to separate what a maintainer learns by rescoring a revised benchmark from what the old scores and a stated edit limit already guarantee.

Our clearest finding is about shortlists. Starting with the same saved GSM8K responses, original key and items, switching from strict to flexible answer extraction reversed 126 of the 1,634 comparisons whose old margins exceeded 1.645 item-sampling standard deviations. Correcting answers, removing items or doing both reversed none of those same comparisons. One model scored 0/1,319 under marker-based extraction but entered the top three and the top five under flexible extraction; the winner did not change. Three models account for much of the effect: at least one participates in 106 of the 126 reversals. These are not 126 independent events.

The four counts you flagged answer a different question: if the scoring rule stays fixed, how many of all 1,770 comparisons reverse after answers and items are revised together? It was 10 with public strict extraction (first number after the required marker), 9 with public flexible extraction (last number anywhere), 10 with a candidate reconstruction of the historical rule (first marked match with archive-compatible normalization), and 8 with our common parser (last marked answer line). The revision defines each rule beside these counts and removes the sequence from the abstract.

The practical lesson is to name the decision before deciding what to rescore. On WorkBench, a nine-task cap with fixed execution protects the old top-five set without rescoring, but not the winner (Section 5). The next research question is whether extraction changes shortlists on independently maintained benchmarks. Counting more pairs from this GSM8K revision would not answer it.

Comments on the paper

1 comment

Thank you for all the reviews. We rewrote the main text around a maintainer's decision: after a benchmark changes, can the same models still be recommended without rescoring their stored answers? The title is unchanged and no model was rerun.

Figure 1 illustrates the distinction on WorkBench. Under a nine-task edit cap with fixed execution, the old top-five set is fixed (lead 19, loss bounded by 18); first place still needs checking (lead 7). Section 2 works through the rule. In the arithmetic replay, changing how numbers are extracted reverses 126 of the same 1,634 screened comparisons; correcting answers, removing items or doing both reverses none. The shortlist changes even though the winner does not.

Section 5 distinguishes an edit cap a maintainer can enforce from a budget inferred from past changes. Appendix H reports the November checkpoint: only one task, Vehicle Supply, had a revision published after registration. None of 209 certified comparisons reversed; six of 116 uncertified ones did. The cell-level hypothesis failed (38/300 above budget; ceiling 10%), and a post hoc screen retaining as many large margins also kept no reversal. One event does not establish a success rate for future revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.