acceptodds
Under review as a conference paper at ICLR 2027

Same Outputs, Different Winners: Evaluator-Dependent Comparisons in Code Benchmarks

Abstract

AI coding research relies on benchmarks to compare code models and coding agents, assess progress, and select research baselines. Yet changing the evaluator can reverse statistically supported leads even when code outputs remain fixed: 60 of the 66 supported reversals we observed had point-estimate advantages of at least 2 percentage points in both opposing directions. To characterize this dependence, we hold tasks, system configurations, and generated outputs fixed and study evidence reinterpretation, criterion strengthening, and specification revision through evaluator contrasts in SWE-bench Verified, EvalPlus, and LiveCodeBench, respectively. Within each study, joint inference over all system pairs and evaluator conditions provides a common evidence standard, separating support lost through an expanded inference family from support lost after evaluator changes. We find that only 38.5%-55.0% of originally supported leads retain support for their original direction across evaluators; among cross-evaluator support losses, roughly two-thirds gain support for the opposite direction. The consequences also depend on where the affected relations lie: a leading group can retain supported leads over all remaining systems while its internal winner remains unresolved, whereas a single winner-critical conflict can block a unique-winner claim. We therefore propose Claim-first interpretation, which begins with a target claim and its evaluation scope, examines the pairwise relations required to support it, and determines what conclusion the available evidence supports. In practice, the framework guides researchers in assessing progress and selecting baselines and shortlists, authors in reporting conclusions and their evaluation conditions, and maintainers in reassessing critical comparisons after evaluator updates. It thus moves benchmark use beyond reading scores and ranks toward evidence-based scientific conclusions that can be reexamined as evaluation conditions change.

Primary area:
datasets and benchmarks
Keywords:
Benchmark Evaluation, Code Generation, Coding Agents, Benchmark Reliability, Evaluator Dependence, Simultaneous Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fDm4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how changes in grading rules affect model comparisons on SWE-bench Verified, EvalPlus, and LiveCodeBench. By regrading identical saved outputs, it shows that many previously clear model leads disappear or reverse, demonstrating that leaderboard conclusions can depend heavily on the grader. The paper therefore advocates “claim-first” reporting: researchers should state the specific conclusion they want to establish and verify the comparisons required for it, rather than drawing conclusions directly from leaderboard rank.

Critical strengths

None

Critical weaknesses
  • The paper is not quite readable.
  • The paper studies something that is trivial - if you change the grading of a benchmark, the eval outcome changes too. I can't see any value provided by this paper.
Questions for authors

None

1 comment
AuthorsOpenReview ↗

Readability and the decisions affected by regrading

Dear Reviewer fDm4,

Thank you so much for taking the time to review our paper and for drawing attention to its presentation and contribution. Even when a critique is brief, we genuinely value any signal that helps us make the paper clearer to readers and more transparent in its motivation. With that spirit, we would like to share our revision on readability and clarify the core idea, novelty, and practical benefit of our framework.

Logic Mainline and Readability Improvements

In the submitted draft, we introduced formal notation before motivating why a researcher needs it. In the revision, we reorganized the logic so that a concrete research decision on page 1 leads naturally into the problem, the notation, and the joint inference:

  • Motivating Decision on Page 1: Suppose a team has two slots to follow up on SWE agents. The same two agents each outperform all ten alternatives under both ways of grading the saved patches. With one slot, the team also needs to know whether either leader outperforms the other: their estimated gaps are 0.30 and 1.04 percentage points under the two evaluators, and both intervals include zero. Thus the benchmark supports keeping both agents on the shortlist while leaving their internal order open.
  • From Paired Judgments to Decision Tables: Figure 2 (p. 3) illustrates how grading a frozen output twice produces paired judgments for system comparisons, and Table 2 (p. 8) places score-based choices side by side with the exact comparisons each decision requires.

We would be very grateful if you could let us know which specific passage, figure, or step would benefit most from further clarification—we are more than happy to refine it further.

Core Idea, Novelty, and Practical Benefit

You are right that changing how a benchmark is graded can change a score. The core idea and novelty of our paper lie not in observing that scores or rankings shift, but in providing a principled way to decide what scientific claims remain supported—and which baselines or comparisons are actually required—when an evaluation changes:

  • Why It Matters Beyond Score Shifts: Consider LiveCodeBench. S01 leads S03 by 16.43 points under the old tests (simultaneous interval [14.08, 18.79]), whereas on the exact same programs under the new tests, S03 leads by 4.99 points ([1.88, 8.10]). A researcher taking the old point leader as the sole baseline would miss the stronger of these two systems under the new tests. In SWE, we can retain the shortlist even as scores change; in LCB, choosing a single reference depends on which tests the progress claim covers.
  • How Claim-First and PaCES Help Researchers: Section 4 shows how to specify a claim, identify its required pairs, and read their intervals under each relevant evaluator. Because the joint calibration in §2.3 covers every system pair—including pairs a reader selects after viewing the leaderboard—it supports post-hoc claim auditing rigorously. Across our three fixed-task analyses, 66 comparisons have simultaneous support for opposite directions, and for 60 of them the observed advantage reaches at least two percentage points in each direction (Table C.5). For a team evaluating a new system against both historical LCB leaders, this yields a concrete design: compare it with S01 and S03 under each snapshot, recognizing that beating the revised leader on the revised tests establishes only one of the four comparisons that broader claim requires.

Thank you again for your time, your feedback, and your consideration of our revised manuscript. We warmly welcome any further comments or suggestions you may have.

Reviewer_T22mOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether changing only the post-generation evaluator, with tasks, systems, and generated outputs held fixed, can alter pairwise comparisons between code-generating systems. It studies three contrasts: revised log parsing and annotations for SWE-bench Verified, Base versus Base+Plus tests for EvalPlus, and two private-test snapshots for LiveCodeBench. Using simultaneous intervals over all system pairs and both evaluators, the authors find that only 38.5%–55.0% of originally supported leads remain supported, and about two-thirds of the lost leads become supported reversals (66 in total). They also propose “claim-first” interpretation, which assesses a system-level claim (e.g., a unique winner or a leading set) through the pairwise comparisons it requires.

Critical strengths

S1: The fixed-output design is useful. Re-evaluating the same programs isolates changes in the evaluator from changes in model generation. This makes the observed pairwise differences easier to interpret.

S2: The statistical treatment is more careful than a comparison of leaderboard positions. The paired analysis, simultaneous intervals, and separation of multiplicity effects from evaluator-dependent changes make the reported reversals more informative than raw rank changes, although the inference is conditional on the fixed task set.

Critical weaknesses

W1: Limited novelty. Rank changes under these same evaluator updates are already reported by EvalPlus and UTBoost, as the paper’s related work acknowledges. The main new element is counting statistically supported reversals.

W2: Evaluator validity is not addressed. The paper treats E0 and E1 as equally valid. If E1 is more correct (e.g., Base+Plus, the fixed SWE-bench parser), reversals are corrections of E0’s errors rather than a reliability problem. Without case-level evidence, the reversal statistics are hard to interpret.

W3: Task-level uncertainty is ignored. The inference treats tasks as fixed (Section 2.3), so nearly all pairs are already supported under E0 (60/66, 26/28, 107/120). The paper’s own Section 3.3 shows reversal retention dropping from 28 to 17 under different task compositions.

W4: Contrast selection and confounding. The selection of the E0/E1 pairs is not described, and each operation is tested by a single contrast on a different benchmark and system set, so operation effects are confounded. Moreover, 359/387 LiveCodeBench tasks only add tests, making “specification revision” largely criterion strengthening.

Questions for authors

Q1: What was the full set of eligible evaluator versions or contrasts considered for each benchmark? What criteria selected the reported E0/E1 pairs, and were those choices made before examining pairwise reversals?

Q2: For LiveCodeBench, can you provide a task-level audit of verdict changes caused by added, removed, and reordered tests? In particular, how were the expected outputs of consequential new tests validated against the original problem statements, and how many supported reversals are attributable to each type of change?

Q3: How do the retention and reversal results change when task-level uncertainty is included in the inference (e.g., resampling tasks jointly with attempts)? How many of the 66 reversals remain supported?

4 comments
AuthorsOpenReview ↗

1/4 · W1 and W2 · Contribution beyond EvalPlus and UTBoost, and when regrading corrects an error

Dear Reviewer T22m,

Thank you for placing our work alongside EvalPlus and UTBoost, and for raising these important questions on what our analysis adds and how revised grading decisions should be interpreted. We deeply appreciate your thoughtful feedback, which helped us sharpen both the Introduction and our illustrative examples.

We agree that EvalPlus and UTBoost established that a grading revision can change rankings. The submitted paper introduced Claim-first in §4. The revised introduction now names PaCES, explains what joint intervals establish, and illustrates that distinction with the two SWE leaders. Figure 2 (p. 3) traces paired judgments into the comparison.

Response to W1: Inferential Claim Auditing vs. Ranking Shifts

What does the analysis add? EvalPlus asks whether stronger tests expose incorrect programs, while UTBoost investigates SWE-bench evaluation faults and how fixing them changes patch judgments. Our question is what a researcher can still claim about two systems after the same outputs are regraded. PaCES grades each saved output under both rules and builds simultaneous confidence intervals for every system pair's gap, determining whether regrading warrants changing an outperformance claim even when the revised evaluator is preferable.

  • SWE-bench Verified Example: S07 ranks above S11 and S10 originally, and below both after regrading. What should a researcher report?

    • S07 minus S11: revised gap is -2.00 pp, simultaneous interval [-3.50, -0.50].
    • S07 minus S10: revised gap is -0.62 pp, interval [-2.12, 0.88] (Table C.4).
    • A paper reporting both original leads should now report S11 ahead of S07 and withdraw the directional claim about S07 and S10. Only the first pair enters the supported-reversal count, though the second also requires revising the published conclusion.
  • Connecting Pairwise Results to Decisions: Claim-first connects pairwise results to reader decisions, as shown with the two SWE leaders (p. 1): all 20 comparisons with ten outsiders support selecting the pair, while their internal order remains open. Across all three benchmarks, 191 of 193 initially supported leads survive joint-family expansion; regrading preserves 91 leads, supports the opposite system in 66, and leaves 34 unsupported in either direction (Figure 3).

Response to W2: Validating Revisions and Task-Level Mechanics

When E1 corrects a demonstrable error, we use the corrected verdict. A rank change alone cannot tell whether grading improved: if a stronger test catches a genuine error, a prior success should be revised, answering which verdict applies to that program. Deciding whether to revise a published comparison then requires checking how often each system gains or loses passes and whether the gap supports a direction. To check the grading decision itself, we examined a sample log line and constructed programs with verifiable answers:

  • SWE (Identifying a Test in a Log): On Reading fixtures ... test_empty_input (tests.reader.ReaderTest) ... ok, both parser branches detect a pass. The native branch includes "Reading fixtures ..." in the test key, whereas the revised branch returns the bare test name matching the required-test list—a specific parser correction. The historical SWE comparison also changes annotations for 271/500 tasks; attributing a patch's verdict to either component requires its log and applicable test lists.
  • EvalPlus (Missed Boundary): HumanEval/0 requires a distance strictly below the threshold. A candidate using <= passes all seven checked Base inputs. For [1,2,3,4,5] at threshold 1, it returns True although the smallest distance is 1; the Plus input correctly flips it to fail.
  • Concealing an Error via Removed Checks: In our controlled XOR example, four of ten B programs reject [1,1,1,1], while [0,1,0,1] witnesses the true answer True. Shrinking the suite from 100 tests to 12 raises B from 6/10 to 10/10 (A stays 10/10); restoring that single check returns B to 6/10.
  • Reversing a Task Lead via Added Checks: For the product task, incrementing the zero in [0,2,3] yields 6, while incrementing the largest element yields 0. Six A programs and one B program make this error. Adding this input to the eight old checks drops A from 10/10 to 4/10 and B from 9/10 to 8/10. Three other added inputs independently catch this mistake; removing all four restores original counts.

Appendix B.6 details the HumanEval/0 and XOR examples. For historical LCB programs, the snapshot audit supports separate quantitative attribution to tasks gaining tests, detailed in Q2 below.

AuthorsOpenReview ↗

2/4 · W3 and Q3 · Task-sampling uncertainty for the 66 original reversals

Task-Sampling Sensitivity Under Hoeffding–Bonferroni Bounds

Under 95% Hoeffding–Bonferroni bounds with tasks as the sampling units, 0 of the 66 original reversals retain support on both sides. All 20 comparisons placing the two SWE leaders ahead of the other ten systems retain support under both evaluators. Your question prompted us to recalculate the intervals for every system pair during rebuttal using the existing scores, with the same tasks, outputs, and paired judgments—no model or evaluator rerun was needed.

The original analysis asks about expected performance over generations on a fixed set of tasks; here we ask about the mean difference over tasks from the stated benchmark population. On each task, the difference between two systems’ pass rates over ten attempts lies in [-1, 1]. Taking whole tasks as independent sampling units allows the systems, evaluators, and attempts on the same task to be dependent. The target retains the HumanEval+/MBPP+ composition in EvalPlus and covers the 387 changed tasks in LCB.

We recovered the exact aggregate pass counts from the score tables. All 72 rounded scores identify unique counts because one pass changes the percentage by more than the tables’ rounding unit. We then used a two-sided Hoeffding bound with a Bonferroni correction over every pair under both evaluators, separately within each benchmark. For T tasks and G pair-by-evaluator comparisons, the 95% simultaneous half-width in percentage points is 1002log⁡(2G/0.05)/T100\sqrt{2\log(2G/0.05)/T}, which uses the range of task differences without estimating their covariance:

  • SWE (500 tasks, 132 comparisons): Half-width 18.52 pp; largest E1 reverse margin 9.28 pp (Table C.5); retained 0/18.
  • EvalPlus (542 tasks, 56 comparisons): Half-width 16.87 pp; largest E1 reverse margin 7.31 pp; retained 0/10.
  • LCB (387 tasks, 240 comparisons): Half-width 21.77 pp; largest E1 reverse margin 9.17 pp; retained 0/38.

Even the largest reverse margin in each setting falls inside the half-width. Matching Figure 4’s original reversals to these intervals gives a further distinction: SWE S07–S11 and S07–S03 retain support for the original E0 lead and lose support for the reverse E1 lead, while the other 64 pairs lose directional support on both sides (with their observed gaps retaining their original values).

Lead Retention Under the Task-Sampling Bounds

Using the paper’s original retention definition (denominator = leads supported under E0-only calibration for that analysis):

  • SWE: Of 66 pairs, 25 have a supported E0 lead under E0-only calibration (previously 60). All 25 keep E0 support in the joint family, and 20 also keep the same direction under E1 (20/25 = 80.0% lead retention, rising from 55.0% even as leads supported under both evaluators move from 33 to 20).
  • EvalPlus: Of 28 pairs, none has a supported E0 lead under E0-only calibration (previously 26), so the retention fraction has a zero denominator and is undefined.
  • LCB: Of 120 pairs, 37 have a supported E0 lead under E0-only calibration (previously 107). Of these, 35 keep E0 support in the joint family and 20 also keep the same direction under E1 (20/37 = 54.1% lead retention).

Surviving Conclusions and Comparison with Subset-Mix Results

Which conclusions survive? In SWE, the two leaders still beat every outsider, and their mutual intervals include zero. All 28 EvalPlus comparisons are unresolved under both evaluators: even the widest observed gap among the eight configurations is only 9.96 pp on Base and 8.51 pp on Base AND Plus, below the 16.87 pp half-width in each case. For LCB S01–S03, the original simultaneous intervals are [14.08, 18.79] and [-8.10, -1.88] pp, whereas the task-sampling bounds are [-5.33, 38.20] and [-26.76, 16.78] pp; we therefore retain the reversal as a finding about the fixed benchmark tasks and leave its direction open for the broader task population.

The 28-to-17 result you cite is the submitted §3.3 median number of LCB reversals retained when the task mix changes at fixed subset size. Revised Figure 5(c) reports 25 to 19 out of the original 38 for the corresponding comparison, where each subset has 200 tasks and ten attempts per system and task. The new calculation asks how much directional evidence remains under a task-distribution target, changing both the target and the interval procedure. The 0/66 result is specific to this distribution-free bound; a covariance-based task-level Gaussian maxT analysis could give a different count and requires per-task system scores. Table 1 in §3.3 (p. 8) reports all three half-widths and retention counts, and Appendix D.4 (p. 30) gives the derivation and results across all 214 system pairs.

AuthorsOpenReview ↗

3/4 · W4 and Q1 · Candidate evaluators, selection timing, and scope

Candidate Benchmark Contrasts and Inclusion/Exclusion Rationale

We fixed the three evaluator contrasts and their versions before examining which system comparisons reversed. We considered 14 candidate entries: three supplied the final contrasts, two (HumanEval and MBPP) are represented within EvalPlus, and nine were set aside. We looked for two identifiable grading rules that could be run on the same task and saved output, with enough information to trace both judgments. The submission described the three adopted contrasts; revised §2.2 specifies when we fixed them, and the new Appendix B.5 records the alternatives and their disposition:

  • SWE-bench Verified (Included): Native versus UTBoost parser and annotations; both can read the same patch execution log.
  • EvalPlus (Included): Base versus Base AND Plus; the same extracted program faces a nested success requirement.
  • LiveCodeBench (Included): The same archived programs and common checker can be used with two historical private-test snapshots.
  • APPS: Original versus augmented boundary/random tests; the second suite would have required us to construct and validate its oracle.
  • DS-1000: Functional success versus functional plus API-use or other form constraints; this comparison also requires stable dependency settings and a task-specific account of which constraints belong to the specification.
  • BigCodeBench: Complete versus Instruct changes the prompt before generation; a fixed-output test extension would require a separately constructed contrast.
  • HumanEval: Original versus expanded tests; incorporated as the 164 HumanEval+ tasks in EvalPlus.
  • MBPP: Base versus Base AND MBPP+; incorporated as the 378 aligned MBPP+ tasks in EvalPlus.
  • CodeContests: Public and private tests versus those tests plus generated tests; matching model outputs, language environments and execution records would require further alignment.
  • MultiPL-E: Across languages, the program and runtime change together; a contrast within one language would require separate grading specifications.
  • ClassEval: Method tests versus additional state and interaction tests; we would also have to fix the object lifetime, fixtures and order of calls.
  • RepoBench: Retrieval, completion and combined settings have different output types and success criteria.
  • Project CodeNet: Historical judge label versus a rerun; recovering an old verdict requires the old tests and execution policy as well as the saved submission.
  • SWE-bench Lite / Full: Comparing subset scores changes task membership; we chose a common set of Verified tasks for both grading conditions.

Artifacts and Scope of the Three Adopted Contrasts

For the three adopted contrasts, Appendix B.2–B.4 identifies the relevant artifacts:

  • SWE uses the native harness at 4c21a4831d80 and the UTBoost revisions at 4f32782517f2.
  • EvalPlus evaluates the same cleaned program on Base and on Base AND Plus.
  • LCB uses snapshots d9790747f3e2 and 2c35eba5dd5f with checker 28fef95ea8c9.

The scope of the comparison matters here. Within SWE, for example, the same tasks, configurations, patches and logs enter both conditions, allowing us to measure the effect of the specified joint parser-and-annotation update on those comparisons. Across the three benchmarks, the tasks and configurations also differ, and each operation is represented by one benchmark and its own system cohort. The 18, 10 and 38 supported reversals therefore describe three specific contrasts, so comparing those counts cannot establish which operation would cause more reversals on a common population. In particular, adding tests is part of both the EvalPlus and LCB contrasts, and the LCB audit counts 359 tasks with additions while identifying changes to retained test order within that group.

AuthorsOpenReview ↗

4/4 · Q2 · LiveCodeBench additions, deletions, and attribution

LiveCodeBench Snapshot Audit and Task-Group Attribution

359 of the 387 changed LiveCodeBench tasks (92.8%) gained private tests, so your description of the update as predominantly additive is accurate. We compared both snapshots on the same 400 aligned tasks, counting repeated tests with their original multiplicities; public tests and task statements agree in the snapshot audit:

  • Strict additions to the test multiset: 359 tasks (primary analysis).
  • Strict removals from the test multiset: 20 tasks (primary analysis).
  • Same test multiset, different order: 8 tasks (primary analysis).
  • Identical test sequence: 13 tasks (unchanged-task controls).

This explains why we call the LCB contrast a snapshot revision. EvalPlus explicitly requires a program to pass Base and Plus, so an existing Base failure remains a failure, whereas the LCB checker uses the private tests present in the specified snapshot—meaning a later snapshot can remove a check that previously rejected a program.

The order of retained tests also changes in 264 of the 359 addition tasks and 17 of the 20 removal tasks. For example, Codeforces task 1873_B goes from eight to twelve private test entries, with four additions and a change in the order of retained entries. The eight tasks in the order-only row have the exact same test multiset on both sides, and the 13 unchanged controls give 2,080 matched program judgments (1,747 pass–pass and 333 fail–fail records, with zero disagreements).

Quantitative Bounds on S01–S03 and Category-Level Attribution

  • Lower bound on the addition group for S01–S03: The addition group must account for at least 6.95 pp of the observed movement toward S03. Table C.3 gives 3,320 vs. 2,684 passes under E0 and 2,522 vs. 2,715 under E1 (using 3,870 outputs per system). The S01–S03 difference shifts from +636 to -193, a movement of 829 passes. Since each system contributes between 0 and 10 passes per task, the difference can move by at most 20 per task; even if all 28 removal and order-only tasks shifted maximally toward S03, they could account for at most 560 of the 829 passes. Thus at least 269 passes (269 / 3,870 × 100 = 6.95 pp) must come from the addition group (which could not overturn the original 636-pass lead on its own). Because additions sometimes coincide with reordering, this bound applies to the observed group of tasks.
  • Input-level validity checks: For historical task 1873_B, we verified the changes in test entries and ordering, though we do not have program-level attribution linking a specific added input there to one of the 38 system reversals. Input-level validity checks are provided by our controlled examples in W2: for XOR, [0,1,0,1] directly witnesses the true answer True, and for the product example, enumerating allowed increment positions verifies all four expected answers.
  • Assigning the 38 supported reversals across categories: The 38 count reflects the joint update. Because a single pair can gain or lose passes across multiple task categories and reversals are evaluated only after combining contributions across tasks, neither task counts nor published totals alone determine how many pairs would retain opposite simultaneous support under a single-category update—which would require within-task paired judgments and a new joint calculation. The 6.95 pp bound addresses the narrower attribution question resolved by existing records for S01–S03 (§3.2, p. 6; count calculation in Appendix B.4.1, p. 24; full-update gaps of +16.43 [14.08, 18.79] pp and -4.99 [-8.10, -1.88] pp in Table C.4).

Summary of Revision Additions and Closing

The submitted paper counted the changed tasks; the revision adds the historical 1873_B example in §3.2 and the lower-bound calculation in Appendix B.4.1, while Appendix B.4 reports the 13 unchanged controls separately.

Thank you so much for pressing us to quantify how much of the change in the leaders’ gap can be traced to tasks that gained tests, and for all of your careful, constructive feedback throughout this review process. We remain deeply grateful for your time and would warmly welcome any further questions or suggestions you may have.

Reviewer_Fra6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how changes to code benchmark evaluators affect system comparisons while keeping generated outputs fixed. Across SWE-bench Verified, EvalPlus, and LiveCodeBench, the authors use joint statistical inference to show that evaluator changes can reverse statistically supported performance advantages. They propose a “Claim-first” approach that bases conclusions about winners and leading groups on the specific pairwise evidence needed to support them.

Critical strengths
  • Novel and important problem. I find that the idea of isolating the impact of evaluator changes is novel and important. The proposed evaluation framework, together with the shift from leaderboard rankings to explicitly supported claims, offers a valuable conceptual contribution to interpreting code benchmark results.
  • Well-motivated and carefully designed methodology. The fixed-output design helps isolate evaluator effects, while joint statistical inference accounts for dependencies across comparisons and separates evaluator-induced changes from support losses caused by expanding the comparison family. These methodological choices provide a clear basis for interpreting the reported reversals.
Critical weaknesses
  • Limited depth of empirical analysis. Given the paper’s focus on evaluation and its framing as an empirical study, I would expect deeper insights into why the observed reversals occur. Beyond documenting reversals and their implications for system comparisons, the analysis could investigate which specific evaluator changes, task characteristics, or system behaviors drive them. Such findings would make the study more informative and actionable.
  • Limited experimental detail and configuration-specific insights. The main text provides insufficient explanation of the selected agents, LLMs, and their configurations, while the analysis does not examine how these choices relate to the observed results. I would expect more concrete findings, such as whether particular agent types, model families, or reasoning settings exhibit distinct patterns of sensitivity to evaluator changes. Such analysis would help explain the reported reversals and make the current high-level findings more informative.
  • Limited validation of the Claim-first approach. The paper provides a reasonable conceptual framework for connecting claims to pairwise evidence, but its practical benefits remain insufficiently demonstrated. I would expect concrete case studies showing how this approach changes baseline selection, shortlist construction, or progress assessment compared with conventional benchmark reporting. Such evidence would help establish whether Claim-first meaningfully improves research decisions beyond encouraging more careful interpretation of results.
Questions for authors
  1. What are the experimental settings of this paper? In the main experiment, do you evaluate code agents or LLMs? Specifically, which ones are used, and why? And how might these selections affect the reported reversal patterns?
  2. Could you provide a worked example of baseline selection or shortlist construction that shows how Claim-first changes the resulting decision and what practical benefit it provides?
3 comments
AuthorsOpenReview ↗

1/3 · W1 · Which changes drive a supported reversal?

Dear Reviewer Fra6,

Thank you so much for raising this question and giving us the opportunity to walk through what drives these reversals. We greatly appreciate your careful reading and thoughtful engagement with our results.

EvalPlus: Asymmetric Disqualification Under the Nested Pass Rule

In EvalPlus, the additional tests disqualify a larger share of MiniMax-M3’s programs than of Kimi-K3’s. That difference is large enough to overturn MiniMax-M3’s original lead. Each candidate had already been generated, and the revised evaluator requires a pass on both Base and Plus.

  • MiniMax-M3, S03 moved from 89.21% to 77.38% (-11.83 pp).
  • Kimi-K3, S06 moved from 86.13% to 81.79% (-4.34 pp).

The nested pass rule lets us count exactly which verdict transition accounts for the shift. From Table C.2, MiniMax-M3 goes from 4,835 to 4,194 passes among its 5,420 programs, so 641 Base passes fail Plus. For Kimi-K3 the corresponding counts are 4,668 and 4,433, a loss of 235. The difference of 406 exceeds MiniMax-M3’s original advantage of 167 passes and leaves Kimi-K3 ahead by 239. The S03-S06 gap and its 95% simultaneous interval change from +3.08 [2.03, 4.13] pp to -4.41 [-5.95, -2.87] pp (Table C.4).

LiveCodeBench (LCB): Bounds on Changed Task Groups

For the two LCB point leaders, we can also locate part of the change in the historical data. Across the two snapshots, the S01-S03 gap moves by 829 passes toward S03. The 28 removal or order-only tasks could account for at most 560 of those passes. Even at that extreme they could not erase S01’s original lead of 636 passes. The 359 tasks that gained tests must account for at least 269 passes (6.95 percentage points) of the movement toward S03 (§3.2 and Appendix B.4.1). Some of those tasks also reordered retained tests, so this bound applies to the group as a whole.

SWE: Opposite-Direction Shifts on Reinterpreted Logs

SWE gives a different mechanism. On the same r2e-gym scaffold at temperature 0, Devstral-2 and Qwen3-32B move in opposite directions when the parser and required-test annotations reinterpret their saved logs.

  • Devstral-2, r2e, T=0, S07 moved from 38.16% to 23.06% (-15.10 pp).
  • Qwen3-32B, r2e, T=0, S11 gained +5.84 pp.

For S07 minus S11, the gap changes from +18.94 pp [17.31, 20.57] to -2.00 pp [-3.50, -0.50] (Table C.4). The 20.94 pp shift is the difference between their responses to the combined revision. Both conditions read the same patch execution, so no new model generation is needed to produce the change. Assigning separate shares to parser and annotation changes would require both intermediate conditions on those logs.

Task Composition and Subset Survival Frequencies

Task composition also changes which comparisons remain supported. Appendix D holds the number of tasks and attempts fixed while varying the task mix.

  • In EvalPlus, all five conflicts within the original top five survive jointly in 87 of 200 subsets with 30 HumanEval+ and 270 MBPP+ tasks, in 56 with a 90/210 split, and in 10 with a 150/150 split.
  • In LCB, S01-S03 survives in 164/200, 150/200, and 129/200 subsets as the addition/removal mix changes from 195/5 to 190/10 to 185/15.

Each LCB subset has 200 tasks and ten attempts per system and task. These frequencies show how often a conclusion survives a change in task mix at the same sample size.

Single-Program Controlled Examples and Scope Boundaries

What changes for a single program? Appendix B.6 (p. 25) gives two controlled examples using constructed candidates.

  • In HumanEval/0, a candidate treats a distance strictly below vs. equal to the threshold differently: it passes all seven Base inputs, and on the additional input [1,2,3,4,5] with threshold 1, it returns True instead of the required False.
  • The XOR example shows how removing a valid check can hide an error: four of ten programs in one group reject a valid array of four ones, and removing the checks that expose this error raises the group’s passing count from six to ten (restoring just the four-ones check brings it back to six).

In these controlled examples we can change one input and observe the resulting verdict. The submitted paper reported the system reversals. Revised §3.2 traces the LCB leaders’ gap to the changed task groups, and Appendix B.6 gives the inputs for the two controlled checks. Linking a particular historical program’s error to one of those reversals would require matching its identity and verdicts to that input.

AuthorsOpenReview ↗

2/3 · W2 and Q1 · Systems, generation settings, and three configuration comparisons

Evaluation Cohorts and Setup

We evaluated repository-level agents in SWE, direct model requests in EvalPlus, and historical program submissions from 16 source configurations in LiveCodeBench. In each case a system is a particular configuration. For each task we have ten outputs from that configuration, and we grade each saved output twice. This is why a scaffold, temperature or reasoning setting belongs beside the model name when interpreting the results.

  • SWE-bench Verified: 12 configurations, 500 tasks. Patches from the ASSERT-KTH/agentic-evals-artifacts release accompanying On Randomness. Original and UTBoost-revised parsers and annotations read the same replay log.
  • EvalPlus: 8 request configurations, 164 HumanEval+ and 378 MBPP+ tasks. Python programs extracted and sanitized once. E0 requires Base success and E1 requires both Base and Plus success.
  • LiveCodeBench: 16 historical configurations, 400 aligned tasks. Ten programs per configuration and question, matched by source sample index. We keep the checker fixed across two private-test snapshots. The main comparison covers 387 changed tasks. The other 13 are unchanged controls.

Rationale for Selecting These Cohorts

Why these cohorts? The SWE release lets us compare scaffolds at a fixed model and temperature. For each of Devstral-2, DeepSWE-preview and Qwen3-32B we have both nano-agent and r2e-gym. The two temperatures are 0/0.2, 0/1.0 and 0/0.6, respectively.

EvalPlus uses a fresh context and the same prompt template for each direct request, with one candidate and no tools. Appendix B.3 lists the parameters we set and identifies those left at the service defaults for DeepSeek V4 Pro, MiniMax-M3, Qwen3.8-Max, GPT-5.6-Sol, Claude Opus 5, Gemini 3.7 Flash, GLM-5.2 and Kimi-K3.

LCB supplies already-generated code across DeepSeek, GPT, Claude, Gemini, Mistral and Codestral configurations. Its inclusion criterion is that ten historical programs can be aligned on the same question across configurations. The IDs in a comparison retain their meaning under both evaluators and are local to the benchmark.

How the Evaluator Update Affected Individual Configurations

How did the evaluator update affect individual configurations? Three comparisons in the revised Tables C.1–C.4 illustrate different outcomes:

  • Scaffold: At temperature 0, Devstral-2/nano-agent (S01) goes from 63.80% to 58.32%. Devstral-2/r2e-gym (S07) goes from 38.16% to 23.06%. The nano-agent advantage widens from 25.64 to 35.26 pp. Both simultaneous intervals support it.
  • Temperature: For DeepSWE-preview on nano-agent, the temperature-0 configuration (S03) rises from 19.20% to 25.04%. The temperature-1 configuration (S04) falls from 31.40% to 17.98%. Under the original rules, the temperature-1 configuration leads by 12.20 pp. Under the revised rules, the temperature-0 configuration leads by 7.06 pp on the same saved patch collections. These are point differences for this model and scaffold.
  • Model request: MiniMax-M3 with adaptive thinking (display S03) and Kimi-K3 with maximum effort (display S06) both lose successes on Plus. The former loses enough to reverse a supported lead of 3.08 pp into a supported deficit of 4.41 pp. Model identity and reasoning setting both differ in this comparison. The sensitivity we observe belongs to these two request configurations. A within-model comparison with varied reasoning effort would be needed to attribute the difference to effort alone.

Manuscript Organization and Cross-References

The submitted paper described the output sources and EvalPlus requests. The revision brings the three cohorts together in §2.2. Appendix B.2 maps each SWE configuration to its display ID, and Table C.2 gives the EvalPlus request ID beside each pair of scores. For LiveCodeBench, Appendix B.4 lists the 16 historical source configurations and the sample-matching rule. Table C.3 reports their scores under the benchmark-local anonymous IDs used in Figure 4.

AuthorsOpenReview ↗

3/3 · W3 and Q2 · Choosing a shortlist and a baseline

Concrete Research Decisions: Shortlists and Baselines

For a SWE shortlist we would retain S01 and S02. For an LCB study claiming progress under both evaluators we would use S01 and S03 as references. These are two decisions a reader could make from the submitted §3.2 results.

  • Choosing a shortlist (SWE): Suppose we can follow up on a small number of SWE configurations. Devstral-2/nano-agent at temperature 0 (S01) has the highest point score under both evaluators (63.80% under E0 and 58.32% under E1). Changing only the recorded temperature to 0.2 gives S02, the runner-up (63.50% under E0 and 57.28% under E1). The estimated S01-S02 gaps are +0.30 [-1.32, 1.92] pp and +1.04 [-0.64, 2.72] pp—both intervals cross zero. Yet each configuration beats every outsider: across the 20 member-outsider relations, the smallest lower endpoints are 23.70 pp under E0 and 30.68 pp under E1 (Appendix C.2). Thus all twenty required external comparisons support retaining the two configurations. Choosing the higher point estimate alone would discard S02 without establishing that it is worse. If the next experiment must pick one, it can concentrate on their mutual comparison or measure a decision-relevant quantity such as latency, as the data leave their relative performance open.

  • Selecting baselines for a progress claim (LCB): In LCB, S01 is the E0 point leader at 85.79%, while S03 is the E1 point leader at 70.16%. Their direct fixed-task comparison supports S01 by +16.43 [14.08, 18.79] pp under E0 and S03 by +4.99 [1.88, 8.10] pp under E1. Using only S01 would leave a study claiming progress under E1 without a comparison against the stronger of these two E1 references.

Mapping Scientific Claims to Required Comparisons

Let M denote a prospective new system. Specifying the claim determines the exact evidentiary checks required:

  • Progress over the selected E0-leading reference: Compare M with S01 under E0.
  • Progress over the selected E1-leading reference: Compare M with S03 under E1.
  • Superiority to both selected baselines under both evaluators: Compare M with each baseline under both E0 and E1.

For the last claim, we would grade M’s frozen programs under both snapshots, include M in the joint calibration, and check that the lower endpoints of all four intervals for M minus its reference exceed zero. A study reporting progress over S03 on the revised tests needs only the corresponding supported comparison. A top-two rule in SWE or a union of point leaders in LCB could yield the same candidate sets in these examples. With two SWE slots the score rule gives a defensible shortlist; with one slot that rule still returns a name, although the S01-S02 intervals leave the choice open. Specifying the claim tells the researcher exactly where the evidence for the score-based recommendation runs out.

Generalizability Across Task Draws (§3.3)

The task-sampling analysis in §3.3 asks whether the same advice extends to another draw of tasks. Under its conservative bound, all 20 SWE comparisons with outsiders still hold, while the S01-S03 intervals on LiveCodeBench span zero under each snapshot. We therefore report the LCB reversal for the fixed benchmark tasks; for the broader task population, both systems remain relevant references.

Where to Find the Decision in the Revision

  • Introduction (p. 1): The submitted §4 introduced Claim-first; the revised introduction now opens the comparison directly with the SWE shortlist.
  • Table 2 (p. 8): The first two rows start from the same scores and change just one thing—whether the researcher has two slots or one.
  • Section 4.2 (p. 9): Works through those decisions and shows which additional comparisons a new system would need against the two LCB references.

Thank you so much for asking us to carry the examples through to the decision itself, and we warmly welcome any further questions or suggestions you may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.