acceptodds
Under review as a conference paper at ICLR 2027

The Convention Is the Grader's: How Model Proof Graders Price Silently Repaired Errors

Abstract

The grading rubric bands a small repairable error at 5–6 and says nothing about a false line the proof itself corrects; what it costs is the grader's judgement. We measure that price and propose no grader. Into 24 IMO 2026 proofs, model-judged closed, not human-verified, we plant three single-hunk edits for 96 texts — fatal, harmless (nothing later uses it), or a deleted argument. Falsity is machine-checked; class is a recorded label (8 of 72 disputed). A panel of 5 graders scores the texts, and every decision rule is read under two conventions for the repaired line. Graders notice harmless slips and price them differently: on the 15 harmless texts where both pair graders quote the planted slip, one awards full marks on 2/15 and the other on 14/15, reversing which grader looks most accurate on the strict mark; a retest keeps the reversal. The completeness flag sits between score thresholds and differs from them only within grade 6. An obligation audit whose prompt says not to repair, read as a comparator, orders the graders the same way (12/24 against 24/24 harmless mutants accepted under its rule); its held-out gap to the strict mark is exploratory and unresolved (p ≥ 0.18; its gain three announced deletions), and a pre-registered replication confirms only its gap to the pass mark. On human-majority-labelled competition proofs, the strict mark and flag err mostly by false rejection. With the audit's no-repair sentences in the grade prompt, the strict mark moves to 6/32 and 13/32 (exploratory).

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, proof grading, planted defects, mutation testing, evaluation methodology, decision rules, grading conventions, pre-registration, olympiad mathematics, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tFc1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyzes the effect of introducing specific types of mistakes in an LLM-written proof on LLM judges for those proofs. It finds that LLM judges can respond differently to these introduced mistakes: some models close to ignore harmless ones, while others deduct points for them.

Critical strengths

None

Critical weaknesses

This paper is poorly written, contains only incremental experiments, and has no results of note. I do not understand most of what the authors talk about, except for the main message behind the experiment. Among many other specific flaws, here are some:

  • The experiment is performed on a very small scale, with no confidence intervals
  • The abstract contains non-sensical or undefined sentences like "We measure that price and propose no grader.", "class is a recorded label (8 of 72 disputed)", "the strict mark", " its held-out gap to the strict mark is exploratory and unresolved", etc. None of these sentences can be understood without reading the entire paper first (and even then they remain somewhat vague).
  • There is no sense of story with a motivation, proper setup description, or proper description of the experiments.
  • The paper seems mostly LLM-written. Not a problem by itself, but it is clear that the output has not been sufficiently scrutinized for novelty and proper contributions.

Overall, the paper is far away from an interesting contribution.

Questions for authors

See above

Reviewer_j8wVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how LLM proof graders price mistakes that the standard 0–7 rubric for IMO problems leaves open: a false statement that nothing downstream uses. The study focuses on 24 proofs from the IMO 2026 competition.

Critical strengths

The general topic of the paper is important and timely: As LLMs claim to solve increasingly challenging and even previously unsolved math problems, there is an increasingly high burden to check the generated proofs for correctness, and how to price different kinds of mistakes in those proofs. This is especially true in the setting considered in this paper where the proofs are in natural language.

Critical weaknesses
  1. The motivation and contribution of the paper is extremely narrow. It focuses on a specific kind of error: where a false statement is introduced in an IMO-level proof and immediately corrected. No supporting evidence is provided for whether or how often this kind of error occurs in practice, and why this kind of error is worth studying by itself over all the possible kinds of errors in math proofs.

  2. The paper seems to be mostly AI-generated, and the authors acknowledge the use of AI. But the writing uses unnatural idioms that make it very hard to follow. Secondly, the core ideas could be explained in just a page or two, and the use of AI seems to be to fill 8 pages with low-information content.

Questions for authors

None.

Reviewer_7jgrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a synthetic mutation suite by inserting three kinds of single-location edits into 24 proofs written by a model for IMO 2026. The proofs were judged structurally complete by model evaluators, not by human reviewers. The edits are fatal (F), harmless or silently repaired (H), and gap/deleted argument (G). This yields 96 texts. A panel of five frontier LLM graders scores the corpus on the standard 0–7 scale. The authors then evaluate decision rules—score thresholds, a yes/no completeness flag, and a rule that combines two graders—under two ground-truth conventions for the harmless class: “as-written,” where H is rejected, and “repair-tolerant,” where H is accepted. The headline claim is that the two main graders in the decision pair, GPT-5.6-Sol and Claude Opus 5, treat quoted harmless slips very differently: they award 7/7 on 2/15 versus 14/15 texts where both quote the slip. This reverses which grader appears more accurate under the two conventions. The paper also includes a second-call “obligation audit” prompt that tells the model not to repair errors, as a comparator. It then reports a pre-registered replication on 48 texts and an external comparison on human-labeled competition proofs from the Open Proof Corpus.

Critical strengths
  1. The authors pair machine-checked executable refutations, via finite computation, for the 48 fatal and harmless edits with an explicit, inspectable dependency certificate. This is a clever and sensible way to separate whether a local statement is false from whether downstream lines cite it, thereby isolating the scoring step (“price”).
  2. The paper uses paired exact McNemar tests and problem-clustered sign-flip permutations. It reports granular per-grader operating points rather than hiding heterogeneity in pooled averages. It also explicitly refuses to over-claim dominance when held-out comparisons are underpowered (p ≥ 0.18). These choices reflect strong statistical discipline.
Critical weaknesses
  1. The headline reversal is true by design. As the authors acknowledge (“The reversal follows from the design; the size of the difference between two graders under one prompt does not”), the truth table fixes Originals as accepted and Fatal/Gap mutants as rejected under both conventions, while flipping the ground-truth label of all 24 Harmless (H) mutants from rejected (as-written) to accepted (repair-tolerant). Mathematically, any two graders that differ sufficiently in how often they award 7 versus 6 to uncited slips will automatically swap accuracy rankings when the ground-truth label of those 24 items is inverted. The rank reversal is therefore an artifact of the label definition. The empirical finding is simply that GPT-5.6-Sol deducts one point for a noticed typo (awarding 6), whereas Claude Opus 5 forgives it (awarding 7). Rebranding a 6-versus-7 leniency or calibration split as an identifiable “grader convention” overstates the conceptual depth of the observation.

  2. The premise that the rubric is “silent” is debatable. The paper’s framing rests on the claim that the standard 0–7 rubric “bands a small, straightforwardly repairable error at 5–6, and it is silent on a false line the proof itself corrects a line later.” However, a localized sign typo in a displayed equation—such as writing N-b+x = N-k instead of N-b-x = N-k in the worked example of §J—is arguably a textbook instance of a “small, straightforwardly repairable error.” Under a plain reading of the rubric, GPT-5.6-Sol, which quotes the false equation, notes that the correction is immediate, and awards a 6, is strictly following the rubric’s 5–6 band. Conversely, Claude Opus 5, which explicitly logs the false equation in its errors list as a typo yet still awards a 7, is overriding the rubric band. The manuscript never justifies why a false equation becomes exempt from the rubric’s 5–6 band merely because the subsequent sentence proceeds with the intended expression.

Questions for authors
  1. Two-sided instruction ablation. What happens when the grading prompt is augmented with an explicit one-sentence disambiguation rule—either (Arm A) “If a displayed calculation contains a typo that is unused later and silently corrected on the next line, do not deduct points (score 7 if otherwise complete)” or (Arm B) “If any displayed equation is false as written, deduct at least 1 point (score at most 6) even if the next line uses the correct value”? Which of the five graders actually comply with both directions?
  2. Human-priced harmless slips. In the external Open Proof Corpus (§8) or in real Olympiad coordination, how do human coordinators and the tested LLM graders score naturally occurring “silently repaired” slips? Do human judges uniformly award 7, or do they split between 6 and 7?
  3. Selection of the decision pair. Why were GPT-5.6-Sol and Claude Opus 5 selected as the specific two-grader decision pair out of the five-grader panel, since GLM-5.3 and Kimi K3 behave like Claude Opus 5 on harmless slips, scoring 7 on 11/11 and 11/14? Was this specific pair locked in before any calibration responses were inspected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.