acceptodds
Under review as a conference paper at ICLR 2027

Learning Strategies to Break Judges

Abstract

As AI agents surpass human performance, it becomes exceedingly hard for system designers to evaluate them directly and understand their failure modes. Consequently, agents themselves are being deployed extensively to evaluate, judge, and provide feedback on model traces. But this raises an important question: how can we trust the judge? In this work, we propose an agent-guided method to find weaknesses of agentic judges that expose interpretable failure mechanisms. Our method focuses on mathematical reasoning and proceeds in two stages: first, we deploy adversarial agents to mutate a set of sound proofs by introducing errors, attempting to misguide judges—in other words, injecting errors that judges are unable to catch. Then, we distill these attempts into a small set of mutation strategies which allow us to analyze the failure modes of the judges. To ensure that these strategies are not overfit to the initial set of proofs, we evaluate them by applying the mutation strategies to a held-out set of proofs and querying the same judge. We deploy our method on GPT-5.6-sol and Claude Opus 5, paired with their agent orchestrators, Codex and Claude Code, respectively. These are used both as mutators to introduce errors and as judges to evaluate correctness of mathematical reasoning. We find that across all the agentic judges, we are able to distill mutation strategies that consistently bypass their evaluations, thereby enabling us to ascertain actionable failure modes. Our analysis also reveals that judge reliability degrades at the frontier: errors in Olympiad-level proofs or graduate-level mathematical texts are detected more consistently, whereas flaws in research-level manuscripts are more likely to escape detection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agentic Judges, Adversarial Agents, Trustworthy judges, Failure modes, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BT9bOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an agent-guided framework for identifying systematic failure modes in LLM-based mathematical proof judges. The framework uses adversarial mutators to introduce logical errors into mathematical proofs, evaluates whether judges detect these errors, and distills successful attacks into interpretable mutation strategies. Experiments across two frontier agent systems and four mathematical corpora demonstrate that these strategies transfer to held-out proofs and reveal specific weaknesses in mathematical reasoning verification.

Critical strengths
  • Important and well-motivated problem: Frontier models are increasingly used to verify mathematical reasoning that humans can’t easily check. Knowing when these agentic judges fail is a prerequisite for trusting them in self-improvement or oversight loops.

  • Interpretable and actionable failure modes: The framework distills adversarial attempts into a natural-language strategy library: dos and don’ts plus mechanisms. The two case studies in Section 4 are mathematically correct and genuinely illuminating: exact->leading-order substitution inside relative asymptotics, and weakening a fine condition while keeping a coarse surrogate. The “don’t” rules also give a useful negative picture of what judges do catch reliably.

  • Careful experimental controls and transparency: The experiment covers the mainstream models and datasets used in frontier math problems, with strong provenance and openness.

Critical weaknesses
  • Reliance on LLM-based validation of the introduced errors. One concern I have is that the ground-truth labels in this work are largely determined by LLM agents themselves. Although the authors separate the mutation checker, blind judge, and error matcher into independent calls, these components may still share similar reasoning limitations, especially when dealing with subtle mathematical errors. The paper mentions manually inspecting ten mutation checker outputs during discovery for each model, but I wonder whether this is sufficient to establish the reliability of the labels across different mathematical domains. This is particularly relevant for research-level proofs, where even determining whether a proposed mutation introduces a genuine logical error can be challenging. I understand that formally verifying all these mutations would be impractical, but a more systematic human evaluation on a representative subset of successful attacks would help strengthen the empirical conclusions.

  • Limited comparison with alternative approaches to mathematical proof verification. The main experiments compare strategy-guided mutation against unguided mutation, which is a reasonable baseline for evaluating whether the learned strategies are useful. However, I would be interested in seeing how the discovered failure modes interact with more structured approaches to proof verification. For example, the paper discusses pseudo-formalization and other methods that decompose mathematical arguments into smaller, more explicit verification steps, but these approaches are not evaluated experimentally. Since some of the identified failures involve missing dependencies or subtle changes to mathematical definitions, it would be interesting to understand whether such errors remain difficult when judges are given a more structured verification workflow. I do not think a full Lean-based baseline is necessarily required for this paper, but even a smaller comparison could help clarify which weaknesses are specific to the current agentic judging setup.

  • Some experimental counts could use clarification. I also noticed a few apparent inconsistencies in the reported judge-call counts. For example, Appendix E, Table 3 reports 301 completed GPT-5.6-sol judge calls on GraduateCourses, whereas Appendix C, Table 1 reports 293 completed reviews, and the evaluation protocol specifies three independent reviews per valid mutation. Similar discrepancies appear in some other dataset/model combinations. These differences may come from how failed calls or retries are counted, but I could not find a clear explanation in the paper. It would be helpful for the authors to clarify the counting procedure and ensure that the reported statistics are consistent across the main text and appendices.

Questions for authors

As stated in weaknesses section:

  • How reliable are the mutation validity and error-matching checks? Since the framework relies on LLM agents to establish whether an introduced error is genuine and whether the judge detected it, I wonder whether the authors have considered a larger-scale human validation study. In particular, are there cases where the mutation checker and blind judge disagree because the mathematical argument itself is ambiguous, rather than because the judge missed a genuine error?

  • Can the learned strategies actually help improve judge reliability? The paper demonstrates that the strategies transfer to unseen proofs, but I wonder whether they could also be used to improve the judges themselves. For example, could the authors incorporate a few discovered failure patterns into the judge's instructions and evaluate whether this reduces the miss rate on held-out mutations, without introducing too many false positives?

  • Could the authors clarify some of the experimental statistics? In particular, Appendix E, Table 3 reports different completed-call counts from Appendix C, Table 1 for some configurations, including GPT-5.6-sol on GraduateCourses. How are these counts reconciled with the three-review evaluation protocol? Also, how exactly are the overall 1.5× and 3.5× improvements reported in the introduction aggregated across datasets?

1 comment

We thank the reviewer for their questions and suggestions; we have updated our submission and changes are highlighted in blue. This has improved the clarity and reproducibility of our work. Here are responses to the questions:

  • How reliable are the mutation validity and error-matching checks? Since the framework relies on LLM agents to establish whether an introduced error is genuine and whether the judge detected it, I wonder whether the authors have considered a larger-scale human validation study. In particular, are there cases where the mutation checker and blind judge disagree because the mathematical argument itself is ambiguous, rather than because the judge missed a genuine error?

    As described in Section 2, a PhD student with an undergraduate degree in mathematics manually verified ten mutations that were validated by the MutationChecker during the discovery phase of each model.

    We emphasize that the Mutator agent provides both a mutation and an argument of why it breaks the proof, and so the MutationChecker only exists to perform an additional validation of this argument. The frontier models we have chosen are capable of completing this easy task accurately.

  • Can the learned strategies actually help improve judge reliability? The paper demonstrates that the strategies transfer to unseen proofs, but I wonder whether they could also be used to improve the judges themselves. For example, could the authors incorporate a few discovered failure patterns into the judge's instructions and evaluate whether this reduces the miss rate on held-out mutations, without introducing too many false positives?

    We have added a baseline-versus-guided judge experiment in Appendix L. This guidance consists of a rubric that was manually synthesized from the mutation strategies.

    Each model judges its own model family’s mutations on held-out proofs, using one fresh medium-effort review per condition. Separate GPT-5.6-sol medium calls are used as JudgeCheckers to assess detection.

    The results indicate that this additional text-based guidance has limited impact on the performance of the judge. Improving the judge will likely require using our pipeline to generate data for tuning the harness or further post-training of the models themselves.

JudgeBaseline detectedGuided detected
GPT-5.6-sol28/5030/50
Claude Opus 524/4728/47
  • Could the authors clarify some of the experimental statistics? In particular, Appendix E, Table 3 reports different completed-call counts from Appendix C, Table 1 for some configurations, including GPT-5.6-sol on GraduateCourses. How are these counts reconciled with the three-review evaluation protocol? Also, how exactly are the overall 1.5× and 3.5× improvements reported in the introduction aggregated across datasets?

    Table 3 contains run-time data from a few additional judge calls that did not produce a usable output and were retried. We have recomputed and updated the paper with the run-time data using only the completed judge calls used for all the main results.

    The numbers 1.5x and 3.5x correspond to the averages across the four datasets of the judge miss-rate ratios of guided vs. unguided mutations, for GPT 5.6-Sol and Claude Opus 5 respectively. We have also updated the paper with this clarification.

Reviewer_tLgGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an agent-guided mutation framework to discover logical errors in correct mathematical proofs that evade LLM/agent judges, then distills successful attempts into interpretable mutation strategies. Across Olympiad, graduate, OpenAI-TCS, and arXiv math corpora, strategy-guided mutators increase judge miss rates relative to unguided mutators (reported 1.5×1.5\times and 3.5×3.5\times for GPT-5.6-sol and Claude Opus 5). Held-out evaluations support transfer, show higher miss rates on research-level texts, and show that higher reasoning effort does not eliminate misses. Qualitative analysis identifies two recurring successful strategies: exploiting asymptotic approximations and weakening fine-grained conditions while preserving coarse surrogates.

Critical strengths

The paper’s most consequential strength is its problem framing and experimental staging: it moves beyond anecdotal judge failures by separating discovery, distillation, and held-out evaluation, and by using disjoint proofs, strict judge prompts, independent mutation checking, three blind judges, cross-model probes, and higher-reasoning checks. This gives the transfer claim substantially more credibility than a single-model case study.

Critical weaknesses
  1. The core measurement depends on LLM-based MutationChecker and JudgeChecker labels. Only ten mutation-checker outputs per model were manually inspected, and judge misses are not validated by humans or formal verification at scale. Because the central metric is “judge miss rate,” ambiguous mutations, checker mismatches, or pre-existing proof issues could materially bias the results. This is the most serious threat to construct validity.
  2. The generality and actionability claims are overbroad relative to the evidence. Corpora are small (10–40 proofs per dataset), only two model families at medium reasoning are used for the main comparison, cross-model comparisons are success-selected and not controlled, and the paper does not test whether revising judge prompts or workflows using the learned strategies actually improves held-out judge reliability. Thus “general failure mechanisms” and “actionable oversight” are plausible but not fully established.
Questions for authors
  1. What is the comprehensive manual verification protocol and sample size used for the outputs of MutationChecker and JudgeChecker? Does this protocol account for inter-rater agreement and the procedure for adjudicating ambiguous cases? Such information is critical for establishing the credibility of the core metric—the "miss-rate."
  2. Beyond the introduction of a "frozen strategy library," how did the authors ensure that the "strategy-guided" and "unguided" mutation-generation prompts remained consistent in terms of length, format, level of detail, and information content? This is essential to confirm that any observed performance gains stem from the extracted strategies themselves, rather than from variations in the prompts.
  3. Could the authors conduct (or report on) a controlled, cross-model evaluation—where the same set of mutations is assessed by different model families or by evaluators with stronger reasoning capabilities—rather than filtering candidate samples based solely on their "success" as judged by the original evaluator?
  4. Have the authors tested whether the learned strategies can be used to optimize "judge prompts" or verification workflows, and whether doing so enhances the system's reliability when applied to previously unseen proofs?
2 comments

We thank the reviewer for their questions and suggestions; we have updated our submission and changes are highlighted in blue. This has improved the clarity and reproducibility of our work. Here are responses to the questions:

  1. What is the full human-validation protocol and sample size for MutationChecker and JudgeChecker outputs, including inter-rater agreement and how ambiguous cases were adjudicated? This materially affects confidence in the central miss-rate metric.

    As described in Section 2, a PhD student with an undergraduate degree in mathematics manually verified ten mutations that were validated by the MutationChecker during the discovery phase of each model. The per-dataset breakdown was: 2 for Olympiad, 2 for GraduateCourses, 3 for OpenAI-TCS and 3 for ArxivMath. No mistakes were flagged.

    We emphasize that the Mutator agent provides both a mutation and an argument of why it breaks the proof, and so the MutationChecker only exists to perform an additional validation of this argument. The frontier models we have chosen are capable of completing this easy task accurately.

  2. Beyond adding the frozen strategy library, how were the strategy-guided and unguided mutator prompts matched for length, format, specificity, and informational content? This is needed to attribute gains to the distilled strategies rather than prompt differences.

    Both of these use the same base prompt. The strategy-guided mutator additionally had access to a file containing the strategies and so we are able to attribute the gains to the strategies.

  3. Can the authors run or report a controlled cross-model evaluation where the same mutations are judged by both model families and by higher-reasoning judges, rather than selecting candidates conditioned on success against the source judge?

    We have added a controlled comparison in Appendix K in which all four judge configurations review the same 20 frozen mutations: GPT-5.6-sol at medium and ultra effort, and Claude Opus 5 at medium and max effort.

    We select at random five unique valid strategy-guided mutations from each dataset/source-model combination, without conditioning selection on whether the original source judge missed the error. Separate GPT-5.6-sol medium JudgeChecker calls match each judge’s report against the planted error.

    Entries below report misses / completed reviews; lower is better.

DatasetMutation sourceGPT mediumGPT ultraClaude mediumClaude max
OpenAI-TCSGPT-5.6-sol4/53/51/51/5
OpenAI-TCSClaude Opus 51/52/52/51/5
ArXivMathGPT-5.6-sol3/51/53/51/5
ArXivMathClaude Opus 53/52/52/52/5
OverallBoth models11/208/208/205/20

Higher reasoning effort reduces aggregate misses for both models, but does not eliminate them. These results provide a small descriptive comparison on shared inputs, rather than establishing a general ranking of judge quality or compute efficiency.

  1. Have the authors tested whether using the learned strategies to revise judge prompts or verification workflows improves reliability on unseen proofs?

    We have added a baseline-versus-guided judge experiment in Appendix L. This guidance consists of a rubric that was manually synthesized from the mutation strategies.

    Each model judges its own model family’s mutations on held-out proofs, using one fresh medium-effort review per condition. Separate GPT-5.6-sol medium calls are used as JudgeCheckers to assess detection.

    The results indicate that this additional text-based guidance has limited impact on the performance of the judge. Improving the judge will likely require using our pipeline to generate data for tuning the harness or further post-training of the models themselves.

JudgeBaseline detectedGuided detected
GPT-5.6-sol28/5030/50
Claude Opus 524/4728/47

Thank you for the detailed response. The current rating I have given is reasonable.

Reviewer_BEyrOpenReview ↗
Rating
2(−2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies which logical errors agentic judges miss when they check mathematical proofs. The method has two stages. In discovery, a Mutator agent plants a concise logical error in a correct proof, a MutationChecker confirms the error is valid, a Judge reviews the proof blind, and a JudgeChecker decides whether the Judge found the error; this result is fed back to the Mutator. In distillation, an agent summarizes all attempts into a strategy file of at most 20,000 characters. The authors compare a StrategyGuidedMutator with an UnguidedMutator on disjoint held-out proofs, using Codex (GPT-5.6-sol) and Claude Code (Claude Opus 5) on four corpora: Olympiad, GraduateCourses, OpenAI-TCS, and ArXivMath. The paper reports three findings: the strategies transfer to unseen proofs (guided mutation raises the judge miss rate by about 1.5× and 3.5×), judges are less reliable on research-level text, and higher reasoning effort does not remove these misses.

Critical strengths
  1. The problem is important. LLM judges and reward models for mathematical proofs are now common, and a systematic way to find their blind spots has practical value.
  2. The framework is clear and well designed. The roles are cleanly separated, and all prompts are given in full in Appendix I. The authors state that code, data, and strategy libraries are in the supplementary material.
  3. The evaluation uses a held-out split and a paired guided/unguided comparison on the same proofs. This controls for differences between proofs. For Claude Opus 5, guided is above unguided on all four datasets, which is fairly strong evidence of transfer for that model.
  4. The data span competition problems to arXiv papers from 2023–2026, so difficulty varies widely. Appendix B documents sources and their reliability in detail.
  5. The two qualitative examples in Section 4 (Mutation 1 and Mutation 2) are mathematically correct and help readers see what kind of error is hard to detect. I checked the ratio in Mutation 1, y^δ Γ(L+3/2)/Γ(L+3/2+δ) ~ (y/L)^δ; with y = L/8 and δ → 1/2 it does tend to 1/√8.
  6. The span and edit-size analysis in Appendix G rules out the simple explanation that misses happen only because the edits are small.
  7. The authors are open about several limitations. Appendix D states that the cross-model comparison is not controlled, and the captions of Figures 3, 6, and 7 state that the Wilson intervals are not corrected for clustering.
Critical weaknesses

Key reasons for my recommendation: (1) The core metric, judge miss rate, rests on same-model LLM judgments (MutationChecker and JudgeChecker) with very little human validation, especially on research-level data. (2) The transfer claim is weak for GPT-5.6-sol and lacks cluster-aware statistics and baselines that isolate the learned strategies as the active ingredient.

1. The validity of misses is not checked by humans, especially on research-level data. The miss rate depends on two LLM judgments: the MutationChecker decides that the error is real, and the JudgeChecker decides that the judge did not find it. Both use the same model as the Judge (end of Section 2). Section 2 reports a manual check of only 10 MutationChecker outputs per model, from the discovery stage. This sample is small, does not cover the evaluation stage, is not stratified by dataset, and does not check the JudgeChecker at all. On research-level text (OpenAI-TCS, ArXivMath), deciding whether a mutation truly breaks the argument is hardest. If the MutationChecker accepts harmless or debatable edits more often there, a judge that "misses" them is behaving correctly. This would inflate research-level miss rates and could create the difficulty gradient in Section 3.2. The JudgeChecker criterion is also strict (Appendix I.4: "There is no partial credit: true only for an exact or logically equivalent causal diagnosis"). A judge that flags a downstream symptom without the exact root cause is counted as a miss. I suggest a blind review by domain experts (at least PhD-student level) of all evaluation-stage misses, or a stratified random sample, reporting (a) the fraction of errors that are truly valid and (b) the fraction already identified in the judge's report. Please report these by dataset and arm. I also suggest using a model from another provider as MutationChecker and JudgeChecker to test whether the conclusions hold.

2. The statistical evidence for transfer is weak. Figure 3 gives only Wilson intervals, which the authors note are not corrected for clustering. The number of independent units is small. Each arm has only 5 candidates per proof (Olympiad: 200 candidates / 20 proofs / 2 arms), and OpenAI-TCS and ArXivMath each have only 5 held-out documents. The 3 reviews per mutation are not independent, and Appendix H shows that about 31% of valid mutations are semantic duplicates (Table 4: only 270 distinct error classes among 392). For GPT-5.6-sol, the two arms are equal on Olympiad (0.7% vs 0.7%), close on GraduateCourses (12.2% vs 10.9%), and have overlapping intervals on ArXivMath (33.3% vs 25.3%). Only OpenAI-TCS shows a clear gap (29.0% vs 11.1%). Figure 6 shows that 14 of GPT's 27 misses on OpenAI-TCS come from the coding theory chapter, and Section 4 notes that 6 of 10 successful mutations in that run use the same asymptotic-approximation strategy. The 2.6× effect may be driven by one or two documents. For Claude on OpenAI-TCS, the unguided 5.6% corresponds to about 4 misses, which makes any ratio with it as denominator unstable. Finally, discovery and distillation were each run once, so there is no variance across seeds or distillation runs. I suggest (a) a paired guided-vs-unguided test with proofs as clusters, using mixed-effects logistic regression or a cluster bootstrap, with p-values; (b) per-proof guided/unguided miss counts; and (c) 2–3 repeated discovery+distillation runs for at least one model, reporting variance across strategy libraries.

3. Headline numbers are not defined, and some claims overstate the evidence. The introduction says guided mutation makes escape "1.5× and 3.5×" more likely, but the paper does not define this number. From Figure 3, a simple average of per-dataset ratios gives about 1.51× and 3.44×, while pooling over reviews gives about 1.45× and 3.0×. The first method gives large weight to ratios from very small counts, such as Claude's 8.1%/2.7% on Olympiad. Please define the computation and give confidence intervals. The abstract says "across all the agentic judges, we are able to distill mutation strategies that consistently bypass their evaluations", but for GPT-5.6-sol only one of four datasets shows a clear effect (Weakness 2). Figure 1 shows only GPT-5.6-sol guided miss rates (0.7%, 12.2%, 29.0%, 33.3%), while its caption says strategy-guided mutators cause misses more consistently than unguided ones. The unguided values (0.7%, 10.9%, 11.1%, 25.3%) are not shown, and three of the four are close to the guided values. I suggest showing both arms in Figure 1 and revising the abstract.

4. The claim that judges are less reliable on research-level text has several confounds. Section 3.2 and the abstract read the rising miss rate across datasets as lower judge reliability on frontier mathematics. Besides the validity issue in Weakness 1, there are four confounds. (a) Contamination and memorization: Olympiad problems and MIT course notes likely appear often in training data, so a judge can compare the mutated text with a memorized original. The 2026 OpenAI-TCS chapters and recent arXiv papers are less likely to be memorized. This effect cannot currently be separated from reasoning difficulty. A control would help, such as course-level material published after the model's training cutoff, or paraphrased Olympiad solutions before mutation. (b) The Mutator's ability also changes with domain; on research-level text it may write errors that are harder to judge. The current design cannot separate "the judge is weaker" from "the mutations are harder". (c) Olympiad proofs differ in length from the other datasets. The authors note this, but Olympiad still anchors the gradient. (d) Claude's results do not show a monotone gradient; its miss rate on OpenAI-TCS is not higher than on GraduateCourses, as Section 3.2 acknowledges. I suggest stating this result as a descriptive observation and listing contamination in the Limitations.

5. The Olympiad data may not consist of correct proofs, and the main text conflicts with the appendix. Section 3.1 and the introduction say dataset quality is "established through human verification" and that OlympiadBench "supplies expert-level step-by-step annotations". Appendix B.1 says the "proofs" are answers generated by gpt-oss-120b, and the TRUE label comes from a binary correctness label in that run; the authors cannot confirm it was assigned by human experts. The subset is OE_TO_maths_en_COMP. In OlympiadBench naming, OE means open-ended (problems with a final answer) and TP means theorem proving. If so, TRUE likely reflects only a correct final answer, and the reasoning may not be rigorous. Flaws in the original solution make it harder for the MutationChecker to judge a "new" error and can distract the judge. I suggest describing the source accurately in the main text and, ideally, switching to the OlympiadBench TP subset or a human-graded proof dataset (for example, IMO/USAMO proofs with human scores).

6. Missing baselines and ablations prevent attributing the gain to strategies learned from judge feedback. The only baseline is the UnguidedMutator. Guided and unguided differ in at least three ways: (a) the content of the strategy file, (b) a longer prompt, and (c) the "Assigned strategy" mechanism in Appendix I.1, which gives each guided candidate a specific strategy and so enforces diversity and specificity. The unguided arm has no counterpart to (c). I suggest these controls: (i) a hand-written generic list of subtle error types of the same length, assigned per candidate in the same way; (ii) distillation from the same mutation logs with caught/missed information hidden; (iii) cross-model use of strategy libraries, for example GPT-learned strategies against the Claude judge, compared with Claude's own strategies, to test whether strategies are judge-specific; (iv) a library distilled only from caught mutations, as a negative control.

7. The "no time limit" claim conflicts with the appendix, and excluded calls may bias results. The introduction and Section 2 state "we also do not impose any time limits on the judges" and use this to argue that judges truly miss the errors. Appendix E describes a 20-minute call timeout and a Claude Code output-token limit. Table 3 shows that 22 Claude calls on OpenAI-TCS (about 16%) and 25 on ArXivMath were excluded, after running for 20 to 101 minutes. The longest reviews may be the cases where the judge is tracing a suspicious step, so excluding them (or counting them as misses) can bias the miss rate. The paper does not say whether these slots are removed from the denominator in Figure 3 or counted as not detected. Figure 6 uses "available review slots" as the denominator, which does not fully match Table 3 (for Claude on OpenAI-TCS, the per-domain slots in Figure 6 sum to 135, while completed plus excluded calls in Table 3 sum to 138). I suggest revising the time-limit statement, reporting exclusions per arm, and either rerunning these calls without a timeout or adding a sensitivity analysis (all counted as detected / all counted as missed).

8. The higher-reasoning and cross-model experiments have selection bias and lack controls. In Figure 4b, the selected mutations are ones the original medium-effort judge missed. Because judges are stochastic, a fresh medium-effort judge would also catch some of them (regression to the mean). Without that control, GPT ultra catching 5/12 cannot be attributed to higher reasoning effort. The two models also use different selection rules: Claude's 6 samples were missed by all 3 original judges, while GPT's set "includes the strongest mutations and several next-ranked ones", some of which may have been caught by some original judges. Each group has only 5–6 samples. For the cross-model experiment (Figure 4a / Table 2), Appendix D states that 8 of the 10 GPT Olympiad candidates had no misses originally, and other cells differ in composition, so the two directions cannot be compared. The introduction uses this result to compare judge quality, which is not well supported. I suggest (a) a fresh medium-effort judge on the same samples as a control, (b) larger samples with a common selection rule for both models, and (c) randomly sampled guided and unguided mutations for the cross-model experiment.

9. Judge behavior on the original proofs is not measured. Appendix I.4 states "No original review was run", so the number of errors judges report on unmutated proofs is unknown. Table 1 shows the GPT judge reports on average 16.9 "other errors" per review on GraduateCourses and 6.05 on ArXivMath. Either the source texts have many problems, which conflicts with the "sound proofs" premise, or the judge has a high false-positive rate. Both affect the reliability of JudgeChecker matching and the reading of "judge misses". On OpenAI-TCS the average is only 0.21, and most missed reviews report no error at all. This points to a strong judge bias on OpenAI-TCS text, possibly linked to its source or style, which is itself worth analyzing. I suggest running the same judge 3 times on each original proof, reporting the false-positive rate, and controlling for the baseline error count in the analysis.

10. The claim of "actionable failure modes" is not tested. The abstract says the method can "ascertain actionable failure modes", but the Limitations section states that the authors did not test whether changing the judge prompt or pipeline using these strategies improves reliability. Relatedly, Section 4 shows only two selected examples with no systematic coding of all misses, and the paragraph "What the distilled strategies advise against" cites text written by the Distillation agent as evidence of what judges do well, without checking it against data. I suggest a small closed-loop experiment: convert the "Do not" items and the most frequent strategies into a judge checklist, then compare miss rates on held-out guided mutations before and after, and measure the change in false positives on original proofs.

Minor issues:

  • The introduction uses an unconfirmed claim (OpenAI solving the Navier–Stokes Millennium Problem) as motivation, citing only a company blog. Please remove it or add a qualifier.
  • Section 3 says evaluation mutations are made "without feedback from earlier candidates", but the "Novelty and reward guidance" in Appendix I.1 is also appended to evaluation prompts, and "Single-submission scheduling" says "Earlier feedback concerns different proofs". Please clarify what feedback, if any, the evaluation Mutator sees.
  • Figure 3 puts discovery and evaluation miss rates in one plot although their units differ. Claude's discovery bars (GraduateCourses 11.2%, OpenAI-TCS 8.3%) are far below the guided bars (32.0%, 30.3%). Please explain why discovery with feedback is lower than zero-shot guided; the duplicate-penalty reward design may be the cause.
  • Please report the MutationChecker pass rate for each arm. Figure 2 says "Compare valid-error & bypass rates", but only bypass rates are reported.
  • No strategy library is shown. Please include at least one full library in the appendix so readers can judge its interpretability.
  • Reproducibility details are missing: model versions or snapshot dates, sampling settings such as temperature, Codex/Claude Code versions, total call counts, and cost. Section 3.2 also refers to "full counts" in Appendix C, but Table 1 covers only GPT-5.6-sol.
Questions for authors
  1. How exactly are the 1.5× and 3.5× figures computed? What are the review-pooled ratios and their confidence intervals?
  2. Among evaluation-stage misses, how many have been confirmed by humans as valid errors? How many were in fact identified by the judge but marked as unmatched by the JudgeChecker? Can you give both fractions by dataset?
  3. In Figure 3, are timed-out or failed judge calls removed from the denominator or counted as not detected? How many were excluded in each arm?
  4. If the "Assigned strategy" mechanism is added to the unguided arm (for example, assigning a generic error type), how much of the guided advantage remains?
  5. Does distillation without caught/missed information, or a hand-written generic checklist, achieve a similar effect?
  6. Is GPT's effect on OpenAI-TCS driven mainly by the coding theory chapter? What are the results without that chapter?
  7. How was the TRUE label in the Olympiad data produced? Did it check only the final answer, or the full reasoning?
  8. For the samples in Figure 4b, what is the miss rate under a fresh medium-effort judge?
  9. How many errors do judges report on average on the original, unmutated proofs? How should the roughly 17 "other errors" per review on GraduateCourses be interpreted?
  10. How stable are the strategy libraries and their effects across repeated discovery and distillation runs?

Suggestions (not affecting my score):

  • Section 2 ends with "we would like to train agents that can critique and improve their own judgment skills". No model is trained in this paper, so I suggest rewording.
  • The Figure 4b caption says two Claude Opus 5 (max) calls exited without a result, but the denominators (5/5 and 6/6, total 11) suggest only one is missing. Please check.
  • Please explain why OpenAI-TCS is missing from Table 5 (Claude semantic uniqueness).
  • The Lean certificates for OpenAI-TCS show that theorems hold, but they do not show that every reasoning step in the Markdown text is correct. Please note this where they are used to support proof quality.
  • Please define how "two of the most successful learned mutation strategies" (contribution 3 in Section 1) were chosen.
  • The related work could cover more on adversarial perturbations in LLM proof evaluation and follow-up work on sneaky provers and prover-verifier games.
1 comment

We thank the reviewer for their questions and suggestions; we have updated our submission and changes are highlighted in blue. This has improved the clarity and reproducibility of our work.

  1. How exactly are the 1.5× and 3.5× figures computed? What are the review-pooled ratios and their confidence intervals?

    The numbers 1.5x and 3.5x correspond to the averages across the four datasets of the judge miss-rate ratios of guided vs. unguided mutations, for GPT 5.6-Sol and Claude Opus 5 respectively.

    When considering a pooled ratio across all the mutations, we get 1.5x and 3.0x respectively, with 95% confidence intervals being 1.0x-2.1x and 2.1x-4.2x.

  2. Among evaluation-stage misses, how many have been confirmed by humans as valid errors? How many were in fact identified by the judge but marked as unmatched by the JudgeChecker? Can you give both fractions by dataset?

    The JudgeChecker is tasked with checking whether the list of mathematical errors identified by the Judge contains the error introduced by the mutation. We use the same frontier agents for this easy task.

    Upon your suggestion, we have conducted a human validation similar to the MutationChecker. 16 JudgeChecker outputs were validated for each agent (GPT and Claude) during the discovery phase. This splits up into 4 per dataset, with 2 each for the two cases where the JudgeChecker did and did not find the error introduced by the mutation. No mistakes were found in the JudgeChecker outputs.

  3. In Figure 3, are timed-out or failed judge calls removed from the denominator or counted as not detected? How many were excluded in each arm?

    All Claude Code judge calls initially subject to a 20-minute timeout were retried without a time limit. Review slots still lacking a completed response after retries were excluded from both the numerator and denominator, rather than counted as undetected errors. Among valid mutations, 2 GPT-5.6-sol slots and 38 Claude Opus 5 judge slots were excluded across both arms. We have clarified this in the revised submission.

  4. If the "Assigned strategy" mechanism is added to the unguided arm (for example, assigning a generic error type), how much of the guided advantage remains?

    Our experiments compare the strategy-guided mutator against the unguided baseline. They do not separately isolate the value of assigning just one specific mutation mechanism because the learned mutation strategy prompts contain both do’s and don’ts.

  5. Does distillation without caught/missed information, or a hand-written generic checklist, achieve a similar effect?

    While we did not evaluate distilling strategies from only the successful mutations, we conjecture that it would decrease the number of successful mutations.

  6. Is GPT's effect on OpenAI-TCS driven mainly by the coding theory chapter? What are the results without that chapter?

    No, the effect persists without coding theory, although the gap becomes smaller. The unguided arm has 3/51 missed judge calls (5.9%) compared to 10/57 missed judge calls with the guided arm (17.5%).

  7. How was the TRUE label in the Olympiad data produced? Did it check only the final answer, or the full reasoning?

    We thank the reviewer for spotting this error in our description. Indeed, only the final answers were validated to produce the label and we have updated our submission to reflect this. Nonetheless, observing the data in Table 1, we see that the judges usually only detected one or two other errors in the Olympiad proofs. Since these frontier models are more than capable of solving Olympiad math, we can remain confident about our results.

  8. For the samples in Figure 4b, what is the miss rate under a fresh medium-effort judge?

    All the mutations evaluated under the higher reasoning budget were missed by all three medium reasoning judge calls.

  9. How many errors do judges report on average on the original, unmutated proofs? How should the roughly 17 "other errors" per review on GraduateCourses be interpreted?

    Table 1 reports the number of other potential errors flagged by the judge within the mutated proof. We ask the judge to be as skeptical as possible in its reporting, and so this roughly corresponds to the average number of errors in the unmutated proofs.

  10. How stable are the strategy libraries and their effects across repeated discovery and distillation runs?

    While this is outside the scope of our paper budget at the moment, it is nonetheless an important evaluation that we hope to run in the future.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.