Key reasons for my recommendation: (1) The core metric, judge miss rate, rests on same-model LLM judgments (MutationChecker and JudgeChecker) with very little human validation, especially on research-level data. (2) The transfer claim is weak for GPT-5.6-sol and lacks cluster-aware statistics and baselines that isolate the learned strategies as the active ingredient.
1. The validity of misses is not checked by humans, especially on research-level data. The miss rate depends on two LLM judgments: the MutationChecker decides that the error is real, and the JudgeChecker decides that the judge did not find it. Both use the same model as the Judge (end of Section 2). Section 2 reports a manual check of only 10 MutationChecker outputs per model, from the discovery stage. This sample is small, does not cover the evaluation stage, is not stratified by dataset, and does not check the JudgeChecker at all. On research-level text (OpenAI-TCS, ArXivMath), deciding whether a mutation truly breaks the argument is hardest. If the MutationChecker accepts harmless or debatable edits more often there, a judge that "misses" them is behaving correctly. This would inflate research-level miss rates and could create the difficulty gradient in Section 3.2. The JudgeChecker criterion is also strict (Appendix I.4: "There is no partial credit: true only for an exact or logically equivalent causal diagnosis"). A judge that flags a downstream symptom without the exact root cause is counted as a miss. I suggest a blind review by domain experts (at least PhD-student level) of all evaluation-stage misses, or a stratified random sample, reporting (a) the fraction of errors that are truly valid and (b) the fraction already identified in the judge's report. Please report these by dataset and arm. I also suggest using a model from another provider as MutationChecker and JudgeChecker to test whether the conclusions hold.
2. The statistical evidence for transfer is weak. Figure 3 gives only Wilson intervals, which the authors note are not corrected for clustering. The number of independent units is small. Each arm has only 5 candidates per proof (Olympiad: 200 candidates / 20 proofs / 2 arms), and OpenAI-TCS and ArXivMath each have only 5 held-out documents. The 3 reviews per mutation are not independent, and Appendix H shows that about 31% of valid mutations are semantic duplicates (Table 4: only 270 distinct error classes among 392). For GPT-5.6-sol, the two arms are equal on Olympiad (0.7% vs 0.7%), close on GraduateCourses (12.2% vs 10.9%), and have overlapping intervals on ArXivMath (33.3% vs 25.3%). Only OpenAI-TCS shows a clear gap (29.0% vs 11.1%). Figure 6 shows that 14 of GPT's 27 misses on OpenAI-TCS come from the coding theory chapter, and Section 4 notes that 6 of 10 successful mutations in that run use the same asymptotic-approximation strategy. The 2.6× effect may be driven by one or two documents. For Claude on OpenAI-TCS, the unguided 5.6% corresponds to about 4 misses, which makes any ratio with it as denominator unstable. Finally, discovery and distillation were each run once, so there is no variance across seeds or distillation runs. I suggest (a) a paired guided-vs-unguided test with proofs as clusters, using mixed-effects logistic regression or a cluster bootstrap, with p-values; (b) per-proof guided/unguided miss counts; and (c) 2–3 repeated discovery+distillation runs for at least one model, reporting variance across strategy libraries.
3. Headline numbers are not defined, and some claims overstate the evidence. The introduction says guided mutation makes escape "1.5× and 3.5×" more likely, but the paper does not define this number. From Figure 3, a simple average of per-dataset ratios gives about 1.51× and 3.44×, while pooling over reviews gives about 1.45× and 3.0×. The first method gives large weight to ratios from very small counts, such as Claude's 8.1%/2.7% on Olympiad. Please define the computation and give confidence intervals. The abstract says "across all the agentic judges, we are able to distill mutation strategies that consistently bypass their evaluations", but for GPT-5.6-sol only one of four datasets shows a clear effect (Weakness 2). Figure 1 shows only GPT-5.6-sol guided miss rates (0.7%, 12.2%, 29.0%, 33.3%), while its caption says strategy-guided mutators cause misses more consistently than unguided ones. The unguided values (0.7%, 10.9%, 11.1%, 25.3%) are not shown, and three of the four are close to the guided values. I suggest showing both arms in Figure 1 and revising the abstract.
4. The claim that judges are less reliable on research-level text has several confounds. Section 3.2 and the abstract read the rising miss rate across datasets as lower judge reliability on frontier mathematics. Besides the validity issue in Weakness 1, there are four confounds. (a) Contamination and memorization: Olympiad problems and MIT course notes likely appear often in training data, so a judge can compare the mutated text with a memorized original. The 2026 OpenAI-TCS chapters and recent arXiv papers are less likely to be memorized. This effect cannot currently be separated from reasoning difficulty. A control would help, such as course-level material published after the model's training cutoff, or paraphrased Olympiad solutions before mutation. (b) The Mutator's ability also changes with domain; on research-level text it may write errors that are harder to judge. The current design cannot separate "the judge is weaker" from "the mutations are harder". (c) Olympiad proofs differ in length from the other datasets. The authors note this, but Olympiad still anchors the gradient. (d) Claude's results do not show a monotone gradient; its miss rate on OpenAI-TCS is not higher than on GraduateCourses, as Section 3.2 acknowledges. I suggest stating this result as a descriptive observation and listing contamination in the Limitations.
5. The Olympiad data may not consist of correct proofs, and the main text conflicts with the appendix. Section 3.1 and the introduction say dataset quality is "established through human verification" and that OlympiadBench "supplies expert-level step-by-step annotations". Appendix B.1 says the "proofs" are answers generated by gpt-oss-120b, and the TRUE label comes from a binary correctness label in that run; the authors cannot confirm it was assigned by human experts. The subset is OE_TO_maths_en_COMP. In OlympiadBench naming, OE means open-ended (problems with a final answer) and TP means theorem proving. If so, TRUE likely reflects only a correct final answer, and the reasoning may not be rigorous. Flaws in the original solution make it harder for the MutationChecker to judge a "new" error and can distract the judge. I suggest describing the source accurately in the main text and, ideally, switching to the OlympiadBench TP subset or a human-graded proof dataset (for example, IMO/USAMO proofs with human scores).
6. Missing baselines and ablations prevent attributing the gain to strategies learned from judge feedback. The only baseline is the UnguidedMutator. Guided and unguided differ in at least three ways: (a) the content of the strategy file, (b) a longer prompt, and (c) the "Assigned strategy" mechanism in Appendix I.1, which gives each guided candidate a specific strategy and so enforces diversity and specificity. The unguided arm has no counterpart to (c). I suggest these controls: (i) a hand-written generic list of subtle error types of the same length, assigned per candidate in the same way; (ii) distillation from the same mutation logs with caught/missed information hidden; (iii) cross-model use of strategy libraries, for example GPT-learned strategies against the Claude judge, compared with Claude's own strategies, to test whether strategies are judge-specific; (iv) a library distilled only from caught mutations, as a negative control.
7. The "no time limit" claim conflicts with the appendix, and excluded calls may bias results. The introduction and Section 2 state "we also do not impose any time limits on the judges" and use this to argue that judges truly miss the errors. Appendix E describes a 20-minute call timeout and a Claude Code output-token limit. Table 3 shows that 22 Claude calls on OpenAI-TCS (about 16%) and 25 on ArXivMath were excluded, after running for 20 to 101 minutes. The longest reviews may be the cases where the judge is tracing a suspicious step, so excluding them (or counting them as misses) can bias the miss rate. The paper does not say whether these slots are removed from the denominator in Figure 3 or counted as not detected. Figure 6 uses "available review slots" as the denominator, which does not fully match Table 3 (for Claude on OpenAI-TCS, the per-domain slots in Figure 6 sum to 135, while completed plus excluded calls in Table 3 sum to 138). I suggest revising the time-limit statement, reporting exclusions per arm, and either rerunning these calls without a timeout or adding a sensitivity analysis (all counted as detected / all counted as missed).
8. The higher-reasoning and cross-model experiments have selection bias and lack controls. In Figure 4b, the selected mutations are ones the original medium-effort judge missed. Because judges are stochastic, a fresh medium-effort judge would also catch some of them (regression to the mean). Without that control, GPT ultra catching 5/12 cannot be attributed to higher reasoning effort. The two models also use different selection rules: Claude's 6 samples were missed by all 3 original judges, while GPT's set "includes the strongest mutations and several next-ranked ones", some of which may have been caught by some original judges. Each group has only 5–6 samples. For the cross-model experiment (Figure 4a / Table 2), Appendix D states that 8 of the 10 GPT Olympiad candidates had no misses originally, and other cells differ in composition, so the two directions cannot be compared. The introduction uses this result to compare judge quality, which is not well supported. I suggest (a) a fresh medium-effort judge on the same samples as a control, (b) larger samples with a common selection rule for both models, and (c) randomly sampled guided and unguided mutations for the cross-model experiment.
9. Judge behavior on the original proofs is not measured. Appendix I.4 states "No original review was run", so the number of errors judges report on unmutated proofs is unknown. Table 1 shows the GPT judge reports on average 16.9 "other errors" per review on GraduateCourses and 6.05 on ArXivMath. Either the source texts have many problems, which conflicts with the "sound proofs" premise, or the judge has a high false-positive rate. Both affect the reliability of JudgeChecker matching and the reading of "judge misses". On OpenAI-TCS the average is only 0.21, and most missed reviews report no error at all. This points to a strong judge bias on OpenAI-TCS text, possibly linked to its source or style, which is itself worth analyzing. I suggest running the same judge 3 times on each original proof, reporting the false-positive rate, and controlling for the baseline error count in the analysis.
10. The claim of "actionable failure modes" is not tested. The abstract says the method can "ascertain actionable failure modes", but the Limitations section states that the authors did not test whether changing the judge prompt or pipeline using these strategies improves reliability. Relatedly, Section 4 shows only two selected examples with no systematic coding of all misses, and the paragraph "What the distilled strategies advise against" cites text written by the Distillation agent as evidence of what judges do well, without checking it against data. I suggest a small closed-loop experiment: convert the "Do not" items and the most frequent strategies into a judge checklist, then compare miss rates on held-out guided mutations before and after, and measure the change in false positives on original proofs.
Minor issues:
- The introduction uses an unconfirmed claim (OpenAI solving the Navier–Stokes Millennium Problem) as motivation, citing only a company blog. Please remove it or add a qualifier.
- Section 3 says evaluation mutations are made "without feedback from earlier candidates", but the "Novelty and reward guidance" in Appendix I.1 is also appended to evaluation prompts, and "Single-submission scheduling" says "Earlier feedback concerns different proofs". Please clarify what feedback, if any, the evaluation Mutator sees.
- Figure 3 puts discovery and evaluation miss rates in one plot although their units differ. Claude's discovery bars (GraduateCourses 11.2%, OpenAI-TCS 8.3%) are far below the guided bars (32.0%, 30.3%). Please explain why discovery with feedback is lower than zero-shot guided; the duplicate-penalty reward design may be the cause.
- Please report the MutationChecker pass rate for each arm. Figure 2 says "Compare valid-error & bypass rates", but only bypass rates are reported.
- No strategy library is shown. Please include at least one full library in the appendix so readers can judge its interpretability.
- Reproducibility details are missing: model versions or snapshot dates, sampling settings such as temperature, Codex/Claude Code versions, total call counts, and cost. Section 3.2 also refers to "full counts" in Appendix C, but Table 1 covers only GPT-5.6-sol.
We thank the reviewer for their questions and suggestions; we have updated our submission and changes are highlighted in blue. This has improved the clarity and reproducibility of our work. Here are responses to the questions:
How reliable are the mutation validity and error-matching checks? Since the framework relies on LLM agents to establish whether an introduced error is genuine and whether the judge detected it, I wonder whether the authors have considered a larger-scale human validation study. In particular, are there cases where the mutation checker and blind judge disagree because the mathematical argument itself is ambiguous, rather than because the judge missed a genuine error?
As described in Section 2, a PhD student with an undergraduate degree in mathematics manually verified ten mutations that were validated by the MutationChecker during the discovery phase of each model.
We emphasize that the Mutator agent provides both a mutation and an argument of why it breaks the proof, and so the MutationChecker only exists to perform an additional validation of this argument. The frontier models we have chosen are capable of completing this easy task accurately.
Can the learned strategies actually help improve judge reliability? The paper demonstrates that the strategies transfer to unseen proofs, but I wonder whether they could also be used to improve the judges themselves. For example, could the authors incorporate a few discovered failure patterns into the judge's instructions and evaluate whether this reduces the miss rate on held-out mutations, without introducing too many false positives?
We have added a baseline-versus-guided judge experiment in Appendix L. This guidance consists of a rubric that was manually synthesized from the mutation strategies.
Each model judges its own model family’s mutations on held-out proofs, using one fresh medium-effort review per condition. Separate GPT-5.6-sol medium calls are used as JudgeCheckers to assess detection.
The results indicate that this additional text-based guidance has limited impact on the performance of the judge. Improving the judge will likely require using our pipeline to generate data for tuning the harness or further post-training of the models themselves.
Could the authors clarify some of the experimental statistics? In particular, Appendix E, Table 3 reports different completed-call counts from Appendix C, Table 1 for some configurations, including GPT-5.6-sol on GraduateCourses. How are these counts reconciled with the three-review evaluation protocol? Also, how exactly are the overall 1.5× and 3.5× improvements reported in the introduction aggregated across datasets?
Table 3 contains run-time data from a few additional judge calls that did not produce a usable output and were retried. We have recomputed and updated the paper with the run-time data using only the completed judge calls used for all the main results.
The numbers 1.5x and 3.5x correspond to the averages across the four datasets of the judge miss-rate ratios of guided vs. unguided mutations, for GPT 5.6-Sol and Claude Opus 5 respectively. We have also updated the paper with this clarification.