acceptodds
Under review as a conference paper at ICLR 2027

VeRA: Renewing Reasoning Benchmarks with Executable Specifications

Abstract

Reasoning benchmarks need fresh questions that preserve the tested computation and harder questions as models improve. We introduce VeRA, which compiles an existing problem into a reusable question template, input generator, and deterministic answer program. VeRA-E varies inputs and wording within a computational family; VeRA-H modifies the computation; VeRA-H Pro selects difficult candidates. Program checks and independent item audits validate the resulting releases. Across 16 models, VeRA-E leaves mean accuracy nearly unchanged on GSM8K (94.85% to 95.20%) and Beyond-AIME (58.34% to 57.30%), but reduces AIME-2024 accuracy from 84.46% to 70.25%. This selective drop makes computation-preserving renewal a diagnostic for benchmark-specific overfitting and possible data contamination. For modified tasks, a judge selects from up to five validated proposals per original problem. On AIME-2024-II, the selected audited release lowers accuracy from 84.91% to 58.57%. Within this budget, selection yields harder releases than the broader pools on all three sources; on AMO-Bench, the selected release's mean accuracy remains above that on the originals. Targeted repair raises usable construction yield from 75.4% to 95.1%. We release executable specifications, audited items, and validation records for repeated benchmark renewal.

Primary area:
datasets and benchmarks
Keywords:
reasoning benchmarks, benchmark generation, dynamic evaluation, executable specifications, benchmark renewal, programmatic verification, benchmark hardening, fresh-instance evaluation, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_H3rROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents VeRa as a targeted approach to address benchmark saturation and staleness. VeRA works by turning each benchmark problem into a smaller program composed of three parts, including a template, a generator, and an answer program, which a larger model (GPT5) is responsible for writing. Once converted/decomposed into a program, the authors have three different versions of VeRA that work in different ways. VeRA-E works by keeping the underlying mathematics unchanged but changing the numbers and wording of the problem(s), thereby testing whether a model can perform a “known” computation on varying inputs. VeRA-h works by transforming the original benchmark problem into a harder one, and VeRA-H Pro has a strong model generate n variants of the original into transformed problems of increased difficulty, and then has that model judge them and pick the hardest one. The authors test their variants against a wide suite of models and present findings for their VeRA variants. On GSM and Beyond-AIME, the scores stay flat, but on AIME-2024 and 2025, they see performance drops, which they attribute to models being overfit on AIME benchmarks.

Critical strengths
  • The authors tackle an important problem: static reasoning benchmarks are becoming saturated and less useful for distinguishing models. This is particularly troubling when the current generation of benchmarks land in the next generation’s training data (i.e. contamination). VeRA targets both of these problems with different variants, including new versions of the problems at the same difficulty level and harder instances derived from the originals.

  • While the idea presented behind turning a benchmark into a template and generating new instances from it has been done in the literature (GSM-Symbolic, TemplateGSM, Functional Benchmarks, to name a few), the authors' proposed solution is shown to scale towards problems that are more difficult, like AIME- and AMO-level mathematical reasoning tasks.

  • The audit/human evaluation performed by the authors is relatively extensive and demonstrates that the AIME and Beyond-AIME VeRA-E transformations test the same relational concept as the original problems. The framework for resolving disagreements is also fairly extensive, and I applaud the authors for the work they put into this section of the paper.

Critical weaknesses
  • W1: I find the writing very difficult to parse and follow. The authors note that coding agents helped with their writing, but to me, the communication of ideas comes across as very choppy and fragmented (e.g., very short clauses used as sentences and a lack of connection between them, which drastically hurts readability).

  • W2: There is a 26-point drop in AIME-2024-II under H Pro, which the authors attribute to difficulty changes, but changing the numbers alone drops by 14 points. This raises the possibility that part of the effect of H Pro may come from newer data rather than anything related to the VeRA pipeline.

  • W3: Much of the analysis centers on the difficulty-changing transformation, but less on just the plain VeRA transformations. It would be nice if the paper were complete in its discussion of the effect of plain VeRA on the AIME problems it measures. This would help to resolve some issues around the contamination claim.

  • W4: Some of the tables show avg@5, but it would be great if the authors could include some measure of variance here as well. AIME is a small dataset, and models generally have a wide spread here, but it’s somewhat masked in the current version of the paper.

  • W5: The main results, and really the only results shown in the main paper (GPQA is briefly mentioned in the appendix), target mathematical reasoning benchmarks. Because of the way VeRA works, the pipeline is fundamentally limited to domains where there is a verifiable answer that can be computed from inputs via a program. This limits the applicability of VeRA as a wide-ranging tool across reasoning benchmarks. That said, the title of the paper itself feels like a bit of a stretch with the “reasoning benchmarks” designation.

Questions for authors
  • The AIME performance degradation is interesting, but I wonder if the authors could extend that analysis to historical AIME datasets (which are surely in every training dataset) and compare against AIME 2026, which may surpass the training cutoff date for even more recent model releases. I think this would help to isolate the claim that AIME scores fall due to memorization, rather than just being more sensitive to transformation than something like GSM.

  • Could the authors, in response to W4, provide input on the generality of VeRA as a means of renewing reasoning benchmarks outside of the math domain?

Presentation notes/typos:

  • While it is nice that Figures 3/4/5 are easy to read without a zoom, they take up an exorbitant amount of space in the current form. I think these can be fairly represented with a significant space reduction.

  • See W1 on the choppiness of the writing, which is an issue that starts in the abstract and extends through the entire paper. There are excessive signs of AI-generated prose (as the authors disclose), which detract from the ideal of conveying/communication the ideas of this paper.

1 comment

Thank you for your review. It credits VeRA with scaling renewal to AIME- and AMO-level problems, calls the audit extensive and rates soundness as fixable, yet recommends a clear reject. Several weaknesses misdescribe the submission.

W1. Writing. We rewrote the prose as connected paragraphs and, following your presentation note, merged the robustness and hardening figures into one (revised Figure 3). Your review, however, cites our AI-usage statement as evidence against the paper. Using a transparency disclosure against the authors penalizes honesty, and prose style is no basis for a clear reject of work whose soundness the review rates as fixable.

W2. Freshness versus hardening. The submission already reports the comparison that separates them: Section 1 and Table 3 give AIME-2024-II accuracy of 84.91% on seeds, 68.71% on H and 58.57% on H Pro. H and H Pro are equally fresh, so their 10.1 pp gap is a selection effect with freshness held fixed; on the raw pools the gap is 9.2 pp (95% CI 3.1–15.4, 15/16 models lower). Your comparison also mixes item sets: the 14-point E drop is measured on all of AIME 2024 and the 26-point drop on AIME-2024-II. Revised Section 5.2 adds two more comparisons. Fresh instances alone bring AIME 2024 to 70.25%, still 11.7 pp above H Pro; and on Beyond-AIME, where freshness costs about 1 pp, H Pro lies 4.4 pp below the seeds (raw pool, 95% CI 1.7–7.0). Freshness is also not an outside factor: producing fresh items is what the pipeline does.

W3. Analysis of plain VeRA. This is factually incorrect. Section 5.1, "Equivalent computations expose a selective AIME drop", is devoted to E (Table 2, Figures 3 and 4, Appendix A) and contains the controls summarized in point 2 of the general response; the revision expands it further.

W4. Variance. Each mean averages five samples per problem, and Appendix A lists every model's score. The revision adds paired 95% intervals over the 16 models, and both AIME drops hold for every model (sign test p = 3×10⁻⁵):

ComparisonChange (pp)95% CIModels lower
AIME 2024, E − seed−14.2[−17.0, −11.4]16/16
AIME 2025, E − seed−7.1[−9.8, −4.3]16/16
Beyond-AIME, E − seed−1.0[−3.0, +0.9]10/16
GSM8K, E − seed+0.3[−0.9, +1.6]1/16
AIME-2024-II, H Pro − seed−26.3[−30.7, −22.0]16/16

Q1. Historical AIME and AIME 2026. Section 5.1 already runs the older-versus-newer comparison you propose, with the data available: AIME 2024, in circulation longer, drops twice as much as AIME 2025 (14.2 versus 7.1 pp), although both years receive edits of the same size (Figure 4), and the ordering holds under all three teachers. Extending E to historical AIME, which enters the paper only as a hardening source, or to AIME 2026 needs new evaluations; the 16 evaluated endpoints have been retired, so the results would not be comparable with the paper's tables. No evaluation can confirm contamination: that requires the training data, which only the model developers hold. What an evaluation can provide is controlled evidence, and Section 5.1 provides it. Because VeRA is a renewal method rather than a fixed leaderboard, its value does not rest on any one panel: the released specifications make your test straightforward on future panels, and revised Section 8 describes it.

Q2 / W5. Scope. Computable answers are required by every program-based precedent you cite, including GSM-Symbolic, TemplateGSM and the functional benchmarks, and the paper reports GPQA separately for that reason. Beyond mathematics, the whole pipeline applies wherever code can compute the answer, such as code reasoning, constraint puzzles, quantitative science and planning with a plan validator, but not to conceptual multiple-choice, commonsense or proof-based tasks (revised Section 8). The title is now "Renewing Mathematical Reasoning Benchmarks with Executable Specifications".

Reviewer_GqPzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VeRA is a framework for renewing reasoning benchmarks by converting existing problems into reusable question templates, input generators, and executable answer programs. VeRA-E generates new instances of the same computational task, while VeRA-H modifies the task to seek harder problems and H Pro selects difficult candidates using a model judge. The pipeline combines automated checks, independent human audits, and repair to validate the resulting questions. Across 16 models, experiments primarily on mathematical benchmarks show that refreshed instances reduce AIME accuracy.

Critical strengths
  • A practical, reusable benchmark-construction workflow supported by validation and evaluation

  • Beyond automated checks, the paper includes independent human solving, adjudication, and targeted repair

Critical weaknesses
  • Hard to differentiate from prior work. For example, functional MATH already uses problem, input-generator, and solution functions; MATH-Perturb distinguishes solution-preserving from solution-changing modifications.

  • H Pro substantially lowers accuracy on AIME-II, offers only a modest reduction on Beyond-AIME, and produces AMO questions that remain easier than the originals. This raises the questions on how consistently it increases difficulty.

  • Limited domain coverage. VeRA is evaluated predominantly on math. The GPQA extension broadens the coverage but switches to model-based answer validation, so it does not demonstrate that the same mechanism works for general reasoning tasks.

Questions for authors
  • How does VeRA differ from works mentioned above?

  • Why does H Pro substantially increase difficulty on AIME-II, yield only modest gains on Beyond-AIME, and produce questions easier than the original AMO problems? Can you provide an analysis separating the effects of the proposed transformations, candidate-generation budget, and difficulty judge, preferably using matched seed families?

  • Which aspects of VeRA’s pipeline transfer beyond mathematics without replacing the answer program with model-based validation?

2 comments

Thank you for your review. Shared evidence is in the general response posted above. Here we answer your specific points.

Q1 / W1. Functional MATH and MATH-Perturb. The difference is automation, which the revised introduction now puts first: in VeRA a model, not a person, writes every specification (Sections 2.1 and 3.2). Functional MATH hand-writes functions for 41.2% of MATH and keeps them private, and MATH-Perturb's 12 PhD annotators wrote 279 static perturbation pairs. VeRA generates such functions automatically for GSM8K, AIME, Beyond-AIME and AMO-Bench seeds and releases them with validation records. VeRA-E is an automatic, re-drawable counterpart of MATH-Perturb's simple perturbations and VeRA-H of its hard ones; the hardened releases alone contain 2,187 audited items. Sharing a decomposition is not sharing a method; by that standard, every functional benchmark would be indistinguishable from the first. Both papers are now cited.

Q2 / W2. Consistency of H Pro. The weakness merges two separate claims. The paper claims that selection beats the unselected pool (abstract; title of Section 5.2), and that holds on every source (general response, point 3; revised Table 4). It never claimed that every release beats its seeds; its abstract reports the AMO-Bench result, and counting a disclosed result as inconsistency penalizes honest reporting. How hard a release ends up depends on how hard its seeds are (Verified releases; historical AIME has only a raw H pool; differences use unrounded scores):

SourceSeedHH ProH − seed (pp)
Historical AIME89.873.6–−16.1
AIME-2024-II84.968.758.6−16.2
Beyond-AIME58.359.555.0+1.1
AMO-Bench31.844.640.2+12.8

H accuracy is roughly 30 + 0.47 × seed accuracy and crosses the seed level near 57% (revised Section 5.2): modification restores headroom on saturated sources and, on very hard ones, settles near the difficulty the pipeline can certify. Three factors keep AMO-Bench releases easier than their seeds. Its seeds were filtered to challenge strong models; an answer program must compute every label, which pushes olympiad problems toward cases that can be enumerated; and the answer check favours tasks a judge can verify, rejecting 44.0% of execution-passing candidates in our Beyond-AIME runs. The design separates transformation (H versus seed) from selection (H Pro versus H at a budget of five), and every claim in the paper holds at that budget with the GPT-5 judge; revised Section 8 states this scope and leaves other budgets and judges to future work.

Q3 / W3. Domain. The paper presents VeRA for tasks with computable answers and reports GPQA separately precisely because its labels come from arbitration (Appendix F); a stated scope is not a defect. To your question: all of the pipeline, answer program included, transfers to any task whose answer code can compute, such as code reasoning, constraint puzzles, quantitative science, planning with a plan validator, and table or SQL reasoning. It does not cover conceptual multiple-choice, commonsense or proof-based tasks (revised Section 8), and the title now states this scope.

The response addresses some of my concerns. However, overall the work appears incremental. I would keep my score.

Reviewer_m1NxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VeRA aims to refresh math reasoning benchmarks while retaining reliable answers. A teacher converts each seed question into a text template, valid-input generator, and executable answer program. VeRA-E resamples within the family, VeRA-H modifies the computation, and VeRA-H Pro selects a difficult candidate from up to five validated proposals. After program checks and expert review, a 16-model evaluation reports AIME-2024 accuracy falling from 84.46% to 70.25% under E and AIME-2024-II accuracy falling from 84.91% to 58.57% under H Pro. E changes GSM8K and Beyond-AIME mean accuracy little; H Pro does not make AMO-Bench harder than its original questions.

Critical strengths
  1. The paper addresses a real need for renewable math evaluation and produces an inspectable, audited resource. Sections 2-3 link each question to its generator and answer program. Table 4 records expert review and repair rather than treating executable labels alone as sufficient validation.
  2. The evaluation reveals a substantive source-specific robustness gap. All 16 models in Appendix A score lower on the AIME-2024 E variants, while Table 2 shows little mean change on GSM8K and Beyond-AIME. Figure 6 gives concrete examples of incorrect dimension use and incomplete case analysis. These findings are informative without proving training-data contamination.
Critical weaknesses

I find VeRA's motivation for renewing mathematical evaluations compelling. My main concern is that the results are not analyzed deeply enough to establish the method's distinctive contribution or the value of a standalone benchmark. The paper needs stronger evidence that VeRA reveals useful new insights about model capabilities, rather than mainly extending robustness and perturbation tests of existing benchmarks.

  1. VeRA is an incremental contribution. GSM-Symbolic (ICLR 2025) already varies parameters within a mathematical problem family. RV-Bench (AAAI 2026) uses per-problem generators and executable solutions. DARG (NeurIPS 2024) varies reasoning structure and checks answers with code. Section 7 acknowledges these works and distinguishes VeRA's combination of reusable specifications, computation modification, and candidate selection. Its specific contribution is LLM-assisted conversion of diverse seed problems into audited, reusable templates, constrained generators, and answer programs for both variation and hardening. However, the paper does not show that this combination improves coverage, construction effort, or the validity of repeated draws over the closest precedents. Without such evidence, the established components limit the novelty of the integrated method.
  2. VeRA needs more quantitative and qualitative analysis to show that its results support its design. Table 2 shows a substantial AIME-2024 drop under E, but Figure 4 measures only edit magnitude and Figure 6 examines two failures. These analyses do not establish what drives the drop across models and problem families. Figure 5 shows that H Pro selects harder releases on AIME-2024-II and Beyond-AIME, but H and H Pro contain different retained items, and AMO-Bench H Pro is still easier than its seeds. Without a broader failure analysis and a matched comparison of selection effects, the E results mainly read as an LLM robustness test on variants of existing benchmarks, while the claimed benefit of H/H Pro remains incompletely explained.
  3. VeRA needs stronger evidence of practical downstream value as a benchmark. Section 6 proposes diagnosing fragile reasoning and restoring evaluation headroom, and Figure 3 shows model-specific score changes. The paper does not yet show whether these measurements improve model discrimination, guide evaluation or model-development decisions, or reveal failures missed by existing robustness benchmarks. Making familiar questions harder and lowering model scores is not by itself a sufficient reason to introduce a new benchmark. The paper should demonstrate what additional, actionable insight VeRA provides for evaluating LLMs or agents.
Questions for authors
  1. Do Table 2's 100 Beyond-AIME E variants include the 11 seed-restating fallbacks reported in Appendix I? If so, what is the result on the nonfallback variants?
  2. Can existing scores provide family-paired uncertainty for the AIME-2024 E drop and an equally family-weighted H versus H Pro comparison on shared seeds?
  3. Section 7 distinguishes VeRA from GSM-Symbolic, RV-Bench, and DARG conceptually. What measurable advantage does VeRA provide over these methods in construction effort, family coverage, or validity of later draws?
1 comment

Thank you for your review. Shared evidence is in the general response posted above. Here we answer your specific points.

W1 / Q3. "Incremental." Your review states the contribution correctly: a model converts diverse seed problems into audited, reusable templates, generators and answer programs, for both variation and hardening. No precedent does this:

Who writes the familySeeds coveredHardest sourceHardening
GSM-SymbolicHumans100 GSM8K templatesGSM8KAdded clauses, manual
RV-BenchHumans, LLM-drafted230 kept of 500 sampledMATH, LeetCodeNone
DARGAutomatic, one-off itemsPer itemGSM8K, BBHGraph width/depth
VeRAAutomatic, re-drawable families100% GSM8K and AIME 2024/2025, 89% Beyond-AIMEAMO-BenchModification + selection

This delivers the three advantages you ask for. Coverage is complete on GSM8K and AIME 2024/2025. Construction costs about USD 100 for all GSM8K seeds and about USD 11 per AIME seed (revised Section 3), and people only audit. As for later draws, blind auditors accepted all 220 audited E items, including all 209 generated variants, so the one-sided 95% bound on the error rate of a new E draw is 1.4%, and new E draws need no review, unlike one-off rewrites such as DARG. A combination is not incremental when it achieves what none of its parts does.

Q1. Fallbacks. Yes, and the paper reports them (Table 8). A restatement is the original question, so its expected change is zero. Attributing the whole change to the 89 generated variants gives about −1.2 pp; even if each restatement had moved by 10 pp in either direction, the change would stay between −2.4 and +0.1 pp, far from AIME 2024's −14.2 pp.

Q2a. Uncertainty. The AIME-2024 drop is 14.2 pp (95% CI 11.4–17.0), all 16 models score lower (sign test p = 3×10⁻⁵), and the drop exceeds Beyond-AIME's change by 13.2 pp (9.9–16.4) for every model. We pair by model because the claim concerns the evaluated panel (Appendix D), and the 16/16 sign test needs no assumption about how items vary.

Q2b. Matched H versus H Pro. The raw pools are already matched. Each AIME-2024-II and Beyond-AIME seed contributes exactly five H candidates (Appendix J), so H is the expected accuracy of one random candidate per seed, and H Pro is the judge's pick for the same seeds. H Pro is harder than H by 9.2 pp on AIME-2024-II (CI 3.1–15.4), 4.8 pp on Beyond-AIME (3.3–6.3) and 4.6 pp on the nearly balanced AMO-Bench pool (3.3–5.9), with 15/16 models lower on each. The Verified releases keep the ordering (10.1, 4.4 and 4.4 pp); revised Table 4 shows all of this.

W2. What drives the drop. Table 2, which your summary cites, contains the controls (general response, point 2), and Figure 4 is one of them. It is not meant to explain the drop; it rules out larger edits as the cause, since the AIME 2024 and 2025 variants change their seeds by the same amount (KS p = 0.51, Appendix C) while 2024 drops twice as much. Nor is the drop driven by a few models: all 16 score lower on both AIME years (Figure 3). Figure 6 shows what the failures look like. In both examples a model solves the seed but carries a feature of the seed's solution over to the variant: it treats a 1×2 grid as square, or it stops a case analysis at four vertices when the 12-gon needs six.

W3. Downstream value. Your concern is the value of a standalone benchmark, but VeRA is not one: it is a method for renewing existing benchmarks, and the released items demonstrate it. Lower scores are therefore not the point; under E, GSM8K and Beyond-AIME scores do not fall at all. What the method adds is what you ask for. It restores discrimination: on AIME-2024-II seeds, 7/16 models score at least 90% and the top four lie within 1.4 pp, whereas on the raw H Pro items no model exceeds 79% and the top four span 11.5 pp. It changes decisions: on AIME 2024, GLM-4.6 falls from a tie for 4th to 12th, and MiniMax M2 loses 29.0 pp, twice the panel mean, so choosing a model from static AIME 2024 scores would give a different ranking. And it finds fragility that a grade-school robustness test misses: on GSM8K, the setting of GSM-Symbolic, fresh instances lower only one of the 16 models, while all 16 drop on AIME 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviews

We thank the reviewers. Most objections are answered by results already in the submitted paper, and the rest turn limitations the paper itself reports into grounds for rejection. All evidence comes from the existing 16-model evaluation, with models as paired units. Numbers in our replies refer to the submitted version unless marked "revised".

1. Novelty. VeRA is a method for renewing the benchmarks the community already uses, not another benchmark. Every precedent the reviews cite either needs people to write each template, function or perturbation (GSM-Symbolic, Functional MATH, RV-Bench, MATH-Perturb) or is automatic only for grade-school or simple reasoning tasks and does not turn a benchmark's problems into reusable families (TemplateGSM, DARG). In VeRA, a model writes the template, generator and answer program for each seed (Sections 2.1 and 3.2), and people only audit; validating a specification is a one-time investment that every later draw inherits. The pipeline built a working specification for every GSM8K seed (1,319) and every AIME 2024/2025 seed (60), and for 89 of 100 Beyond-AIME seeds (Table 8). Synthesis cost about USD 100 for all of GSM8K and about USD 11 per AIME seed (revised Section 3). Blind auditors accepted all 220 audited E items, which bounds the error rate of a new E draw at 1.4% (one-sided 95%). Reviewer H3rR acknowledges that this scales renewal to AIME- and AMO-level problems.

2. The E result is controlled. Table 2 contains the controls. E leaves GSM8K (widely circulated, easy) and Beyond-AIME (hard, less circulated) unchanged (+0.3 and −1.0 pp), but lowers AIME 2024 by 14.2 pp (95% CI 11.4–17.0, all 16 models) and AIME 2025 by 7.1 pp (4.3–9.8). A pipeline that simply made problems harder would also lower Beyond-AIME, which is harder than AIME; instead, the AIME-2024 drop exceeds Beyond-AIME's change by 13.2 pp (9.9–16.4), and it does so for every model. Within AIME, the year in circulation longer drops twice as much, although both years receive edits of the same size (Figure 4), and this ordering holds under all three teachers (Section 5.1).

3. Hardening: selection beats the pool on every source. This is what the paper claims (abstract; title of Section 5.2), and it holds everywhere: on the raw pools, H Pro is harder than H by 9.2, 4.8 and 4.6 pp on AIME-2024-II, Beyond-AIME and AMO-Bench, with intervals excluding zero and 15/16 models lower on each, and the Verified releases keep this ordering. The paper never claimed that every release beats its seeds; its abstract reports that AMO-Bench H Pro remains easier, and Reviewers m1Nx and GqPz read this disclosed result as inconsistency. Where renewal is needed, hardening delivers: on the AIME sources, whose seeds score 85–90%, accuracy falls by about 16 pp from seeds to H, and on AIME-2024-II selection removes another 9–10 pp with freshness held fixed.

4. Disclosed limitations are not grounds for rejection. The reviews raise our own disclosures as weaknesses: the AMO-Bench result, stated in the abstract, and the arbitration-based GPQA extension, which Appendix F reports separately for exactly that reason. Reviewer H3rR also cites our AI-usage statement against the paper. All three reviews rate soundness as having "minor or fixable concerns", yet all recommend rejection.

What changed (revised PDF). The title and abstract are scoped to mathematical reasoning. Section 1 centres on automation, and Section 7 adds Table 6 comparing prior work and three added citations. Section 3 adds the construction yield (Table 2), the synthesis cost and the audit outcomes, and Section 4 names the evaluated models and sources. Section 5.1 adds paired intervals, the AIME-2024 versus Beyond-AIME difference and the fallback bound, and moves the worked failures into the main text (Figure 5); Figure 3 merges the robustness and hardening figures. Section 5.2 adds the matched H versus H Pro comparison with paired intervals (Table 4), separates freshness from hardening, and shows that excluding one model changes no conclusion (Appendix K). Section 6 proposes a reporting protocol, and Section 8 states the scope of executable labels and the fixed candidate budget and judge. The prose is rewritten throughout for better accessibility and readability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.