- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents VeRa as a targeted approach to address benchmark saturation and staleness. VeRA works by turning each benchmark problem into a smaller program composed of three parts, including a template, a generator, and an answer program, which a larger model (GPT5) is responsible for writing. Once converted/decomposed into a program, the authors have three different versions of VeRA that work in different ways. VeRA-E works by keeping the underlying mathematics unchanged but changing the numbers and wording of the problem(s), thereby testing whether a model can perform a “known” computation on varying inputs. VeRA-h works by transforming the original benchmark problem into a harder one, and VeRA-H Pro has a strong model generate n variants of the original into transformed problems of increased difficulty, and then has that model judge them and pick the hardest one. The authors test their variants against a wide suite of models and present findings for their VeRA variants. On GSM and Beyond-AIME, the scores stay flat, but on AIME-2024 and 2025, they see performance drops, which they attribute to models being overfit on AIME benchmarks.
Critical strengths
-
The authors tackle an important problem: static reasoning benchmarks are becoming saturated and less useful for distinguishing models. This is particularly troubling when the current generation of benchmarks land in the next generation’s training data (i.e. contamination). VeRA targets both of these problems with different variants, including new versions of the problems at the same difficulty level and harder instances derived from the originals.
-
While the idea presented behind turning a benchmark into a template and generating new instances from it has been done in the literature (GSM-Symbolic, TemplateGSM, Functional Benchmarks, to name a few), the authors' proposed solution is shown to scale towards problems that are more difficult, like AIME- and AMO-level mathematical reasoning tasks.
-
The audit/human evaluation performed by the authors is relatively extensive and demonstrates that the AIME and Beyond-AIME VeRA-E transformations test the same relational concept as the original problems. The framework for resolving disagreements is also fairly extensive, and I applaud the authors for the work they put into this section of the paper.
Critical weaknesses
-
W1: I find the writing very difficult to parse and follow. The authors note that coding agents helped with their writing, but to me, the communication of ideas comes across as very choppy and fragmented (e.g., very short clauses used as sentences and a lack of connection between them, which drastically hurts readability).
-
W2: There is a 26-point drop in AIME-2024-II under H Pro, which the authors attribute to difficulty changes, but changing the numbers alone drops by 14 points. This raises the possibility that part of the effect of H Pro may come from newer data rather than anything related to the VeRA pipeline.
-
W3: Much of the analysis centers on the difficulty-changing transformation, but less on just the plain VeRA transformations. It would be nice if the paper were complete in its discussion of the effect of plain VeRA on the AIME problems it measures. This would help to resolve some issues around the contamination claim.
-
W4: Some of the tables show avg@5, but it would be great if the authors could include some measure of variance here as well. AIME is a small dataset, and models generally have a wide spread here, but it’s somewhat masked in the current version of the paper.
-
W5: The main results, and really the only results shown in the main paper (GPQA is briefly mentioned in the appendix), target mathematical reasoning benchmarks. Because of the way VeRA works, the pipeline is fundamentally limited to domains where there is a verifiable answer that can be computed from inputs via a program. This limits the applicability of VeRA as a wide-ranging tool across reasoning benchmarks. That said, the title of the paper itself feels like a bit of a stretch with the “reasoning benchmarks” designation.
Questions for authors
-
The AIME performance degradation is interesting, but I wonder if the authors could extend that analysis to historical AIME datasets (which are surely in every training dataset) and compare against AIME 2026, which may surpass the training cutoff date for even more recent model releases. I think this would help to isolate the claim that AIME scores fall due to memorization, rather than just being more sensitive to transformation than something like GSM.
-
Could the authors, in response to W4, provide input on the generality of VeRA as a means of renewing reasoning benchmarks outside of the math domain?
Presentation notes/typos:
-
While it is nice that Figures 3/4/5 are easy to read without a zoom, they take up an exorbitant amount of space in the current form. I think these can be fairly represented with a significant space reduction.
-
See W1 on the choppiness of the writing, which is an issue that starts in the abstract and extends through the entire paper. There are excessive signs of AI-generated prose (as the authors disclose), which detract from the ideal of conveying/communication the ideas of this paper.
Thank you for your review. It credits VeRA with scaling renewal to AIME- and AMO-level problems, calls the audit extensive and rates soundness as fixable, yet recommends a clear reject. Several weaknesses misdescribe the submission.
W1. Writing. We rewrote the prose as connected paragraphs and, following your presentation note, merged the robustness and hardening figures into one (revised Figure 3). Your review, however, cites our AI-usage statement as evidence against the paper. Using a transparency disclosure against the authors penalizes honesty, and prose style is no basis for a clear reject of work whose soundness the review rates as fixable.
W2. Freshness versus hardening. The submission already reports the comparison that separates them: Section 1 and Table 3 give AIME-2024-II accuracy of 84.91% on seeds, 68.71% on H and 58.57% on H Pro. H and H Pro are equally fresh, so their 10.1 pp gap is a selection effect with freshness held fixed; on the raw pools the gap is 9.2 pp (95% CI 3.1–15.4, 15/16 models lower). Your comparison also mixes item sets: the 14-point E drop is measured on all of AIME 2024 and the 26-point drop on AIME-2024-II. Revised Section 5.2 adds two more comparisons. Fresh instances alone bring AIME 2024 to 70.25%, still 11.7 pp above H Pro; and on Beyond-AIME, where freshness costs about 1 pp, H Pro lies 4.4 pp below the seeds (raw pool, 95% CI 1.7–7.0). Freshness is also not an outside factor: producing fresh items is what the pipeline does.
W3. Analysis of plain VeRA. This is factually incorrect. Section 5.1, "Equivalent computations expose a selective AIME drop", is devoted to E (Table 2, Figures 3 and 4, Appendix A) and contains the controls summarized in point 2 of the general response; the revision expands it further.
W4. Variance. Each mean averages five samples per problem, and Appendix A lists every model's score. The revision adds paired 95% intervals over the 16 models, and both AIME drops hold for every model (sign test p = 3×10⁻⁵):
Q1. Historical AIME and AIME 2026. Section 5.1 already runs the older-versus-newer comparison you propose, with the data available: AIME 2024, in circulation longer, drops twice as much as AIME 2025 (14.2 versus 7.1 pp), although both years receive edits of the same size (Figure 4), and the ordering holds under all three teachers. Extending E to historical AIME, which enters the paper only as a hardening source, or to AIME 2026 needs new evaluations; the 16 evaluated endpoints have been retired, so the results would not be comparable with the paper's tables. No evaluation can confirm contamination: that requires the training data, which only the model developers hold. What an evaluation can provide is controlled evidence, and Section 5.1 provides it. Because VeRA is a renewal method rather than a fixed leaderboard, its value does not rest on any one panel: the released specifications make your test straightforward on future panels, and revised Section 8 describes it.
Q2 / W5. Scope. Computable answers are required by every program-based precedent you cite, including GSM-Symbolic, TemplateGSM and the functional benchmarks, and the paper reports GPQA separately for that reason. Beyond mathematics, the whole pipeline applies wherever code can compute the answer, such as code reasoning, constraint puzzles, quantitative science and planning with a plan validator, but not to conceptual multiple-choice, commonsense or proof-based tasks (revised Section 8). The title is now "Renewing Mathematical Reasoning Benchmarks with Executable Specifications".