- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper’s main idea is that checking correctness and recognising mathematical value are different tasks. In an abstract generation-in-the-limit model, a generator (e.g. an LLM) sees a record of valued results and can query a validity verifier such as Lean. A record cannot certify its own completeness, so the generator cannot always tell an important new result from valid but uninteresting mathematics. Guaranteeing broad discovery therefore requires continued exploration of results of uncertain value. The contributions are:
(i) a verifier identifies the valid statements in the limit but reveals nothing about value;
(ii) Theorem 4.1: if the record may be all or only part of the valuable set, guaranteed coverage of the larger set is half the recorded results plus one unrecorded result per unit of trivia allowed, tight up to lower-order terms;
(iii) hence guaranteed coverage is α/2 under bounded trivia and 1 − α/2 under unbounded trivia (α = recorded share), with the budget setting only the recovery time;
(iv) a compression-based value model in which the record has density zero.
Validation consists of proofs, a Lean formalisation of Theorem 4.1 and several other results, and simulations on integer examples. There are no experiments with LLMs or provers.
Critical strengths
Soundness. Theorem 4.1 is verified in Lean using only standard axioms, and the code compiles. I trust the Lean kernel, and the key ingredients are established in the literature, so I regard soundness as solid. Corollary 4.3 and Proposition 5.1 remain paper proofs.
Timeliness. I thank the authors for raising the problem of trivia, which has become a major concern in the community since OpenAI published openai/math.
Critical weaknesses
Contribution. Read a valid but uninteresting output as a false positive and an interesting new result as a true positive. The main result then says that guaranteeing true positives beyond the record requires accepting false positives one for one. This is a classical classifier trade-off, and the paper gives no substantially better bound:
- the half-coverage term is from Kleinberg & Wei (2026);
- the corollaries follow directly;
- with every statement valid (Section 4), “trivia” is just hallucination, as in existing hallucination–coverage work (e.g. Ganju et al., 2026).
Significance and presentation. The paper explains its formal contribution more clearly than its significance; the formal results are routine derivations of the kind current LLMs produce easily. Significance is limited because the generator is given both candidate notions of value, so the hard part of discovery is assumed away. The paper is also very hard to read: coined terms and dense notation bury a simple idea.
Questions for authors
Beyond the familiar point that a system unable to tell valuable from uninteresting results must accept some uninteresting ones to find new valuable ones, what concrete lesson do your results offer people building or evaluating AI systems for mathematics? Specifically, is there a decision they would make differently because of your theorems?
4 comments
Response to Reviewer YhKe (2/3)
W3 (trivia as hallucination). Once the formal world is fixed, trivia is an output outside the target under the smaller reading, which is a hallucination in the sense of Ganju et al. Theorem 3.1(i) is why dropping the verifier inside a formal world loses nothing. In the submission, Sections 1 and 6, Table 4 and Appendix G.3 contrasted the two error notions. The revision removes that contrast in all four places and states the identification; Section 6 now gives the comparison below, and Table 4 summarizes it. The results differ in generators, metric and scope, so they do not conflict:
- Ganju et al. treat randomized generators, timely density with a deadline, and every countable collection. On a constructed nested chain, a supremum hallucination rate of forces timely density 0 (Lemma 1). Under partial enumeration, a vanishing rate attains timely density given a superlinear deadline (Theorem 9).
- Kleinberg & Wei: some eventually valid generator attains density , and on a nested pair no eventually valid generator does better (Theorem 3.3). This is the bounded-trivia value of Corollary 4.2.
- This paper treats deterministic generators, coverage at every round and lower density of the limit set, on one nested pair, in the worst case over complete records of . There, unbounded trivia at any vanishing rate raises the guaranteed density from to (Corollary 4.2), and Theorem 4.1 gives the frontier for every prefix from a round linear in its length on.
The recovery time in Corollary 4.3 has the same inverse form as the scale of Ganju et al. The paper does not claim a general error–time trade-off as new (Appendix G.3); what is new there is the leading constant under a square-root budget. The effect is a property of ambiguous records: a record that leaves refutes the smaller reading, and from then on only the race matters (Appendix C.5).
W2 (corollaries follow directly). Both follow directly from Theorem 4.1, and the paper presents them as corollaries. Corollary 4.2 says which feature of a trivia regime matters for eventual coverage: boundedness, not rate. Its bounded value is Kleinberg & Wei's sharpness, and the value for every unbounded regime is the new half. Corollary 4.3 gives the recovery time, with an exact leading constant under a square-root budget.
W4 (significance; both candidates given; presentation).
Both candidates. The converse says that even full knowledge of both candidate accounts leaves a one-for-one price. It holds in every model that contains and , whatever other candidates the model contains, so a generator that knows less pays at least as much. Attainment says that the price can be paid without ever learning which account is right. The positive results do use known candidates. Proposition C.4 (Appendix C.6) replaces the pair by any known countable family of candidates in a known common ambient set. Its generator outputs every unrecorded valuable point with at most trivial rounds, where is finite but instance-dependent. Its coverage is at least one minus the core's density, without the race share. The exact frontier with more than two nested readings is open (Appendix I.3). Section 4 now states these conditions (paragraph "Scope").
The hard part of discovery. Selection is a separate problem (Appendix I.1): the generator covers the unrecorded region without learning whether it is valuable. The paper prices exploration without value feedback. How much feedback selection needs is open (Appendix I.3).
Proofs. The paper does not claim proof difficulty as a contribution; Section 3 calls the proofs of Theorem 3.1 simple. The contribution is the statements: an exact anytime price, where it applies, and what it implies.
Presentation. The introduction now opens with a plain-language paragraph, and Section 2 has a table that maps the model to AI-for-mathematics practice. In the main text, "ledger", "localizes" and "clock" have given way to "double count", "decides one formal world" and "recovery time". "Fiber" is kept only as the term that Section 2 defines. "Flood" and "harvest" appear only in the title and Figure 1, and the first paragraph defines them. The general response (item 3) quotes the new opening paragraph.
Response to Reviewer YhKe (3/3)
Q1 (a decision made differently). Within the model, yes. These are consequences of the theorems, not tested claims about systems. They bear on whoever sets the tolerance for off-record output, for instance an evaluation that counts outputs outside a curated set of known results as failures.
- Only whether the tolerance keeps growing matters, not how fast. Over complete records of , every bounded tolerance gives the same guaranteed density, . Every unbounded one, however slow, gives (Corollary 4.2). A finite count and a vanishing rate of tolerated off-record outputs therefore fall on opposite sides. A tolerance that does not grow caps what any system can be guaranteed to find beyond the record, however long it runs. At a given horizon, Theorem 4.1 says exactly what the tolerance's value there buys: one unrecorded result per unit.
- The growth rate sets the run length. Consider a tolerance of . Coming within of the best eventual coverage of the first valuable results then takes rounds, up to a factor (Corollary 4.3).
- Where to explore. When candidate accounts of value are available, exploration buys discoveries at rate one only where the accounts disagree. A sweep of all valid statements buys one discovery per three trivia in the running example of Section 4 at .
- What would lower the price. Even a perfect validity verifier changes no guarantee inside a formal world (Theorem 3.1(i)). The price measured here is that of exploring without value feedback. In the two-reading family, a single value label settles which reading holds. How many labels are needed when the record admits many readings is open (Appendix I.3).
Appendix I.2 of the submission states these points briefly; the revised introduction now states all four (paragraph "What the theorems change").
If these answers resolve the concern about the contribution, we would appreciate it if you would reconsider your rating.
Thank you for the response.
Classical trade-off. You concede that the one-for-one count is “a classical two-point argument”; exactness and uniformity add no new idea. The proof of Theorem 4.1(a) applies the cap from Kleinberg & Wei’s racing record to the recorded part of each prefix and the double count to the rest. Attainment interleaves the race with listing the known unrecorded set in order. Since those points are never observed, the two interact only through shared rounds, at a lower-order cost (for sublinear budgets) that you identify as the price of uniformity. The frontier is thus the classical trade-off plus Kleinberg & Wei on disjoint parts, and the corollaries follow directly, as you agree.
Trivia and hallucination. The verifier is idle in the main results (Theorem 3.1(i)), whose proofs, you note, do not use Section 3. With trivia identified as hallucination, Section 4 is therefore partial-enumeration generation under a hallucination budget. The title’s necessity claim, that bounded trivia cannot guarantee more than α/2 (the bounded half of Corollary 4.2), is then, by your own account, Kleinberg & Wei’s sharpness result on a nested pair. More broadly, Ganju et al.’s Lemma 1 already shows, via nested languages, that hallucination is necessary for timely dense generation.
Both responses narrow the contribution rather than distinguish it; I will raise my confidence and I am considering lowering my rating.
Response to Reviewer YhKe (1/3)
Thank you for the review and for compiling the Lean artifact. Theorem 4.1 gives an anytime frontier: one generator, fixed before the prefix length is chosen, secures the race share and the full one-for-one exchange for every prefix, from a round linear in its length on and up to a lower-order term for every sublinear budget, and no generator beats this on its own racing record. The one-for-one count it builds on is a classical two-point argument, and the bounded-trivia value α/2 is Kleinberg & Wei's.
W1 (classical trade-off; no better bound).
What is classical. Eq. (2) is the double count: every output in K′∖K is a true positive under K′ and a false positive under K. The inequality "discoveries beyond the record ≤ trivia under K" follows from it by a two-point argument. In the submission, Section 1 ("the second is new"), Section 6 and Appendix G.2 worded the novelty of the exchange term too broadly, and we have reworded them to claim only its exactness and consequences. The submission credits the term mN/2 and the endpoint α/2 to Kleinberg & Wei (2026) in Section 1 and Appendix G.2. The nested pair is also the device of their sharpness proof (Theorem 3.3 of arXiv:2511.05295v1) and, as a nested chain, of Lemma 1 of Ganju et al. (2026). Section 1 now cites both at that point, Section 6 cites Kleinberg & Wei at the nested pair and Ganju et al. in its next paragraph, and Table 4 lists both.
What Theorem 4.1 adds. In words: from a round linear in N on, a single generator is guaranteed half of the recorded results among the first N valuable ones, plus one unrecorded result per unit of trivia allowed, up to a lower-order term for every sublinear allowance, for every N at once. Formally, take any budget g that grows by at most one per round. One generator Gg is fixed in advance. On every complete record E of K, for every N and every T≥τg(mN) (every T≥2N+2 when g(T)≤T),
OT(Gg,E)∩K′[1…N]≥2mN+min{g(T),nN}−21+g(τg(mN)).
For every generator in the class, one record holds it to mN/2+min{g(T),nN}+log2mN+3 for all T and all N with mN≥2 at once. Uniformity is the point. Gg does not know which prefix it will be judged on, so its sweeps must interrupt the race while prefixes are still being contested. The error term is that cost, and it is lower-order for every sublinear budget. On the converse side we claim the combination of Kleinberg & Wei's racing record with the double count on one record, not either piece.
What follows. Over complete records of K, the bounded-trivia value α/2 is Kleinberg & Wei's sharpness. Corollary 4.2 adds that every unbounded trivia regime, however slow, gives 1−α/2, so only boundedness matters, not rate. Under a square-root budget, Corollary 4.3 pins the recovery time to (1−α−ε)2N2(1+O(N−1/2)).