- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes FAR (Find, Attempt, Recommend), a cascade that takes a research direction where i) an LLM labels papers in a corpus, extracts unresolved statements, checks with web search whether they are still open, ii) a stronger model attempts each one once, three samples of the same model judge the claimed resolutions, and iii) a grader sorts the survivors by novelty and significance for human review. A combinatorics pilot (51,110 papers, gpt-oss-120b, Gemini 3.5 Flash, Gemini 3.1 Pro, GPT-5.5 at xhigh) yields 4,717 "open" conjectures, 1,050 claimed resolutions, 598 judged correct and 77 graded publishable. The authors hand-check 15 chosen by interest. A second part frames which conjectures to attempt as budgeted selection, validates LLM difficulty and importance scores against the run's outcomes, and compares ranking strategies by 5-fold cross validation. Appendix C writes up the 15 results.
Critical strengths
- The pipeline produced real mathematics. I checked the arguments in C.2, C.5, C.8, C.10, C.13 and C.14 in detail and spot-checked C.3 (the m = 1 values 81 and 200 are right), C.9 and C.15. I found no errors. I also confirmed that Conjecture 2 of Davies et al. (2018), Conjecture 1.4 of Lund et al. (2018) and Conjecture 5.2 of Ikenmeyer et al. (2024) are stated exactly as the paper quotes them, so the three flagship results address what they claim to address. Section 4.4 is candid that one reviewed result was already known, one is a classical construction, and C.13 was published independently by two groups.
- The funnel is reported stage by stage, the prompts are given in full, and the paper documents its own literature-check failure (Erdős #635). The writing is clear.
Critical weaknesses
1. The cascade lacks a representative audit and a matched comparison with existing problem pools.
Only 15 of the 77 recommended artifacts were reviewed, "chosen by our own interest" (Section 4.4), so 15/15 is not an estimate of precision for the 77, let alone for the 598 that passed judging. No reviews from the 452 FAIL or 2,905 NONE outcomes were reported, so the false-negative rate is unknown.
The pool is also softer than the headline number. The attempt stage reclassified 443 + 319 = 762 of the 4,717 "apparently well-posed and still-open" conjectures (16%) as known or defective, grading removed 75 more as known, and Figure 3's caption mentions a "later status recheck" that reclassified further candidates without saying how many.
The model that wrote the proofs also judges and grades them, and the "three independent judges" (Section 4.1) are three runs of that model. The paper gives no agreement statistics and uses no second model family.
Finally, for a paper whose premise is that frontier reasoning is a scarce resource, no token, dollar or wall-clock cost is reported for any stage, and there is no comparison with the obvious alternative of pointing the same solver at an existing pool. Feng et al. (2026), which the paper cites, already swept 700 Erdős problems with a review stage. Formal Conjectures (Firsching et al., 2026) formalizes 1,029 open conjectures in Lean, and pipeline-math (Peng et al., 2026) attempts published open-problem lists. The paper cites all three but does not say what literature mining buys over them in yield per attempt or per dollar. The appendix shows the cascade can produce correct new mathematics. How often it does so, at what cost, and whether mining beats an existing pool are not measured, and without those the abstract's "demonstrate the effectiveness" is not well supported.
2. The allocation analysis is thin as it relies on automated outcome labels and retrospective strategy selection.
The bandit framing is decorative. Each arm is pulled once (Section 2), so the problem is supervised ranking, and Appendix B.1 is linearity of expectation plus the textbook (1−1/e) greedy bound.
Score validation is LLM against LLM, with Gemini 3.1 Pro's scores tested against outcomes GPT-5.5 produced. The association is real but overstated. Section 4.3.1 reports AUC 0.69 for difficulty and 0.60 for importance and calls both "strong positive correlation" on the strength of Mann-Whitney p-values. A small p-value only shows that an association exists, and at these sample sizes a weak one is enough to produce it. The AUC is the size of the effect. By the paper's own definition, 0.60 means the importance score ranks a random publishable resolution above a random non-publishable one 60% of the time, against 50% for a coin flip. The stratified AUC of 0.56, also in Section 4.3.1, is said to show that difficulty "still effectively separates" solved from unsolved attempts within an importance stratum. It shows a modest conditional effect, since about two thirds of the excess over 0.5 disappears once importance is held fixed. The scores carry signal, as the gains below show, but not a strong one.
The recommendation, "discard all but the most important part of P and rank what remains", was chosen after seeing the outcomes ("of the fractions we tried, keeping the most important 1/10 did best on this run"). Tables 1 to 3 suggest how little it rests on. Inside the top decile the strategy plateaus at 2.00 expected artifacts, 1.70 expected total importance and 0.850 expected max importance from B = 200 onward, which reads most naturally as two artifacts of importance about 0.85. If so, the conclusion depends only on two conjectures.
On , the paper notes that ranking on beats the prescribed at every budget and leaves it there. Since is optimal for under the true , the reversal most likely comes from error in or from noise in a single run's outcomes. The simplest explanation could be that overestimates success on important conjectures. They are rarely solved (the top decile by importance yields 2.00 artifacts at B = 300 where uniform random yields 5.17, Table 1), and the ranking avoids them (the max importance among the artifacts it returns averages 0.50 at B = 300, against 0.85 for the top-decile strategy, Table 3), so weighting by pushes the selection back toward conjectures that fail. A plot of against realized outcomes by importance bin would show whether this is right. "Narrowly" also undersells the gap at small budgets. At B = 10, Table 2 gives 0.09 for against 0.16 for and 0.08 for uniform random, so importance weighting gives up most of the gain over random.
What the section does establish is that ranking on raises the expected artifact count from 5.17 to 7.92 at B = 300 (Table 1) and the expected total importance from 2.51 to 3.87 (Table 2), about 1.5x uniform random in both cases. However, this is a result on one run, with outcome labels that GPT-5.5 produced and strategies compared after the fact, and the paper itself notes that it depends on the scoring model, the pool and the cascade (Section 4.3.2). Whether it transfers to another pool or solver is untested.
Lastly, the paper does not discuss directly relevant work on predicting difficulty to allocate inference compute. For example,
Snell, Lee, Xu, Kumar (2024). Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv:2408.03314.
Damani, Shenfeld, Peng, Bobu, Andreas (2024). Learning how hard to think: input-adaptive allocation of LM computation. arXiv:2410.04707.
Questions for authors
-
Could you verify a uniformly random sample of the 598 PASS outcomes and of the 77 artifacts for correctness and novelty, and report the novelty status of all 77? How many of the 77 are already in the literature?
-
Could you report cost per stage (tokens or dollars, and wall-clock) and cost per correct new artifact? Would it be possible to run the same attempt, judge and grade stack on an existing pool of comparable size, e.g. Erdős problems, Formal Conjectures, Open Problem Garden, and compare yield per attempt and per dollar with the mined pool?
-
Could you give the distribution of judge votes over the 1,050 claims (3/3, 2/3, ...) and the PASS rate when one judge is replaced by a different model family? How often do the three GPT-5.5 judges disagree?
3 comments
Thank you again for your time and feedback. We would like to kindly check whether our responses have addressed your concerns. If anything remains unclear, we would be happy to clarify further, and we would be grateful if you would consider updating your score.
Thanks for the reply, and for agreeing to tone down the wording on the scores and add Snell et al. and Damani et al.
I'm keeping my score, because the 15 results only show FAR can discover real math but not how often its picks are correct and new.
Releasing everything later is great, but some of it you could report now, like the judge vote split over the 1,050 claims and whatever per-stage cost numbers you logged. One attempt per conjecture doesn't tell us what the pipeline cost. Also, how many of the 77 artifacts fall in the top tenth by importance? That would show how many conjectures the result rests on.
I get that checking all 77 is out of scope. My ask for a correctness check was a random sample. At minimum, the abstract should say 77 items were recommended and you checked 15, chosen by interest.
We thank the reviewer for the very careful reading. We respond to the weaknesses and questions below.
W1. Audit, pool quality, judging, and cost.
gpt-5.5attempt (pass@1), for demonstration only, and users can replace it, or the judges, with stronger agent systems of their own.W2. Allocation analysis.
Q1. Random-sample audit. We did not run one, for the reasons in W1. All 77 artifacts will be released with the grader's novelty reports.
Q2. Cost and comparison with an existing pool. See W1. Each conjecture received one
gpt-5.5xhighagent run, which is the unit of budget in our demonstrative run and our analysis.Q3. Judge votes. We did not use a second model family, for the reasons in W1. All judge verdicts on the 1,050 claims will be released. The mathematician makes the final check, and every stage is customizable. When we ran the pilot in April–May 2026, in our experience no other model available to us came close to
gpt-5.5on research-level mathematics, so we used it throughout and kept each stage minimal for demonstration. Users can add judges from other model families.We thank the reviewer again for the constructive comments. We hope these answers address the reviewer's concerns, and that the reviewer will reconsider the evaluation.