acceptodds
Withdrawn from ICLR 2027

Beyond Selected Breakthroughs: Evaluating Language Models on Randomly Sampled Mathematical Conjectures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The last year has seen a shift in AI-assisted mathematics, with frontier language models producing proofs and counterexamples for long-standing open problems. However, curated benchmarks and selected breakthroughs provide an incomplete picture of LLMs performance and the computational resources and problem-specific mathematical guidance needed to achieve reported results. To address this, we introduce a new dataset, OpenConjecture, which contains most conjectures from math papers published on the arXiv since February 2026. We apply frontier LLMs to a sample of 200 of these using a reproducible protocol without problem-specific mathematical guidance. Taken together, the various models produce candidate solutions for roughly 60% of the conjectures, with counterexamples more common than proofs. We then perform a qualitative analysis of solution characteristics such as novelty and connections to existing literature. Our analysis distinguishes proposed conjecture resolutions that expose missing assumptions or recover existing results from those that may contribute new mathematics. Finally, we document the prompts and computational resources required, and release the model responses to support further study.

Primary area:
datasets and benchmarks
Keywords:
mathematical reasoning, large language models, AI-assisted mathematics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2F5xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper highlight a very interesting problem on ai for math community. beyond cherry-picked breakthroughs, how llms perform on in-the-wild uncurated problems. and introduce OpenConjecture (1.9k problems from recent arxiv). by evaluating 200 samples without specific guidance on GPT family models, it analysis provides several insights such as most proposed counterexamples merely use missing assumptions or standard constructions rather than offering genuine math contributions.

Critical strengths

I like that this paper steps away from cherry-picked breakthrough problems in math communicty. instead, looking at uncurated, in-the-wild conjectures directly from recent arXiv papers asks a more practical question: how do frontier llms actually perform on everyday math research?

the paper looks deeper into what the llms are actually doing. it separates technical correctness from geniuine math contribution, eg showing that most counterexamples just exploit missing draft assumptions or recycle standard textbook tricks. this is very timely, especially given recent claims like open about AI math breakthroughs.

the 3-pass setup is a practical touch by trying cheaper models first and only switch to stronger models when stuck. however, the author need to provide clarification how's different level llm math ability are measured; espeically consider move this setup to cross-family or open-sourced model.

Critical weaknesses

however, major concerns lie on the fairness and coverage of experiments. the entire study is locked into one model family (openAI's GPT). without testing other frontier reasoning models (like Claude or open-source reasoners like qwen), it’s hard to tell whether behaviors like are generalize to other llms. also, while 200 conjectures sounds decent, spreading them across the whole of math means many subfields only have 1-2 data points.

considering correctness is judged by another instance of the exact same llm, the verification passrate feels like a circular argument. Without even a small golden sample verified, we can't be sure how many solutions are actually valid. because of these issues, although the problem is interesting, I feel this work is a better fit for a workshop than the main conference track.

bad presentation. unpolished presentation and figures. the figures feel quite rough and hastily put together. such as fig 1, the sizing and layout is awkward, and the math category codes (CO, NT, etc.)

Questions for authors

pls refer to weakness and strength

in appendix A, there are raw code-like labels such as real_open_conjecture directly into the text, which look like unformatted artifacts from a coding agent. additional, table 1 in appendix lists a long row of raw paper counts; a better illustration will be hist plot, I hope author carefully polish the paper by human before submission.

1 comment

Thank you for your careful reading of our paper and the constructive suggestions.

Reviewer_mYwAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces OpenConjecture, a dataset of conjectures sampled from recent arXiv mathematical preprints and were updated weekly. The authors study 200 sampled conjectures using a three-stage pipeline involving GPT-5.6 and GPT-6 models, without providing problem-specific mathematical guidance. They assess proposed proofs and counterexamples through LLM review. They proposed the evalutions on the output of the LLMs by correctness, mathematical novelty, fidelity to the intended conjecture, connections to existing literature, and computational cost. The study aims to characterize the mathematical contributions and limitations of model-generated responses to conjectures drawn from contemporary research.

Critical strengths
  1. An up-to-date dataset of mathematical conjectures. Collecting conjectures from recent preprints and updating the dataset weekly provides a useful resource for studying mathematical reasoning on contemporary problems.

  2. An informative, multidimensional evaluation framework. The paper distinguishes apparent correctness from mathematical novelty, fidelity to the intended statement, and dependence on existing works. Also, recording computational costs helps characterize the practical experience of using LLMs.

Critical weaknesses
  1. Insufficiently validated evaluation reliability.
  • Each proposed resolution is assessed by a fresh instance of the same model that generated it, creating a risk of correlated mathematical errors.
  • Although the authors perform limited manual review and manually classify some relationships to existing literature, they do not provide specialist verification or formal validation. Nor do they report evaluator accuracy or false-positive rates against expert judgments. Consequently, the reliability of the correctness assessments and associated qualitative findings remains uncertain. Expert validation of a representative subset would materially strengthen the evidence.
  1. Limited task characterization and generalizability of the empirical contribution.
  • Recent arXiv conjectures vary substantially in difficulty, precision, and unresolved status, which would impact the evalution results.
  • The restricted model family and sample size limit what the study establishes about broader LLM mathematical capabilities. The contribution remains primarily descriptive, and stronger task validation and controlled analyses are needed to demonstrate transferable insights of sufficient significance.
Questions for authors
  1. What evidence supports the reliability of the same-model correctness and novelty assessments? Please clarify the extent of manual review and whether any independent checks allow the evaluators’ error rates to be estimated.

  2. Which findings provide transferable insights beyond the sampled conjectures and the particular GPT pipeline? In particular, how should these observations inform the design or evaluation of mathematical reasoning systems, given the limited model coverage and lack of a validated difficulty classification?

1 comment

Thank you for the time you spent reviewing our paper. We appreciate your comments and thoughtful questions.

Reviewer_Yyx9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces OpenConjecture, a weekly-updated dataset of conjecture environments extracted from arXiv papers. They extract 200 uniformly sampled conjectures, and run a 3-stage pipeline with OpenAI models of increasing capability to attempt to resolve the conjecture, where only unresolved ones are passed to the stronger model. They then use this data to assert counterexample novelty, literature relation, and provability of the conjectures.

Critical strengths
  1. The message that LLM performance in mathematics should not be cherry-picked on successes, but instead be evaluated across a wide range of conjectures seems valuable.
Critical weaknesses

Overall, this paper reads more like an exploratory case or a position paper. The methodology is not polished, and some design choices invalidate the claims or leave them up to interpretation.

  1. Evaluation methodology and conclusions

    • (Critical) While LLM judges are common and generally sensible, the authors evaluated each response with the same model. This carries multiple risks. For one, the same model is less likely to spot mistakes in its own generations [1]. Even if all instances were evaluated with the strongest model, given all were developed by OpenAI, it still carries model-family bias risks [2]. Even worse, any rejections are not really isolated beyond a "by the way, here's how many proofs are judged wrong", citing a "production of a proof" value in the abstract, not a fraction that reflects correctness. Finally, even if one to resolve the judging biases, the rejected proofs from simpler models are never retried with stronger ones, leaving open what is withing reach for AI models to tackle.
    • (Major) The counterexample results, as the authors suggest, concede that models are likely to exploit missing asusmptions or mistakes in the statement. This can be genuine imprecision from the original paper's formulation, but also might be an issue with the porting of the task from the papers, which would constitute a significant problem with the dataset design.
    • (Major) The proof-to-literature labels are manual but explicitly "non-specialist", which makes their value, relevance, or accuracy unclear.
    • (Major--medium) The paper makes no attempt to evaluate whether the resolutions are actually impactful to the meaningful fields.
  2. Running models -> I have several issues with how the research was conducted with respect to the model inference.

    • (Major) Inference is inconsistent: GPT-5.6-Sol was provided web access, while the rest of the models were not. This makes any conclusions on literature review and citations (a major contribution according to the discussion section), only limited to the weakest model's first pass. If I am not mistaken, the authors also change the prompt, and the input package.
    • (Medium) Despite wanting to claim tackling realistic applications of LLMs, all the calls were one-shots and not agentic instances, which would actually enable claims of realistic use and genuine comparison of literature review.
    • (Medium) The authors evaluating a single model family, even if its among the strongest ones. In order to cover what can be solved by AI systems, one should at least consider Anthropic models as viable alternatives.
  3. Problem sampling

    • (Major) While not cherry-picking results is important, uniform sampling has a separate problem. Given that some conjectures might be crucial open problems, while others might be niche hypotheses with narrow impact, the former should not weigh the same as the latter in terms of consideration.
    • (Medium--Major) If the authors are sampling papers from February onward, it makes the likelihood of data contamination non-trivial, which is especially important when judging novelty. It is thus also not surprising that a lot of proofs were actually resolved in later preprints, as the authors themselves find in 5.2.
    • (Medium) It nevertheless remains unclear whether any quality filtering was performed to filter out low-quality research.

References

[1] Krumdick, Michael, et al. "No free labels: Limitations of llm-as-a-judge without human grounding." arXiv preprint arXiv:2503.05061 (2025).

[2] Dekoninck, Jasper, et al. "The open proof corpus: A large-scale study of llm-generated mathematical proofs." International Conference on Learning Representations. Vol. 2026. 2026.

Edit (06/11): Added the citations which I had previously missed, sincere apologies to the authors.

Questions for authors
  1. Are the authors planning to perform any form of expert verification with people in the respective fields for at least the most impactful claims?

  2. Can you address the concerns from the "Weaknesses" section?

1 comment

Thank you for the time you spent reading our paper and the thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.