- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
this paper highlight a very interesting problem on ai for math community. beyond cherry-picked breakthroughs, how llms perform on in-the-wild uncurated problems. and introduce OpenConjecture (1.9k problems from recent arxiv). by evaluating 200 samples without specific guidance on GPT family models, it analysis provides several insights such as most proposed counterexamples merely use missing assumptions or standard constructions rather than offering genuine math contributions.
Critical strengths
I like that this paper steps away from cherry-picked breakthrough problems in math communicty. instead, looking at uncurated, in-the-wild conjectures directly from recent arXiv papers asks a more practical question: how do frontier llms actually perform on everyday math research?
the paper looks deeper into what the llms are actually doing. it separates technical correctness from geniuine math contribution, eg showing that most counterexamples just exploit missing draft assumptions or recycle standard textbook tricks. this is very timely, especially given recent claims like open about AI math breakthroughs.
the 3-pass setup is a practical touch by trying cheaper models first and only switch to stronger models when stuck. however, the author need to provide clarification how's different level llm math ability are measured; espeically consider move this setup to cross-family or open-sourced model.
Critical weaknesses
however, major concerns lie on the fairness and coverage of experiments. the entire study is locked into one model family (openAI's GPT). without testing other frontier reasoning models (like Claude or open-source reasoners like qwen), it’s hard to tell whether behaviors like are generalize to other llms. also, while 200 conjectures sounds decent, spreading them across the whole of math means many subfields only have 1-2 data points.
considering correctness is judged by another instance of the exact same llm, the verification passrate feels like a circular argument. Without even a small golden sample verified, we can't be sure how many solutions are actually valid. because of these issues, although the problem is interesting, I feel this work is a better fit for a workshop than the main conference track.
bad presentation. unpolished presentation and figures. the figures feel quite rough and hastily put together. such as fig 1, the sizing and layout is awkward, and the math category codes (CO, NT, etc.)
Questions for authors
pls refer to weakness and strength
in appendix A, there are raw code-like labels such as real_open_conjecture directly into the text, which look like unformatted artifacts from a coding agent. additional, table 1 in appendix lists a long row of raw paper counts; a better illustration will be hist plot,
I hope author carefully polish the paper by human before submission.
Thank you for your careful reading of our paper and the constructive suggestions.