- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper identifies duplicated math problems generated by LLMs. More specifically, it may be possible to detect the LLM procedure and solve the pattern rather than solving the actual problem. The authors call this phenomenon "recipe-matching". MathNet-Retrieve is used as a benchmark because a three-tier system is used to disguise gold documents given a corpus and a query. The authors train an embedding model based on Qwen3 on LLM-generated pairs against a symbolically verified SymPy baseline. The proposed method achieved ~45R@1 advantage on the easy tier and 9.42 R@1 on the hard tier. Similar trends are observed on the Olympiad dataset.
This paper is extremely hard to read and understand, and it has 54 pages of Appendix. Given that I already spent a significant amount of time trying to understand it and I still don't, I decided I will not spend more time on it as long as the manuscript hasn't been entirely revised. I submit this review after having discussed with the ACs.
Critical strengths
N/A
Critical weaknesses
Poorly written, hard to understand.
Questions for authors
Please review the entire manuscript.
2 comments
A brief follow-up: the paper does not propose a retriever. The 45-point gain is the flaw we expose. It comes from matching the benchmark’s LLM writing style, and it nearly vanishes on real duplicates. The revised Section 1 (in red) explains this in plain language, and we would be grateful if you would take a look and consider revisiting your assessment.
Thank you for your time, and we're truly sorry the paper was hard to read.
What the paper shows, in plain terms. We do not propose a retrieval method. We audit a benchmark. MathNet-Retrieve writes its own test documents with an LLM under one fixed prompt. We show that a retriever trained on pairs written the same way gains about 45 points on the benchmark but under 10 on real duplicates written by people, and none on same-language reprints. The 45 points are the symptom we study, not a contribution.
What we changed for readability:
We would be grateful if you could look at the revised Section 1, and we are looking forward to your feedback.