- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates AI detection in math-heavy scientific writing and constructs a dataset pairing human-written passages with LLM-generated versions. It proposes SNAPJUDGE, an image-based detector that outperforms text-based detectors. The authors also evaluate its generalization to unseen models, different domains, and varying levels of AI editing.
Critical strengths
-
The dataset construction provides a well-controlled experimental design. The paper constructs paired samples of original passages and LLM-regenerated versions, with three input representations, enabling a systematic investigation of how different representations affect AI detection.
-
The dataset construction and experiments are solid. The authors build a large-scale dataset from real scientific papers, covering mathematical proofs and prose, and conduct extensive experiments involving 15 generative models.
Critical weaknesses
-
The specific role of mathematical content and its visual representation in AI detection remains insufficiently explained. The paper emphasizes that mathematical formulas and formatting contain signals useful for identifying AI authorship, but doesn't provide direct features detected by models. Furthermore, the experiments that remove mathematical content or replace it with placeholders also disrupt the semantic integrity and input distribution of the passages.
-
The paper lacks sufficient discussion of data updates and the long-term effectiveness of detectors as generative AI becomes increasingly widespread. The authors acknowledge that recent texts cannot be guaranteed to be entirely human-written and therefore exclude them from dataset construction. I have no objection to this choice itself. However, as generative models and writing practices continue to evolve, detector training and calibration cannot rely indefinitely on a limited pool of historical data. The authors should expand their discussion of this issue.
-
The construction of synthetic data may introduce quality and coherence differences unrelated to authorship. When LLMs regenerate missing proofs or prose passages, they may introduce writing quality issues that make the generated content easier to identify. Section 4.2 only evaluates proof correctness and semantic equivalence to the original passages, which is insufficient to rule out this concern.
Questions for authors
-
Could the authors provide concrete examples of the specific features that the LaTeX detector and SNAPJUDGE rely on?
-
How can reliable human-written samples be collected from scientific papers published after 2022 to support continuous detector updates and false-positive rate calibration? Could the authors at least discuss potential approaches to data collection and verification, particularly how to avoid circular bias when using the detector's own predictions to generate training labels?
3 comments
Thanks for the response. The proposed approaches to collecting human writing after 2022 do not specifically address mathematical research papers, and the 78% positive user feedback provides limited support for reliable long-term updates. This remains an open challenge.
Nevertheless, these concerns do not undermine the core contributions, and my remaining questions have been adequately addressed. I have therefore raised my score.
Thank you very much for reading our response and for raising your score. We truly appreciate the time and thought you have put into this review.
We agree that collecting reliable human-written mathematical research text after 2022 remains an open challenge, and this will be an important direction of our future work.
Response to Reviewer 9G8n
Thank you for the review. We answer each point below.
Re: “How can reliable human-written samples be collected from scientific papers published after 2022”
We agree this is a central challenge. Unless one watches people while they write, it is impossible to guarantee that a text is human-written, which is why detectors, including Pangram 4 (Glickenhaus et al., 2026), are calibrated on pre-2022 text. Reliable new human data therefore requires deliberate collection under known conditions, for example writing under supervision such as exams, or partnerships with writing programs and workshops, such as institutions that offer MFA or similar degrees and take writing seriously. The labels of such data consequently come from how the text was produced rather than relying on assumptions or detector labels. . Detectors can periodically be retrained and recalibrated on these data.
Nevertheless,we publicly hosted our models for the research community to try, and our demo received over 79,000 submissions in four weeks, many of which are for papers from 2026. We additionally solicited (optional) feedback and 78% of the 3026 feedback responses left by users said the predictions matched their expectations. Other users often disagreed with the fraction of AI usage rather than whether AI was used or not. These results, and the initial traction, suggests that detectors perform well on recent papers and offer practical value.
Re: “Could the authors provide concrete examples of the specific features that the LaTeX detector and SNAPJUDGE rely on?”
We identify concrete formatting differences in the AI-generated and human-written proofs. The table gives the share of human and AI proofs that use each formatting style .
\label\begin{equation}display\\\begin{align}aligned,gatheroreqnarray$$display\[...\]display\(...\)inline math$ x $)=without spaces in inline math ($a=b$)$x=1.$)~tiesWe believe that these statistical formatting differences, which are preserved in the LaTeX source, are exploited by detectors, such as SnapJudge. As might be obvious, many of these formatting signals are lost if we operate on rendered text.
Re: “the experiments that remove mathematical content or replace it with placeholders also disrupt the semantic integrity and input distribution of the passages”
Following your suggestion, we retrain the 9B LaTeX detector with every equation replaced by a placeholder in all training, calibration and test data. At 0.1%FPR, the detection of frontier proofs drops from 87.9% to 79.9%, which implies that there are signals in these equations that aid detection. Section 4.2 of our paper ("Detection across mathematical density and proof quality") likewise shows that mathematical content is detectable. On proofs where at least 66% of the LaTeX characters are in math mode, SnapJudge still detects 85.5% of frontier proofs at 0.1% FPR, versus 29% for recalibrated Pangram 4. These results clearly highlight that AI-generated mathematical text remains detectable.
Re: “The construction of synthetic data may introduce quality and coherence differences unrelated to authorship. When LLMs regenerate missing proofs or prose passages, they may introduce writing quality issues that make the generated content easier to identify. Section 4.2 only evaluates proof correctness and semantic equivalence to the original passages, which is insufficient to rule out this concern.”
We agree that the generated passages may differ in quality. At present, however, we cannot efficiently show that the two groups of passages do not differ in quality. We can only proceed by elimination and show that they do not differ noticeably in correctness or semantic meaning. In future work, we will ask human annotators to judge the quality of the passages manually.
References