acceptodds
Under review as a conference paper at ICLR 2027

Multimodal AI Detection for Math-Heavy Scientific Text

Abstract

As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and disclosure around the use of AI in research. Consequently, a veritable subfield of research and a number of commercial services aim to flag AI-generated content automatically. However, existing detectors which rely only on text extracted from PDFs fail to benefit from the rich visual information contained in the formatting of math-heavy scientific content. To study this, we select human-written proof and prose passages from pre-2022 arXiv LaTeX sources and re-generate them using 15 modern LLMs within the context of the same paper. This leaves us with coupled human and AI outputs in three different representations: the LaTeX source, a rendered image of the PDF, and the text extracted from the PDF. We find that the accuracy of the state-of-the-art Pangram 4 drops significantly as the mathematical content in a passage increases, especially when evaluated on text as opposed to LaTeX. However, this is not an inherent limitation of detecting AI-generated math: finetuning Qwen-3.5-27B based detectors on this domain recovers the gap, with LaTeX inputs performing significantly better than text since they preserve more formatting information. Since LaTeX sources are rarely available in practice, we also train SnapJudge: an image-based detector operating on screenshots of both prose and proofs from papers. At 0.1% false positive rate, SnapJudge achieves a detection rate of 92% on proofs and 93% on prose generated by frontier models, substantially outperforming both Pangram 4 and our text-based detector. We will release our data, code and models, as well as a publicly available demo.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI text detection, scientific writing, vision-language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9G8nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates AI detection in math-heavy scientific writing and constructs a dataset pairing human-written passages with LLM-generated versions. It proposes SNAPJUDGE, an image-based detector that outperforms text-based detectors. The authors also evaluate its generalization to unseen models, different domains, and varying levels of AI editing.

Critical strengths
  1. The dataset construction provides a well-controlled experimental design. The paper constructs paired samples of original passages and LLM-regenerated versions, with three input representations, enabling a systematic investigation of how different representations affect AI detection.

  2. The dataset construction and experiments are solid. The authors build a large-scale dataset from real scientific papers, covering mathematical proofs and prose, and conduct extensive experiments involving 15 generative models.

Critical weaknesses
  1. The specific role of mathematical content and its visual representation in AI detection remains insufficiently explained. The paper emphasizes that mathematical formulas and formatting contain signals useful for identifying AI authorship, but doesn't provide direct features detected by models. Furthermore, the experiments that remove mathematical content or replace it with placeholders also disrupt the semantic integrity and input distribution of the passages.

  2. The paper lacks sufficient discussion of data updates and the long-term effectiveness of detectors as generative AI becomes increasingly widespread. The authors acknowledge that recent texts cannot be guaranteed to be entirely human-written and therefore exclude them from dataset construction. I have no objection to this choice itself. However, as generative models and writing practices continue to evolve, detector training and calibration cannot rely indefinitely on a limited pool of historical data. The authors should expand their discussion of this issue.

  3. The construction of synthetic data may introduce quality and coherence differences unrelated to authorship. When LLMs regenerate missing proofs or prose passages, they may introduce writing quality issues that make the generated content easier to identify. Section 4.2 only evaluates proof correctness and semantic equivalence to the original passages, which is insufficient to rule out this concern.

Questions for authors
  1. Could the authors provide concrete examples of the specific features that the LaTeX detector and SNAPJUDGE rely on?

  2. How can reliable human-written samples be collected from scientific papers published after 2022 to support continuous detector updates and false-positive rate calibration? Could the authors at least discuss potential approaches to data collection and verification, particularly how to avoid circular bias when using the detector's own predictions to generate training labels?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9G8n

Thank you for the review. We answer each point below.

Re: “How can reliable human-written samples be collected from scientific papers published after 2022”

We agree this is a central challenge. Unless one watches people while they write, it is impossible to guarantee that a text is human-written, which is why detectors, including Pangram 4 (Glickenhaus et al., 2026), are calibrated on pre-2022 text. Reliable new human data therefore requires deliberate collection under known conditions, for example writing under supervision such as exams, or partnerships with writing programs and workshops, such as institutions that offer MFA or similar degrees and take writing seriously. The labels of such data consequently come from how the text was produced rather than relying on assumptions or detector labels. . Detectors can periodically be retrained and recalibrated on these data.

Nevertheless,we publicly hosted our models for the research community to try, and our demo received over 79,000 submissions in four weeks, many of which are for papers from 2026. We additionally solicited (optional) feedback and 78% of the 3026 feedback responses left by users said the predictions matched their expectations. Other users often disagreed with the fraction of AI usage rather than whether AI was used or not. These results, and the initial traction, suggests that detectors perform well on recent papers and offer practical value.

Re: “Could the authors provide concrete examples of the specific features that the LaTeX detector and SNAPJUDGE rely on?”

We identify concrete formatting differences in the AI-generated and human-written proofs. The table gives the share of human and AI proofs that use each formatting style .

Formatting styleHumanAI
\label30%5%
Numbered \begin{equation} display29%6%
Multi-line display with \\48%19%
\begin{align}32%12%
aligned, gather or eqnarray12%3%
$$ display23%10%
Unnumbered \[...\] display22%41%
\(...\) inline math2%15%
Spaces inside inline math ($ x $)14%2.5%
= without spaces in inline math ($a=b$)61%37%
Period inside inline math ($x=1.$)13%4%
~ ties26%19%
Unicode dashes0.2%5.5%

We believe that these statistical formatting differences, which are preserved in the LaTeX source, are exploited by detectors, such as SnapJudge. As might be obvious, many of these formatting signals are lost if we operate on rendered text.

Re: “the experiments that remove mathematical content or replace it with placeholders also disrupt the semantic integrity and input distribution of the passages”

Following your suggestion, we retrain the 9B LaTeX detector with every equation replaced by a placeholder in all training, calibration and test data. At 0.1%FPR, the detection of frontier proofs drops from 87.9% to 79.9%, which implies that there are signals in these equations that aid detection. Section 4.2 of our paper ("Detection across mathematical density and proof quality") likewise shows that mathematical content is detectable. On proofs where at least 66% of the LaTeX characters are in math mode, SnapJudge still detects 85.5% of frontier proofs at 0.1% FPR, versus 29% for recalibrated Pangram 4. These results clearly highlight that AI-generated mathematical text remains detectable.

Re: “The construction of synthetic data may introduce quality and coherence differences unrelated to authorship. When LLMs regenerate missing proofs or prose passages, they may introduce writing quality issues that make the generated content easier to identify. Section 4.2 only evaluates proof correctness and semantic equivalence to the original passages, which is insufficient to rule out this concern.”

We agree that the generated passages may differ in quality. At present, however, we cannot efficiently show that the two groups of passages do not differ in quality. We can only proceed by elimination and show that they do not differ noticeably in correctness or semantic meaning. In future work, we will ask human annotators to judge the quality of the passages manually.

References

  • Glickenhaus, B., Thai, K., Russell, J., Masrour, E., Han, Y., Spero, M., and Emi, B. (2026). Pangram 4 Technical Report. arXiv:2607.27183.

Thanks for the response. The proposed approaches to collecting human writing after 2022 do not specifically address mathematical research papers, and the 78% positive user feedback provides limited support for reliable long-term updates. This remains an open challenge.

Nevertheless, these concerns do not undermine the core contributions, and my remaining questions have been adequately addressed. I have therefore raised my score.

Thank you very much for reading our response and for raising your score. We truly appreciate the time and thought you have put into this review.

We agree that collecting reliable human-written mathematical research text after 2022 remains an open challenge, and this will be an important direction of our future work.

Reviewer_ofGTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies AI-authorship detection in mathematical proofs and scientific prose, where PDF text extraction can damage mathematical notation. It constructs paired human and LLM-generated passages in LaTeX, extracted-text and rendered-image form, and trains Qwen3.5 classifiers for each representation. SnapJudge classifies the rendered images. On frontier-model proofs, it reports 91.8% TPR at a nominal 0.1% FPR, compared with 87.7% for the trained text detector. The evaluation includes several low-FPR thresholds, held-out generators, external datasets and AI-edited passages. The main contribution is the paired corpus and the empirical study of representations.

Critical strengths
  1. The proof-detection gains are supported by more than the Pangram comparison. The image detector improves over the trained text model across several low-FPR thresholds.
Critical weaknesses
  1. The methodological distinction from existing work is limited.
  • EditLens already uses fine-tuned language models for detection and editing estimation, and concurrent AIdoc studies detection from document images with a vision-language model. The authors acknowledge AIdoc and distinguish their focus on mathematical content, frontier generators and very low FPR. The main contribution is therefore the corpus and the representation study.
  • The external results support transfer, but the evidence for an advance over text detection comes primarily from mathematical proofs. The experimental comparisons need to establish the added value in that setting more convincingly.
  1. The experiments do not isolate the source of the image advantage or sufficiently test recent alternatives.
  • The OCR experiment is useful, but the reported OCR artifacts leave their effect unresolved.
  • DetectAnyLLM (Fu et al., 2025), which is cited, is not evaluated on the target mathematical corpus at a matched FPR.
  • WaveDetect (Liu et al., Findings ACL 2026) is another relevant recent alternative because it learns from token-probability signals and reports low-FPR detection.
  • A justified comparison with a suitable recent detector under comparable data and calibration would strengthen the empirical case.
  1. The reported gain needs clearer validation at the stated false-positive rate.
  • TPR at 0.1% FPR is appropriate, and the threshold sweeps, large calibration pools, external FPR checks and ECE analyses are useful. However, independent in-domain false-positive counts for proofs and prose, together with paired uncertainty for the image–text TPR difference, are not reported. These quantities matter when interpreting a 4.1-percentage-point gain in a low-FPR regime. Pangram’s smaller calibration pool adds uncertainty to that comparison, without establishing that either method was favored.
Questions for authors
  1. Can you justify the selection of recent detectors and provide a matched comparison with a suitable method such as DetectAnyLLM or WaveDetect, with training or adaptation and threshold calibration stated explicitly?

  2. With thresholds fixed beforehand, what are the false-positive counts and rates for proofs and prose on a common independent human test set? What is the paired confidence interval for the frontier-proof TPR difference, accounting for passages from the same paper?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ofGT

Thank you for the review. We answer each point below.

Re: “concurrent AIdoc studies detection from document images with a vision-language model” and “provide a matched comparison with a suitable method such as DetectAnyLLM or WaveDetect”

DetectAnyLLM (Fu et al., 2025) and WaveDetect (Liu et al., 2026) perform poorly on our data. At 0.1% FPR they detect at most 6.6% of frontier proofs with their published weights, and even after retraining on our data with the authors’ code they detect at most 15.8%. They barely work at low FPRs. Pangram 4 is the strongest baseline and is very widely recognized and used, which is why it is the main comparison in our paper.

The image detectors of AIdoc reach an AUROC of only about 0.8, so they are not yet of practical use. In contrast, we publicly hosted our models for the research community to try, and our demo received over 79,000 submissions in four weeks, many of which are for papers from 2026. We additionally solicited (optional) feedback and 78% of the 3026 feedback responses left by users said the predictions matched their expectations. Other users often disagreed with the fraction of AI usage rather than whether AI was used or not. These results, and the initial traction, suggests that detectors perform well on recent papers and offer practical value.

Re: “The experiments do not isolate the source of the image advantage” and “The OCR experiment is useful, but the reported OCR artifacts leave their effect unresolved”

In the paper, we tune hyperparameters for each detector separately. We retrained the 27B text detector on whole passages with exactly SnapJudge’s training settings and instruction, and the gap between the performance of SnapJudge and the text detector widened. Compared with the text detector in the paper, the average lead of SnapJudge grows from 2.8 to 4.1 points. We also trained both detectors under these settings at 9B with three seeds each. In 10,000 bootstrap resamples of seeds and papers, every 95% interval of SnapJudge’s lead lies above zero (e.g. +3.4 [+1.1, +6.0] points on proofs and +0.7 [+0.2, +1.1] on prose at 0.1% FPR), so SnapJudge is significantly better than the text-only detector.

A detector trained on Marker 1.10.2 output performs worse compared to Tesseract in every setting, by 4.7 to 5.3 points over the four FPR ranges (proofs and prose together), and Tesseract in turn is 4.2 to 5.4 points below SnapJudge on the same images. Pangram also reports that OCR output includes formatting that appears to be AI-generated which in turn leads to more false positives (Emi, 2025), so OCR is ill-suited as input for AI text detection.

References

  • Emi, B. (2025). Pangram Predicts 21% of ICLR Reviews are AI-Generated. Pangram Labs blog, 18 November 2025. https://www.pangram.com/blog/pangram-predicts-21-of-iclr-reviews-are-ai-generated.
  • Fu, J., Guo, C.-L., and Li, C. (2025). DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 11229–11238.
  • Liu, Z., Qin, K., He, L., and Xu, Y. (2026). WaveDetect: Robust Framework for Machine-Generated Text Detection via Wavelet Transform. In Findings of the Association for Computational Linguistics: ACL 2026, pages 8712–8727.
AuthorsOpenReview ↗

(cont) Response to Reviewer ofGT

Re: “what are the false-positive counts and rates for proofs and prose on a common independent human test set?”

On the 9,903 human proofs and 12,452 human prose passages of the test set, with thresholds fixed beforehand on the calibration pools, SnapJudge flags 7.5 proofs (0.076%) and 11.0 prose passages (0.088%) at 0.1% FPR, averaged over two seeds, and 0 and 0.5 at 0.005% FPR. The text detector flags 4.5 and 11.0 at 0.1% and 0.5 and 1.5 at 0.005%. The realized false-positive rates match the nominal ones for both detectors.

Re: “What is the paired confidence interval for the frontier-proof TPR difference, accounting for passages from the same paper?”

With a paired bootstrap over papers, the gain of SnapJudge over the text detector on frontier proofs is +4.0 points [+2.9, +5.1] at 0.1% FPR and +3.0 [+1.5, +4.6] at 0.005%. With three 9B seeds per detector under identical settings, the gain over all generators is +3.4 [+1.1, +6.0] and +6.8 [+4.3, +9.0], including the variation between training runs. The gain on proofs is statistically significant.

Re: “Pangram’s smaller calibration pool adds uncertainty to that comparison”

We agree that a threshold calibrated on only about 3,000 human passages is not stable enough, and the actual false-positive rate of the 0.1% threshold can lie between about 0.005% and 0.74%. The significant cost of running Pangram was the main deterrent for us evaluating it on a much larger pool of documents. However, even the upper end of Pangram 4’s bootstrap 95% interval stays below our detectors in every setting except LaTeX prose (e.g. text proofs, at most 81.7% vs 91.8%). Moreover, Pangram 4 reports an official false-positive rate of 0.0041% for its own labels (Glickenhaus et al., 2026), and at the comparable 0.005% false-positive rate, SnapJudge far exceeds Pangram 4, detecting 78.2% vs 32.4% of frontier proofs and 85.6% vs 69.4% of frontier prose. We can therefore still conclude that SnapJudge significantly outperforms Pangram 4.

References

  • Glickenhaus, B., Thai, K., Russell, J., Masrour, E., Han, Y., Spero, M., and Emi, B. (2026). Pangram 4 Technical Report. arXiv:2607.27183.

Dear reviewer, we are approaching the end of the author-reviewer discussion period. We have attempted to address all the concerns pointed out in your review -- if there is any additional information we can provide before the deadline, please let us know!

The retrained baselines, held-out false-positive counts, and paired confidence intervals address my main experimental questions. I find the 4-point gain over the trained text detector on frontier-model proofs convincing.

My concern remains the scope of the contribution. Concurrent work AIdoc also uses a vision-language model for document-image detection. On the four external benchmarks in Table 4, SnapJudge outperforms EditLens-Llama on one, ties on two, and trails on one. I see the paired corpus and the comparison of input representations as the main contributions. I find both useful, but the methodological advance is limited. I’m keeping my weak rejection.

Reviewer_uzNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies AI-generated text detection in mathematical scientific writing. The authors construct approximately 47K proof pairs and 45K prose pairs by regenerating passages from pre-2022 arXiv papers using 15 LLMs, conditioned on the remaining LaTeX source. They train Qwen3.5-27B detectors on LaTeX, PDF-extracted text and rendered images. At 0.1% FPR, the respective detection rates on frontier-model proofs are 93.8%, 87.7% and 91.8%, compared with 65.6% for Pangram 4 on extracted text after recalibration. The paper also examines unseen generators, AI editing and arXiv usage trends, reporting that 17.6% of mathematical proofs were flagged as AI-generated in August 2026.

Critical strengths

The dataset construction is one of the strongest parts of the paper. Each AI passage is generated for the same position in the same paper as the human original. This helps control differences in topic, notation and context. The three representations also make the dataset useful beyond this particular study.

I also appreciate the evaluation at very low false-positive rates. This is important for AI detection, where wrongly flagging human writing can have serious consequences. The results show that AI-generated mathematical proofs can still be detected despite their formal structure. SnapJudge is particularly interesting since it works directly with rendered images, which are easier to obtain than LaTeX sources. The experiments on unseen generators add further value.

Critical weaknesses

My main concern is how well the results generalise to newer scientific writing. The human training and calibration data comes from 2018–2021, but the detector is used to analyse papers from 2026. The false-positive check also relies on these older years. It is therefore unclear whether the same thresholds would work on contemporary human writing. The authors acknowledge possible pretraining contamination, which could also affect the results if older human passages are more familiar to the backbone model.

The comparison between image and text detection is not entirely fair. SnapJudge uses different LoRA settings, learning rates, batch sizes and an instruction prompt. It also sees the whole passage, while the text detector averages predictions from smaller windows. Some of the improvement could come from these differences rather than the image representation. The gap on prose is also quite small, and results from two seeds are reported without confidence intervals.

The Pangram comparison needs more care as well. Its detection rate on frontier proofs using LaTeX increases to 87.5% after recalibration. This suggests that threshold selection and text extraction explain part of the performance gap. However, Pangram's threshold is estimated using only 3,000 human passages, which seems too small for a reliable estimate at 0.1% FPR.

I am also not fully convinced by the analysis of what makes mathematical writing detectable. Removing equations from a detector trained on mathematical inputs creates a distribution shift. The drop in performance does not necessarily show that equations carry authorship signals. Similarly, the correctness analysis relies on an LLM judge whose validation is limited to 30 proofs.

Finally, the editing experiments raise questions about practical use. SnapJudge detects only 0.7% of sentence-level and 7.7% of paragraph-level proof rewrites at 0.01% FPR, but detects 93.7% of outline-based regenerations. This suggests that it mainly detects substantial AI generation rather than smaller edits. The reported 17.6% figure for arXiv should therefore be treated as the proportion flagged by the detector, rather than a direct estimate of AI use.

Questions for authors

Could the authors test the false-positive rate on human-written passages from later years? It would also be useful to check whether detector scores are related to the Qwen3.5 backbone's perplexity. This might help understand the effect of pretraining contamination.

Could the image and text detectors be compared using the same training settings and access to context? I would also like to see the Marker OCR results and confidence intervals across seeds.

How stable is Pangram's recalibrated threshold when it is estimated from only 3,000 human passages? A bootstrap analysis would make this comparison easier to interpret.

For the mathematical content analysis, could the authors retrain the detector on passages where equations have been removed or replaced? This would provide a clearer test of whether the mathematics itself carries useful signals.

In the real-page experiments, do AI-generated passages introduce unresolved citations or references when the paper is recompiled? If so, how does SnapJudge perform when these artefacts are masked?

Could the authors also test newer generators and clarify how the low detection rates on lightly edited passages affect the interpretation of the arXiv trends?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uzNZ

Thank you for the review. We answer each point below.

Re: “Could the authors test the false-positive rate on human-written passages from later years?”

After the release of ChatGPT in November of 2022, it is difficult to establish that a piece of text is human-written. Therefore, many detectors, such as Pangram 4, report false-positive rates on pre-2022 text (Glickenhaus et al., 2026).

Following the suggestion from the reviewer, we evaluate our detector on 88,473 windows of text from 2022 arXiv papers, which corresponds to a time period roughly 2 years after the period from which we source our training human documents. We find that the false positive rate is 0.005% (exactly our target!), we believe that this is perhaps the latest timeframe where the text can still be assumed to be human written.

In 2023, we observe the fraction of text predicted to be AI generated rise from 0.027% in Q1 to 0.109% in Q4.

Recently, we publicly hosted our models for the research community to try, and our demo received over 79,000 submissions in four weeks, many of which are for papers from 2026. We additionally solicited (optional) feedback and 78% of the 3026 feedback responses left by users said the predictions matched their expectations. Other users often disagreed with the fraction of AI usage rather than whether AI was used or not. These results, and the initial traction, suggest that detectors perform well on recent papers and offer practical value.

Re: "It would also be useful to check whether detector scores are related to the Qwen3.5 backbone's perplexity. This might help understand the effect of pretraining contamination."

On pre-2022 human passages, the Spearman correlation between perplexity and the detector score is −0.20 (using LaTeX) and −0.09 (using text), so the detector is in fact more likely to predict familiar text as AI-generated. Moreover, Hans et al. (2024) show that perplexity alone cannot reliably detect machine-generated text, because the prompt strongly alters the perplexity of the generated text and the detector does not have access to the prompt, so detection based on perplexity is highly inaccurate. In any case, such a low-to-moderate correlation is nowhere enough to explain the high detection rates of our detectors at extremely low false-positive rates. We hope that this alleviates reviewers’ concerns about contamination.

Re: “Could the image and text detectors be compared using the same training settings and access to context?”

In the paper, we tuned hyperparameters for each detector separately. We retrained the 27B text detector on whole passages with exactly SnapJudge’s training settings and instruction, and the gap between the performance of SnapJudge and the text detector widened. Compared with the text detector presented in our paper, the average lead of SnapJudge grows from 2.8 to 4.1 points. We also trained both detectors under these settings at 9B with three seeds each. In 10,000 bootstrap resamples of seeds and papers, every 95% interval of SnapJudge’s lead lies above zero (e.g. +3.4 [+1.1, +6.0] points on proofs and +0.7 [+0.2, +1.1] on prose at 0.1% FPR), so SnapJudge is significantly better than the text detector.

References

  • Glickenhaus, B., Thai, K., Russell, J., Masrour, E., Han, Y., Spero, M., and Emi, B. (2026). Pangram 4 Technical Report. arXiv:2607.27183.
  • Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., Geiping, J., and Goldstein, T. (2024). Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text. arXiv:2401.12070.
AuthorsOpenReview ↗

(cont) Response to Reviewer uzNZ

Re: I would also like to see the Marker OCR results

A detector trained on Marker 1.10.2 output performs worse compared to Tesseract in every setting, by 4.7 to 5.3 points over the four FPR ranges (proofs and prose together), and Tesseract in turn is 4.2 to 5.4 points below SnapJudge on the same images. Pangram also reports that OCR output includes formatting that appears to be AI-generated which in turn leads to more false positives (Emi, 2025), so OCR is ill-suited as input for AI text detection.

Re: “How stable is Pangram’s recalibrated threshold when it is estimated from only 3,000 human passages?”

We agree that a threshold calibrated on only about 3,000 human passages is not stable enough, and the actual false-positive rate of the 0.1% threshold can lie between about 0.005% and 0.74%. The significant cost of running Pangram was the main deterrent for us evaluating it on a much larger pool of documents. However, even the upper end of Pangram 4’s bootstrap 95% interval stays below our detectors in every setting except LaTeX prose (e.g., for text proofs, Pangram’s performance is at most 81.7% compared to SnapJudge’s 91.8%). Moreover, Pangram 4 reports an official false-positive rate of 0.0041% for its own labels (Glickenhaus et al., 2026), and at the comparable 0.005% false-positive rate, SnapJudge far exceeds Pangram 4’s performance, detecting 78.2% vs 32.4% of frontier proofs and 85.6% vs 69.4% of frontier prose. We can therefore still conclude that SnapJudge significantly outperforms Pangram 4.

Re: “could the authors retrain the detector on passages where equations have been removed or replaced?”

Following your suggestion, we retrain the 9B LaTeX detector with every equation replaced by a placeholder in all training, calibration and test data. At 0.1%FPR, the detection of frontier proofs drops from 87.9% to 79.9%, which implies that there are signals in these equations that aid detection. Section 4.2 of our paper ("Detection across mathematical density and proof quality") likewise shows that mathematical content is detectable. On proofs where at least 66% of the LaTeX characters are in math mode, SnapJudge still detects 85.5% of frontier proofs at 0.1% FPR, versus 29% for recalibrated Pangram 4. These results clearly highlight that AI-generated mathematical text remains detectable.

Re: “do AI-generated passages introduce unresolved citations or references when the paper is recompiled?”

Rarely. Unresolved citations or cross-references appear in 1.69% of AI passages and 0.48% of human passages, so they could explain at most about two points of detection rates above 90%. Given such low prevalence of unresolved citations, the performance of detectors cannot be attributed to it.

Re: “Could the authors also test newer generators”

We have added the detailed results to the Appendix N of the paper. While the observed detection rates are lower for outputs from recent models, We show that,, adding only a small amount of training data improves the performance substantially. Pangram 4, in contrast, performs poorly on outputs from the newer models.

References

Dear reviewer, we are approaching the end of the author-reviewer discussion period. We have attempted to address all the concerns pointed out in your review -- if there is any additional information we can provide before the deadline, please let us know!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the Area Chair

Dear Area Chair,

Thank you for handling our submission. We have carefully read each reviewer’s comments and attempted to address their concerns. Below, we list the main points and our responses.

  • False-positive rate in later years. On 88,473 windows of text from 2022 arXiv papers, roughly 2 years after the period from which we source our training human documents, the false positive rate is 0.005% (exactly our target!).
  • Detector performance in practice. Recently, we publicly hosted our models for the research community to try, and our demo received over 93,000 submissions in the first month, many of which are for papers from 2026. Of the 3488 feedback responses, 78% said the predictions matched their expectations. These results, and the initial traction, suggest that detectors perform well on recent papers and offer practical value.
  • Pretraining contamination and perplexity. The Spearman correlation between perplexity and the detector score is −0.20 (using LaTeX) and −0.09 (using text), so the detector is in fact more likely to predict familiar text as AI-generated. Such a low-to-moderate correlation is nowhere near enough to explain the high detection rates of our detectors at extremely low false-positive rates.
  • Source of the image advantage over text. We retrained the 27B text detector on whole passages with exactly SnapJudge’s training settings and instruction, and the average lead of SnapJudge grows from 2.8 to 4.1 points. We also trained both detectors under these settings at 9B with three seeds each, and every bootstrap 95% interval of SnapJudge’s lead lies above zero.
  • Stability of Pangram 4’s threshold. At 0.1% FPR, even the upper end of Pangram 4’s bootstrap 95% interval stays below our detectors in every setting except LaTeX prose (e.g. text proofs, at most 81.7% vs SnapJudge’s 91.8%).
  • Role of the mathematical content. We retrain the 9B LaTeX detector with every equation replaced by a placeholder in all training, calibration and test data. At 0.1% FPR, the detection of frontier proofs drops from 87.9% to 79.9%, which implies that there are signals in these equations that aid detection.
  • Recent baselines. At 0.1% FPR, DetectAnyLLM and WaveDetect detect at most 6.6% of frontier proofs with their published weights, and even after retraining on our data with the authors’ code they detect at most 15.8%, compared to SnapJudge’s 91.8%.
  • Collecting human-written data after 2022. Reliable new human data requires deliberate collection under known conditions, for example writing under supervision such as exams, or partnerships with writing programs and workshops. The labels of such data consequently come from how the text was produced rather than relying on assumptions or detector labels.

Thank you for reading!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.