- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper analyzes and estimates the impact of large language models on academic papers. Specifically, it examines word-frequency changes in titles and abstracts across more than 3 million arXiv papers. It also simulates academic writing with multiple LLMs from different families and release periods, and uses a direct, interpretable linear approach to estimate their relative stylistic alignment with real papers. The results identify previously overlooked shifts in both distinctive and common words, and show that newer models are becoming more human-like and more similar to each other, making specific-model attribution difficult. Overall, real-world LLM influence appears heterogeneous and dynamic, while simple lexical estimation remains informative where complex classifiers are unreliable.
Critical strengths
- Comprehensive experimental coverage. The paper examines the impact of LLMs from multiple perspectives, combining a large-scale analysis of more than 3 million real papers with cross-domain validation, simulations across a wide range of models and prompts, and complementary analyses of lexical trends, text similarity, classification, and impact estimation. Additionally, it compares both older and newer models over time, making the evaluation relatively complete in both scale and scope.
Critical weaknesses
- Limited novelty and significance of the contribution to the community. The paper focuses on shifts in word usage in academic titles and abstracts and explicitly does not attempt causal attribution of LLM usage, while its analysis largely extends an established line of work that monitors indicative words to estimate LLM influence on academic writing. Its results are also mainly descriptive and trivial, confirming that such influence is growing and heterogeneous, that newer models are becoming more human-like, and that specific-model classification is difficult, without providing a reliable estimate of actual usage or a usable detection method. This raises the concern that both the topic and the findings have limited importance, as they do not substantially change how the community understands or handles the impact of LLMs on academic writing.
Questions for authors
Given the weakness above, I would appreciate it if the authors could provide justifications of novelty contribution to the community.
Thank you for your comments on the contribution and significance of the paper.
Q1. What is the value of the findings without a reliable estimate of actual LLM use or a usable detection method?
A1. The paper already compares classifiers trained on simulated texts and applied to real arXiv abstracts. The main comparison uses LLM2Vec, and the appendix reports other classifiers. About 20% of the pre-ChatGPT abstracts receive an LLM label in that comparison. This historical label share needs to be accounted for before the predictions can be interpreted as usage rates.
The controlled mixtures test the frequency estimator separately. They measure how estimates change when reference models are omitted and how much false weight remains in human-only subject controls. We have revised the Discussion and Conclusion to include these results and retained the classifier comparison. Because actual usage in the real corpus is unknown, that comparison cannot establish which method estimates it more accurately.
Q2. What is the paper’s contribution to the community?
A2. We combine a large corpus of academic papers with paired outputs from different LLMs and prompts. This lets us compare how models change the same source texts and examine related patterns in real papers. We also study changes across model generations, which matter because references based on older models may become less representative over time. Our word-frequency method provides a simple, interpretable way to examine these differences. Controlled mixtures test how missing reference models and changes in subject composition affect the estimates, and whether weighting and subject-area adjustment reduce the errors.