Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers

Through an analysis of more than 3 million arXiv papers, we identify several previously overlooked shifts in word usage that should be driven by large language models (LLMs), such as the increased frequency of "beyond" and "via" in titles and the decreased frequency of "the" and "of" in abstracts. Because of the similarities among different LLMs, experiments show that current classifiers struggle to accurately determine which specific model generated a given text in multi-class classification tasks. By adopting a direct and highly interpretable linear approach and accounting for differences between models and prompts, we characterize lexical patterns potentially associated with LLM influence and show that real-world LLM usage is heterogeneous and dynamic. Meanwhile, the continuous evolution of LLMs also poses greater challenges for detection and estimation.