acceptodds
Under review as a conference paper at ICLR 2027

Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers

Abstract

Through an analysis of more than 3 million arXiv papers, we identify several previously overlooked shifts in word usage that should be driven by large language models (LLMs), such as the increased frequency of "beyond" and "via" in titles and the decreased frequency of "the" and "of" in abstracts. Because of the similarities among different LLMs, experiments show that current classifiers struggle to accurately determine which specific model generated a given text in multi-class classification tasks. By adopting a direct and highly interpretable linear approach and accounting for differences between models and prompts, we characterize lexical patterns potentially associated with LLM influence and show that real-world LLM usage is heterogeneous and dynamic. Meanwhile, the continuous evolution of LLMs also poses greater challenges for detection and estimation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM-Generated Text, AI-Generated Content, Academic Writing, Science of Science

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_b4PqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper analyzes and estimates the impact of large language models on academic papers. Specifically, it examines word-frequency changes in titles and abstracts across more than 3 million arXiv papers. It also simulates academic writing with multiple LLMs from different families and release periods, and uses a direct, interpretable linear approach to estimate their relative stylistic alignment with real papers. The results identify previously overlooked shifts in both distinctive and common words, and show that newer models are becoming more human-like and more similar to each other, making specific-model attribution difficult. Overall, real-world LLM influence appears heterogeneous and dynamic, while simple lexical estimation remains informative where complex classifiers are unreliable.

Critical strengths
  • Comprehensive experimental coverage. The paper examines the impact of LLMs from multiple perspectives, combining a large-scale analysis of more than 3 million real papers with cross-domain validation, simulations across a wide range of models and prompts, and complementary analyses of lexical trends, text similarity, classification, and impact estimation. Additionally, it compares both older and newer models over time, making the evaluation relatively complete in both scale and scope.
Critical weaknesses
  • Limited novelty and significance of the contribution to the community. The paper focuses on shifts in word usage in academic titles and abstracts and explicitly does not attempt causal attribution of LLM usage, while its analysis largely extends an established line of work that monitors indicative words to estimate LLM influence on academic writing. Its results are also mainly descriptive and trivial, confirming that such influence is growing and heterogeneous, that newer models are becoming more human-like, and that specific-model classification is difficult, without providing a reliable estimate of actual usage or a usable detection method. This raises the concern that both the topic and the findings have limited importance, as they do not substantially change how the community understands or handles the impact of LLMs on academic writing.
Questions for authors

Given the weakness above, I would appreciate it if the authors could provide justifications of novelty contribution to the community.

1 comment

Thank you for your comments on the contribution and significance of the paper.

Q1. What is the value of the findings without a reliable estimate of actual LLM use or a usable detection method?

A1. The paper already compares classifiers trained on simulated texts and applied to real arXiv abstracts. The main comparison uses LLM2Vec, and the appendix reports other classifiers. About 20% of the pre-ChatGPT abstracts receive an LLM label in that comparison. This historical label share needs to be accounted for before the predictions can be interpreted as usage rates.

The controlled mixtures test the frequency estimator separately. They measure how estimates change when reference models are omitted and how much false weight remains in human-only subject controls. We have revised the Discussion and Conclusion to include these results and retained the classifier comparison. Because actual usage in the real corpus is unknown, that comparison cannot establish which method estimates it more accurately.

Q2. What is the paper’s contribution to the community?

A2. We combine a large corpus of academic papers with paired outputs from different LLMs and prompts. This lets us compare how models change the same source texts and examine related patterns in real papers. We also study changes across model generations, which matter because references based on older models may become less representative over time. Our word-frequency method provides a simple, interpretable way to examine these differences. Controlled mixtures test how missing reference models and changes in subject composition affect the estimates, and whether weighting and subject-area adjustment reduce the errors.

Reviewer_PNwQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the influence of LLMs on academic writing through word-frequency trends, text similarity, and LLM-source classification. Using more than 3 million arXiv papers and additional PubMed data, the authors examine lexical changes associated with LLM-generated text, including the increased usage of words such as "beyond" and "via." They further propose a linear, word-frequency-based approach to estimate the stylistic influence of different LLMs over time.

Critical strengths

The paper studies a relevant topic and incorporates recent LLMs and a large collection of academic publications. The comparisons across models and time provide some descriptive insights into evolving writing patterns.

Critical weaknesses
  • Very limited novelty compared to existing work. The central research question and word-frequency-based estimation framework have already been extensively investigated in prior work, particularly Mapping the Increasing Use of LLMs in Scientific Papers (Liang et al., 2024; subsequently published in Nature Human Behaviour, 2025). That work provides a more thoroughly validated population-level estimation framework, including evaluations under controlled mixtures of human and LLM-generated text. The current paper primarily extends the data to newer models and time periods without establishing a substantial methodological or conceptual contribution.

  • Limited significance of the main findings. A considerable portion of the paper focuses on changes in the frequencies of individual words, including "beyond," "via," "the," and "of." While these observations may be interesting descriptively, the paper does not convincingly demonstrate why these particular lexical changes provide important new insights beyond existing studies of LLM writing styles. The significance of these findings for understanding actual LLM adoption or its impact on scientific writing remains unclear.

  • Insufficient validation of the estimation approach. The proposed method relies on linear extrapolation of historical word frequencies and assumptions about mixtures of different LLM writing styles. Changes in vocabulary may also reflect shifts in research topics, writing conventions, and author demographics, making the estimated LLM contributions difficult to interpret. Unlike prior work with controlled synthetic-mixture validation, the paper does not convincingly establish the quantitative accuracy of its estimates under known LLM usage proportions.

Questions for authors

See my comments above.

1 comment

Thank you for your comments on prior work and the validation of our estimates.

Q1. How does the method differ from Liang et al. (2024/2025), and what does the paper add?

A1. Liang et al. estimate the proportion of LLM text in a corpus using human and LLM reference texts, and validate their estimates with controlled mixtures. Their 2024 formulation fits a binary human/LLM mixture by likelihood using token-occurrence distributions. We fit ratios of mean word counts per abstract by constrained least squares, with a separate reference for each model and prompt.

Both analyses use a fixed vocabulary during monthly inference. Liang et al. use a broad vocabulary from their reference texts. We select common words using Google Books Ngram ranks and reference counts. Our monthly ratios also use an extrapolated historical baseline.

The references let us test how estimates change across model releases, when reference models are missing, and when subject composition changes. All estimators use the same held-out mixtures. Weighting and subject-area projection reduce within-subject error from 19.57 to 10.96 percentage points compared with our unweighted fit. Weighting alone gives 13.77 points.

Q2. What do the changes in words such as beyond, via, the, and of add beyond previous work?

A2. These words are examples chosen to make the plots easier to understand. They show differences between model outputs and related changes in real papers. The estimates use 435 words in the original time-series analysis and 545–553 words in the controlled tests. We now state that the individual words are illustrations and do not establish statistical significance on their own. Common-word frequencies can also vary with subject and text length. The controlled tests examine how changes in subject composition affect the estimates.

Q3. How accurate are the estimates when the LLM proportions are known? How do you account for changes in subjects, topics, writing conventions, and author populations?

A3. We added controlled tests using 2,000 paired source abstracts. In each of three splits by source paper, 1,200 papers form the reference set, 400 the calibration set, and 400 the test set. All variants of a source paper stay in the same partition. We select words from the reference set and choose parameters using calibration mixtures. All estimators are then tested on the same held-out targets with known proportions. The results are shown below.

EstimatorPooledWithin subjectHuman only
Unweighted6.06 ± 2.3819.57 ± 2.3324.75 ± 6.27
Variance weighted4.52 ± 0.9913.77 ± 1.4817.55 ± 4.15
Weighted Huber5.72 ± 0.8322.82 ± 4.6930.75 ± 7.59
Weighted and projected4.44 ± 1.0710.96 ± 0.9610.14 ± 0.84

Values are errors in total fitted weight, measured in percentage points and reported as mean ± sample standard deviation over three overlapping source splits. The human-only column gives the fitted total weight when the true proportion is zero. It is not a per-paper false-positive rate.

Compared with weighting alone, projection reduces within-subject error from 13.77 to 10.96 percentage points and human-only false weight from 17.55 to 10.14 points. Pooled errors differ by only 0.08 points. The reduction is an average over the tested subjects and does not hold in every subject. The adjustment removes directions estimated from the human references. It does not control finer topic changes or author demographics.

Pooled targets contain 300 abstracts, while single-subject targets contain 77–153. Differences between these conditions may therefore also reflect sample size. Within each condition, all methods use identical targets. Human-only repetitions use the full subject test pool and change only the order of the abstracts. They do not provide new frequency observations. The reported standard deviation is calculated over the three source splits.

We also corrected the description of January–June 2022. We used this period to choose vocabulary thresholds, so its near-zero estimates are not an independent accuracy check. The five selected months from the real corpus are reported separately as sensitivity checks. Their true usage proportions are unknown, so they cannot establish estimation accuracy.

Reviewer_3ARuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In one sentence, this paper analyzes how the use of large language models (LLMs) changes the word-frequency patterns in scientific and scholarly texts.

The authors propose several measures to quantify shifts in word-frequency distributions and assess the influence of prompts on LLM-generated text. Additionally, they conduct simulation experiments in which LLMs are prompted to rephrase human-written texts, and use this to analyze linguistic differences between human-authored and AI-generated content.

Critical strengths

The paper focuses on an important and relevant problem. Given the increasing use of LLMs in scientific writing, identifying and characterizing LLM-generated text is an important research direction.

The authors conduct quite a few experiments to investigate their research questions and provide empirical evidence.

Critical weaknesses
  1. Although the authors claim to focus on the impact of LLMs (Line 450), it is unclear how changes in word-frequency distributions capture the impact of LLMs. Such changes may provide indirect evidence of LLM usage in scientific writing, but they do not necessarily tell about its broader impact on scientific communication, research quality, or scholarly practices.

  2. The broader goal of the paper remains unclear to me. Even if the authors demonstrate that LLM usage changes word-frequency distributions, what broader research question does this finding address? It is already well understood that LLM-generated text may exhibit different lexical distributions from human-written text. Furthermore, some of the LLMs include watermarks in their generated text.

Questions for authors
  1. Can you clarify the rationale behind equations 1 and 6? Are they motivated by prior literature?
  2. Can you briefly clarify how you define and measure the impact of LLMs? Do you aim to quantify the impact of LLMs on scientific writing or on science in a broader sense?
1 comment

Thank you for your comments on the scope of the paper and the estimator.

Q1. What do you mean by the impact of LLMs? Do your estimates concern scientific writing or science more broadly?

A1. We measure changes in word use in academic titles and abstracts. We compare observed frequencies with historical trends and fit weights for texts generated by different models and prompts. The weights describe how closely the corpus matches these reference texts. They do not directly estimate how many papers used LLMs or assess effects on research quality and scientific practice. The Introduction, results, and Conclusion now state this scope.

Q2. What broader research question does the analysis of word-frequency changes address?

A2. We ask how estimates of LLM influence on academic writing should adapt as models and the ways people use them change over time. New models may produce different writing patterns, and users may change how they use LLMs. Estimates based on earlier model outputs may therefore become less reliable. Our analysis examines how changes in models and prompts affect word-frequency patterns and what these changes mean for estimation over time.

Q3. What is the rationale for Equations (1) and (6)? Are they based on prior work?

A3. Equation (1) writes an observed mean as its population mean plus sampling variation. The observed frequency is the number of occurrences per paper,

$$
f_w^d(t) = \frac{1}{N_t} \sum_{i=1}^{N_t} X_{iw}.
$$

Defining $\varepsilon_w(t) = f_w^d(t) - f_w(t)$ gives Equation (1). We have clarified the units and the error term. Departures from the linear trend may reflect changes in topics and writing conventions as well as LLM use. We therefore treat the extrapolation as a historical baseline.

Equation (6) uses the fact that a mixture mean is a weighted average of its component means. Following earlier corpus-level mixture approaches [1, 2], we fit the observed frequency ratio $r_w(t) = f_w^d(t) / f_w^{\mathrm{pred}}(t)$ to the reference ratios:

$$
r_w(t) \approx \eta_0(t)
+ \sum_{m,p} \eta_{m,p}(t) \frac{f_w^{m,p}}{f_w^0},
$$

$$
\eta_0(t) \ge 0,\quad \eta_{m,p}(t) \ge 0,
\quad \eta_0(t) + \sum_{m,p} \eta_{m,p}(t) = 1.
$$

We minimize the mean squared residual over words. The human component appears once, and each model-prompt weight uses its corresponding reference frequency. We assume that these reference ratios also apply to real papers. The appendix describes the weighted and subject-adjusted fits. The controlled mixtures test whether the estimator recovers known proportions.

We kept the original frequency and mixture-weight notation. The main text now has two short additions, Estimator adjustment and Controlled mixtures. The appendix gives the calculations and test settings. We removed repeated definitions and the optional geometric diagnostic. The limitations now distinguish what the controlled tests establish from what can be inferred from the real corpus.

Q4. How does the analysis account for possible watermarks in LLM outputs?

A4. Our focus is on how LLMs influence academic writing, rather than on estimating their usage rate. If watermarking changes the wording of generated text, we consider this an indirect effect of the LLM system and therefore within the scope of our analysis. We examine the resulting word-frequency patterns without separating possible watermark effects from other features of model output.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.