acceptodds
Under review as a conference paper at ICLR 2027

Using LLMs to Detect LLM-Generated Texts: A Cross-Generation Analysis

Abstract

Detecting LLM-generated texts is becoming increasingly important as LLMs have been widely adopted and used in various domains, while their generative outputs can also create risks of misuses and other adverse effects. Various approaches have been developed to design automated detectors, but no single solution remains reliable across models, domains, and evaluation settings. General-purpose LLMs provide an alternative and flexible way to make human/LLM authorship judgements on given texts with explanations to support their decisions, but our understanding of their detection behaviour remains limited, particularly on differences between self-detection (i.e., detection by the same text-generation LLM) and cross-detection (i.e., detection by other LLMs) and how they have evolved over time. This paper presents our study of using 15 LLMs from three generations as both text generators and detectors. We constructed 1,000 human-generated texts (HGTs) and 1,000 LLM-generated texts (LGTs) per model, and instructed each of the 15 detectors to perform binary classification tasks on all 15,000 LGTs and 1,000 HGTs, obtaining 233,428 valid judgments (excluding null/badly-formed ones) with associated natural-language explanations. Our results suggest that detection performance is more strongly related to detector generation than generator generation, while texts produced by the most recent generators remain harder to detect. Rigorous statistical comparisons of self- and cross-detection results provide mixed and model-specific results, suggesting no conclusive evidence of a consistent self-detection advantage or disadvantage across LLMs. Lastly, we analysed error patterns from three perspectives: an analysis based on false-positive and false-negative rates, an LLM-assisted analysis of LLM-generated explanations, and a literature-guided cue-based analysis. The results further show that detectors based on the first (earliest) generation of LLMs have a tendency to predict LGTs as HGTs, detectors based on the second generation of LLMs show stronger tendencies to predict HGTs as LGTs, and detectors based on the latest generation of LLMs have more balanced false-positive and false-negative rates. In addition, potential inconsistent applications of textual cues that would lead to different judgements by different LLMs are also identified and reported.

Primary area:
foundation or frontier models, including LLMs
Keywords:
human-machine text detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_R3pAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the ability of general-purpose LLMs to detect LLM-generated texts across different LLM generations. It evaluates 15 LLMs from three generations as both text generators and detectors, using 1,000 human-written and 15,000 LLM-generated texts. The results show that newer LLMs generally achieve substantially better detection performance, while texts generated by more recent models tend to be harder to detect. The study also revisits the self-detection advantage reported in prior work and finds that it does not consistently hold across models. Finally, the authors analyze patterns of false positives and false negatives, as well as LLM-generated explanations, revealing differences in error tendencies across generations and inconsistencies in how detectors interpret textual cues.

Critical strengths
  • The paper provides an extensive evaluation of 15 LLMs across three LLM generations. Notably, recent LLMs, particularly Gemini 3.1 Pro, achieve remarkably high detection performance.
  • It reveals an overall trend of improving detection performance across LLM generations with different error patterns, such as earlier models tending to classify texts as human-written.
  • It provides valuable evidence that the self-detection advantage reported in prior work may not generalize across models, supported by statistical analyses.
  • It investigates detection errors from multiple aspects, including FPR and FNR, and natural-language explanations, revealing inconsistencies in how different LLMs interpret textual cues.
Critical weaknesses
  • The main novelty of this paper would be extending existing evaluation settings to a broader range of recent LLMs, and its methodological novelty remains somewhat limited.
  • The paper focuses on binary classification of human-written and LLM-generated texts. To better understand the practical effectiveness of LLMs as detectors, evaluations involving diverse generation prompts, paraphrasing, humanization, adversarial attacks, and human-AI collaborative writing would be valuable. Comparisons with commercial detectors such as Pangram and GPTZero would also help contextualize the results.
  • The textual domain ratio in the evaluation dataset is unclear. That information is important to correctly understand the reported detection performance.
  • The paper shows that newer LLMs generally achieve better detection performance but provides limited insight into why this is the case. Although watermarking is briefly discussed, its role in detection performance remains unclear.
  • The paper groups LLMs by their release period, but this does not necessarily reflect their general capabilities. It would be interesting to examine the correlation between detection performance and established measures of general capability, such as the Artificial Analysis Intelligence Index.
Questions for authors

See the weakness part.

Reviewer_V5okOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the question of LLMs detecting LLM-generated text across different generations.

It divides LLMs into 3 generations and chooses 15 LLMs across 3 generations working as both generators and detectors. The constructed datasets is large with 1000 HGTs from M4GT-Bench and 15000 LGTs conditionally generated by 15 generators. Each LLM detects all 16000 texts and makes a binary judgment.

The results show that the detection ability depends on the detector itself, regardless of which generator the text is generated by. Later generation performs much better on detection. This paper also analyzes self-detection and finds it is not consistent. The error pattern analysis shows that detectors can interpret the same textual cues differently.

Critical strengths

The scale of experiments is large. This paper builds a broad datasets including 16000 documents by 15 LLMs, and detects these documents with all chose LLMs as detector. And the experiment details are clearly stated, which enhances the reproducibility.

Critical weaknesses
  1. Dividing LLMs by generation is not profound. Generation is only a confounder and model capability is the key factor. Model capability increases generation by generation and it is intuitive that later models generate harder-to-detect content and detect LGTs more accurately. It is a more valuable question why the G3 models still perform well on LGTS generated by G3. Is that because the generation capability increase slower than the detection capability?

  2. The analyses didn’t give much valuable deep insights in the question or phenomenon. For example, the results show that G3 LLMs nearly detect all LGTs correctly (especially Gemini-3.1), it is more reasonable to investigate the reason why they can reach such a high accuracy instead of conducting error analyses on the few misclassified cases. It is expected to investigate which aspect of capability makes a detector outperforms over other, and which aspect of capability makes a generator generate harder-to-detect texts?

Questions for authors
  1. As the semantic error pattern analyses show that different detectors interpret the same cues differently, does one detector performs consistently with itself on the same LGT? Did the experiment perform repeated trials?

  2. As the LGTs are generated with similar themes, structures and styles to the HGTs, what if the prompt contains one LGT with its corresponding HGT and the detector judges which one is more like LLM-generated?

Reviewer_cxYrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores whether general-purpose LLMs can distinguish between human-written and LLM-generated text. It focuses on the differences between self-detection (detecting text generated by the model itself) and cross-detection (detecting text generated by other models), as well as the performance of models from different generations. Furthermore, the authors analyze the error patterns. The results indicate that newer-generation LLMs demonstrate stronger detection capabilities, but they do not show a stable advantage when identifying texts generated by themselves.

Critical strengths

The authors evaluate 15 models and 225 generator-detector pairs, providing a relatively comprehensive set of experiments.

Critical weaknesses

The paper's main novelty lies in its focus on time and model generations, but the models are categorized into only three broad generations. The experimental data in this paper are primarily based on human-written texts from the M4GT-Bench dataset, which may limit the generalizability of the findings to broader scenarios.

Questions for authors

Would the conclusions of this study still hold when different types of texts are used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.