- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the ability of general-purpose LLMs to detect LLM-generated texts across different LLM generations. It evaluates 15 LLMs from three generations as both text generators and detectors, using 1,000 human-written and 15,000 LLM-generated texts. The results show that newer LLMs generally achieve substantially better detection performance, while texts generated by more recent models tend to be harder to detect. The study also revisits the self-detection advantage reported in prior work and finds that it does not consistently hold across models. Finally, the authors analyze patterns of false positives and false negatives, as well as LLM-generated explanations, revealing differences in error tendencies across generations and inconsistencies in how detectors interpret textual cues.
Critical strengths
- The paper provides an extensive evaluation of 15 LLMs across three LLM generations. Notably, recent LLMs, particularly Gemini 3.1 Pro, achieve remarkably high detection performance.
- It reveals an overall trend of improving detection performance across LLM generations with different error patterns, such as earlier models tending to classify texts as human-written.
- It provides valuable evidence that the self-detection advantage reported in prior work may not generalize across models, supported by statistical analyses.
- It investigates detection errors from multiple aspects, including FPR and FNR, and natural-language explanations, revealing inconsistencies in how different LLMs interpret textual cues.
Critical weaknesses
- The main novelty of this paper would be extending existing evaluation settings to a broader range of recent LLMs, and its methodological novelty remains somewhat limited.
- The paper focuses on binary classification of human-written and LLM-generated texts. To better understand the practical effectiveness of LLMs as detectors, evaluations involving diverse generation prompts, paraphrasing, humanization, adversarial attacks, and human-AI collaborative writing would be valuable. Comparisons with commercial detectors such as Pangram and GPTZero would also help contextualize the results.
- The textual domain ratio in the evaluation dataset is unclear. That information is important to correctly understand the reported detection performance.
- The paper shows that newer LLMs generally achieve better detection performance but provides limited insight into why this is the case. Although watermarking is briefly discussed, its role in detection performance remains unclear.
- The paper groups LLMs by their release period, but this does not necessarily reflect their general capabilities. It would be interesting to examine the correlation between detection performance and established measures of general capability, such as the Artificial Analysis Intelligence Index.
Questions for authors
See the weakness part.