- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the detection of machine-generated text through two approaches: fine-tuning pre-trained Transformer models (RoBERTa, Llama 3.2 3B and Qwen2.5-1.5B) for binary text classification and the use of a compact sequence classifier, known as the Feature Model, which uses probabilistic features extracted from multiple language models at the token level. The authors also explore a hard-voting ensemble that combines these two approaches.
The models were trained using M4 and CUDRT, and were evaluated using five held-out benchmarks, including scenarios involving texts edited by humans and LLMs. The results demonstrate significant variation across the evaluation datasets, indicating that no single detection approach consistently outperforms the others.
Critical strengths
Evaluation across different scenarios. The paper compares several detection approaches on five held-out benchmarks, including edited texts. This provides a useful insight into how they behave under different evaluation conditions.
It also compares complementary detection strategies. The study considers fine-tuned Transformer classifiers and lightweight models based on token-level probabilistic features, offering a comparison between approaches with different computational and modelling characteristics.
Critical weaknesses
1. Limited methodological novelty. My main concern is that the proposed approaches make only a limited methodological contribution. Fine-tuning pre-trained transformers for binary classification is a well-established NLP strategy, as is the use of token-level probabilistic features for detecting machine-generated text. Similarly, combining classifiers through hard voting is a conventional ensemble technique. In this sense, while the paper does evaluate these approaches in different settings, it does not make it sufficiently clear what the methodological contribution of the proposed Feature Model is beyond that of existing probability-based detectors. The paper would benefit from a clearer justification of the novelty of the proposed approach and a more explicit discussion of its differences from prior work.
2. Limited experimental insights and interpretation. I find that the analysis does not sufficiently detail the behaviour of the different detection strategies. The main finding is that performance varies considerably depending on the dataset, with no single approach consistently dominating. While this observation is relevant, the paper does not investigate the underlying factors of these differences sufficiently.
A more detailed analysis of the datasets' characteristics, the effects of human or LLM editing and the types of errors made by the detectors would help to explain why particular approaches perform better under certain conditions. Without such an analysis, it is difficult to identify what new insights the study provides beyond a comparison of established classification techniques.
Furthermore, the evaluation is restricted to English. While this is a valid experimental choice, it limits the scope of the conclusions, particularly since the underlying datasets contain multilingual data.
Overall, while the paper provides an empirical comparison of different detection strategies, I find the methodological novelty and the insights derived from the experiments insufficiently developed to constitute a substantial contribution.
Questions for authors
-
How does the proposed Feature Model differ from existing machine-generated text detectors based on token-level probabilities and lightweight classifiers in terms of its specific methodological contribution?
-
Have the authors investigated which dataset characteristics or types of text modification explain the differences in performance across the five evaluation settings? Such an analysis could help to establish whether the study provides insights that can be generalised beyond the reported benchmark results.
2 comments
Assessment after author rebuttal
I have reviewed the revisions to the paper, which address several of my concerns about the experimental analysis. However, I still have concerns about the limited methodological novelty. While these new experiments provide useful evidence for the effectiveness of the proposed Feature Model, they do not change my assessment of its contribution to existing approaches substantially. Therefore, while I acknowledge that the revised manuscript is stronger, I have decided to maintain my original rating.
We thank the reviewer for the detailed assessment. We revised the paper and added analyses that address both concerns.
Summary of the revision. We retrained the Feature Model with five random seeds and now select the epoch and seed only by macro-F1 on the in-distribution validation split; Table 3 reports this model and Appendix C reports the mean ± sd over seeds. The ensembles (Table 3) and the ablation (Table 5) were recomputed with the same protocol. The main conclusions of the paper are unchanged.
W1/Q1 – Methodological contribution and difference from existing probability-based detectors. Our contribution is not a new family of token-level probability features, but a learned aggregation of token-aligned predictability signals extracted from multiple variants of the same language model. Unlike approaches based on document-level statistics or independently computed detection scores, our Feature Model operates directly on the sequence of aligned token-level feature representations and uses self-attention to learn their interactions before aggregating them into a document-level prediction. The experiments systematically investigate whether this learned aggregation provides benefits over conventional classifiers trained on the same features, particularly under cross-dataset and cross-generator distribution shifts.
The revised Background (Section 2) positions the Feature Model with respect to the closest prior work:
W2/Q2 – Which dataset characteristics explain the differences across benchmarks? The revised Sections 4.2, 4.5 and 4.6 and Appendices D–E identify three factors:
The results suggest that learned aggregation can improve cross-dataset generalisation over conventional classifiers using the same token-level features. However, this benefit does not eliminate limitations in the underlying feature representation. In particular, generalisation remains challenging for older unseen generator families, whose outputs exhibit predictability patterns that resemble human-written text under our feature extractors. Thus, our findings highlight both the potential of learned aggregation and the importance of the feature representation and its coverage of generator families.
English only. We agree. The feature generators are predominantly English models; we now state this explicitly as a limitation and list multilingual evaluation as future work (Section 4.6).