- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper makes two main contributions: The first is MuST-Mix, a multilingual dataset for span-level detection of AI-generated text. The second is MuST, a detection approach that has been evaluated across eight languages.
MuST-Mix is created by combining existing corpora of human-written and machine-generated texts. The authors use semantic matching to align semantically similar sentences, and controlled replacement to generate hybrid human–AI paragraphs containing different numbers of transitions in authorship. This procedure enables the creation of word-level annotations and sentence-level boundary labels.
The proposed detector is evaluated against several existing approaches using complementary metrics, including mean absolute error, the ratio of AI-generated content, word-level F1, boundary-type F1 and boundary-position F1. The experimental results indicate that MuST outperforms the evaluated baselines across the reported settings, although performance varies across languages and evaluation metrics.
Critical strengths
1. Multilingual coverage. The dataset covers eight languages, providing a useful resource for evaluating AI-generated text detection beyond English and investigating cross-linguistic differences in detection performance.
2. Comprehensive evaluation metrics. The evaluation considers complementary aspects of span-level detection, including word-level classification, AI-generated content proportion, and boundary type and localisation. This provides a more detailed assessment of detection performance than relying solely on document-level classification metrics.
Critical weaknesses
1. Dataset construction and representativeness The dataset uses semantic matching and the controlled replacement of complete sentences to generate hybrid human–AI documents. While this procedure does provide precise annotations, it is unclear to what extent these artificially generated examples accurately reflect realistic human–AI writing scenarios. Human–AI texts may involve partial rewriting, editing within sentences, paraphrasing and iterative revisions, all of which are not necessarily captured by the proposed construction procedure.
The dataset's description in the paper could be enhanced by including representative annotated examples and a clearer analysis of its composition, including differences across languages, source datasets and span lengths. Tables 1 and 8 provide general statistics but they do not report the distribution of samples across individual LLM generators and domains, nor the number of original documents used to construct the hybrid paragraphs. Moreover, the dataset is highly imbalanced across languages, with English, Russian, and Chinese accounting for approximately 76% of the samples. There are also substantial differences in average paragraph length and domain coverage across languages. These differences deserve further discussion, as they may influence the reported detection performance. Given the span-level nature of the task, additional statistics on sentence counts, span lengths, and the proportion of AI-generated content would also help characterize the dataset. A more detailed breakdown, potentially included in the supplementary material, would improve transparency and facilitate the interpretation of the experimental results.
Therefore, I am not fully convinced that the paper successfully demonstrates the broader significance and practical applicability of the proposed approach. While identifying AI-generated spans is a legitimate research problem, the distinction between human-written and AI-generated content is not always straightforward in realistic collaborative writing scenarios. It would therefore be useful for the motivation section to clarify which practical applications the proposed detector is intended to support, and whether its performance on synthetically constructed hybrid texts can be expected to generalise to these scenarios.
2. Limited experimental analysis and interpretation I find the experimental analysis insufficient to understand the proposed framework. The results would improve with a detailed interpretation of performance differences across languages and source datasets, as well as an examination of how span length and the number of human–AI transitions affect detection performance.
Besides, the lack of a qualitative error analysis makes it difficult to understand the types of mistakes made by the detector. Representative examples of false positives, false negatives, and incorrectly located boundaries could help identify whether errors are associated with linguistic characteristics, discourse coherence, or artifacts introduced during dataset construction.
Overall, while the quantitative results are promising, the limited analysis makes it difficult to draw broader conclusions about the detector's behaviour and generalisation capabilities beyond the reported benchmark scores.
Questions for authors
- How do the authors ensure that the construction of MuST-Mix does not introduce artifacts that the detector could exploit instead of learning generalizable characteristics of AI-generated text?
- Have the authors investigated performance differences according to the source dataset, span length, or number of human–AI transitions? If not, what conclusions can be drawn about the generalization of MuST beyond the specific construction procedure?
9 comments
Response to Reviewer 8UVu (2/3)
Table D4. Number of reconstructed samples contributed by each generator, aggregated per corpus and language.
| Corpus | Lang. | Generator counts |
|---|---|---|
| DetectRL-X | English | Qwen: 10,557; GPT: 10,511; Gemini: 10,506; DeepSeek: 10,403; Unassigned: 4,681 |
| DetectRL-X | Chinese | Qwen: 11,818; Gemini: 11,801; GPT: 11,741; DeepSeek: 11,470; Unassigned: 34 |
| DetectRL-X | German | Gemini: 10,591; Qwen: 10,562; GPT: 10,544; DeepSeek: 10,488; Unassigned: 4,682 |
| DetectRL-X | French | Gemini: 10,684; Qwen: 10,672; GPT: 10,647; DeepSeek: 10,619; Unassigned: 4,743 |
| DetectRL-X | Portuguese | Qwen: 10,615; Gemini: 10,606; GPT: 10,567; DeepSeek: 10,526; Unassigned: 4,683 |
| DetectRL-X | Arabic | Qwen: 10,004; Gemini: 9,943; GPT: 9,927; DeepSeek: 9,839; Unassigned: 4,404 |
| HC3 | English | GPT: 43,580 |
| HC3 | Chinese | GPT: 54,775 |
| C-ReD | Chinese | gemini-2.5-flash: 17,640; gpt-4o: 17,232; doubao-1.5-pro: 16,556; claude-3.5-haiku: 16,324; gpt-3.5-turbo: 15,844; deepseek-v3: 15,728; qwen-3: 15,016; deepseek-r1: 14,248; qwen-2.5: 14,076; Unassigned: 20,337 |
| M4GT Task 3 | English | LLaMA: 26,180; GPT: 11,315 |
Table D5. Number of reconstructed samples per domain, aggregated per corpus and language.
| Corpus | Lang. | Domain counts |
|---|---|---|
| DetectRL-X | English | Novel: 7,171; Wiki: 7,132; News: 7,130; Academic: 6,933; SEO: 6,814; Webtext: 6,797; Unassigned: 4,681 |
| DetectRL-X | Chinese | Academic: 7,922; Webtext: 7,921; News: 7,892; Wiki: 7,880; Novel: 7,796; SEO: 7,419; Unassigned: 34 |
| DetectRL-X | German | Wiki: 7,163; Novel: 7,118; Webtext: 7,111; News: 7,021; Academic: 7,017; SEO: 6,755; Unassigned: 4,682 |
| DetectRL-X | French | Novel: 7,163; Academic: 7,136; Wiki: 7,129; Webtext: 7,115; News: 7,073; SEO: 7,006; Unassigned: 4,743 |
| DetectRL-X | Portuguese | Novel: 7,101; Wiki: 7,098; Academic: 7,066; Webtext: 7,060; SEO: 6,995; News: 6,994; Unassigned: 4,683 |
| DetectRL-X | Arabic | Wiki: 7,066; Webtext: 6,982; News: 6,814; Novel: 6,768; SEO: 6,341; Academic: 5,742; Unassigned: 4,404 |
| HC3 | English | hc3: 43,580 |
| HC3 | Chinese | 54,775 total |
| C-ReD | Chinese | Question-answer: 56,688; News: 45,384; Composition: 37,757; Film review: 12,832; Paper: 10,340 |
| M4GT Task 3 | English | PeerRead: 26,730; Outfox: 10,765 |
Table D6. Source-document split audit; cells report reconstructed rows / source groups.
| Language | Train | Validation | Test | Source overlap | Text duplicates |
|---|---|---|---|---|---|
| English | 32,645 / 6,719 | 4,681 / 959 | 9,332 / 1,921 | 0 | 0 |
| German | 32,790 / 6,716 | 4,682 / 958 | 9,395 / 1,922 | 0 | 0 |
| Chinese | 32,807 / 6,720 | 4,668 / 958 | 9,355 / 1,915 | 0 | 0 |
A2: Error examples and plausible linguistic or discourse causes.
We appreciate the reviewer's suggestion to connect representative prediction errors with possible linguistic and discourse causes. Appendix B.2 adds three English examples with source metadata, gold and predicted labels, and relevant excerpts. The table below summarizes the proposed qualitative explanations. These are plausible associations in the examples.
| Error type | Observed prediction | Plausible linguistic or discourse contribution |
|---|---|---|
| False positive | A human opening sentence is labeled AI | Formulaic promotional wording and a register change |
| False negative | A generated closing slogan is labeled human | A short span, repeated product vocabulary, and topical continuity |
| Displaced boundary | The human-to-AI boundary is predicted one sentence early | Shared explanatory style and a backward reference across the true junction |
Response to Reviewer 8UVu (3/3)
A3: Performance differences across languages and source datasets.
We thank the reviewer for requesting a detailed interpretation of performance differences across languages and source datasets. The existing language-wise comparisons in Section 5.2 and the component analyses in Section 5.3 address complementary aspects of this question. Section 5.3 explains the different contributions of contextual cues: removing sentence embeddings lowers Chinese Pos. from 0.8803 to 0.6735, while removing the contextual encoder lowers Russian and Spanish Pos. from 0.7284/0.7625 to 0.6524/0.7182. These correspond to relative decreases of 23.49%, 10.43%, and 5.81%, respectively (Tables 3 and 12). Section 5.4 also compares language-specific generalization under unseen generators and domains. MuST's Overall Pos. is 0.7686 under cross-generator evaluation and 0.6990 under cross-domain evaluation, supporting greater difficulty under domain shift in these protocols.
We have expanded Section 5.2 to connect these ablations to the cross-language results. Word F1 ranges from 0.9800 to 0.9928 across languages, and Pos. ranges from 0.7284 to 0.8803. This larger variation in Pos. identifies exact transition localization as a particularly variable part of the task. The text also explains that the language subsets differ in tokenization, generators, and domains, which jointly shape their evaluation conditions.
Section 5.2 now also compares the source subsets used in Figure 4. Relative to HC3, MuST records a 17.32% decrease in Pos. (0.9614 → 0.7949) and a 324.53% increase in MAE (0.0053 → 0.0225) on C-ReD. C-ReD has more transitions per paragraph on average, 1.52 versus 0.99 (Table 29), alongside differences in language, generators, and domains. Figure 4 supports the structural interpretation: MuST Pos. increases from 0.7296 in the shortest-span bin to 0.8627 in the longest, and MAE rises overall from 0.0174 at one gold boundary to 0.0426 at six. MuST leads Qwen-FT and Qwen-CRF in every bin where the corresponding metric is reported. These observations support span structure as a contributor to dataset-specific difficulty.
We have expanded Section 5.6 with an interpretation of the external-source results in Table 7. On LLMTrace, Word F1 decreases by 0.38% (0.9587 → 0.9551) from English to Russian, and Pos. decreases by 26.36% (0.6897 → 0.5079). This contrast highlights sensitivity of exact transition recovery to the target language and source distribution. Against the strongest Qwen baseline, MuST improves Pos. by 93.52% (0.3564 → 0.6897) in English and 156.00% (0.1984 → 0.5079) in Russian. The external comparisons are summarized below; M4GT Task 3 uses training-split adaptation, and LLMTrace uses its original English and Russian test splits.
| Benchmark | MuST Word F1 | MuST Pos. | Qwen-FT Pos. | Qwen-CRF Pos. |
|---|---|---|---|---|
| M4GT Task 3 (EN) | 0.9415 | 0.6433 | 0.3511 | 0.3576 |
| LLMTrace-EN | 0.9587 | 0.6897 | 0.2080 | 0.3564 |
| LLMTrace-RU | 0.9551 | 0.5079 | 0.1256 | 0.1984 |
Together, Sections 5.2, 5.3, 5.4, and 5.6 connect the observed differences to contextual-cue ablations, span structure, and changes in the target data distribution. Appendix C additionally reports the observed variation over three seeds and one to three epochs.
Response to Reviewer 8UVu (1/3)
Thank you for the detailed assessment of the practical setting, dataset composition, representative errors, and generalization across span structures and construction protocols. We have expanded the dataset statistics, added English error examples with plausible linguistic and discourse explanations, and supplied analyses by MGT span length and boundary count together with external-dataset comparisons. Added and substantively revised passages and table entries are highlighted in blue in the revised PDF, including all content from Appendix B onward. The responses below identify the reporting and analysis gaps in the original manuscript and present the corresponding evidence, including the qualitative interpretations supported by the examples.
A1: Practical use and dataset composition.
We thank the reviewer for requesting more detail on practical use, annotation, and dataset composition. MuST is intended to assist human review by identifying candidate AI-origin passages and estimating the AI proportion, with the human reviewer interpreting the evidence. Section 4.1 defines the provenance and boundary labels, and Section 6 discusses finer-grained within-sentence mixing as future work. Appendix B.1 now gives two English construction examples, including the five conditions of an HC3 paragraph and a DetectRL-X example with three human sentences aligned to one rewritten sentence. The following tables report the requested dataset information; the sample counts and average lengths retain the original submission values. MuST-Mix contains 1,398,668 reconstructed samples. English, Russian, and Chinese contribute 1,166,661 (83.4%). The Overall detection results give equal weight to the eight languages, and Section 6 identifies broader and more balanced language coverage as future work. Appendix B supplies the source-document, generator, domain, sentence-count, AI-ratio, span-length, and split-audit details.
Table D1. Language-level sample counts and average paragraph length.
Table D2. Source documents and reconstructed rows per source document.
Table D3. Sentence-count, AI-ratio, AI-span-length, and transition statistics of reconstructed paragraphs.