- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces C-HAT-Bench, a large-scale benchmark for evaluating AI-generated text detection under different human–AI collaborative writing scenarios in Chinese. The benchmark contains over 240,000 machine-involved texts derived from 5,000 human-written sources across five domains, six generators, and four collaborative writing scenarios. The authors evaluate 21 detectors, examining document-level detection performance, domain sensitivity, boundary localization, and cross-condition generalization. The experimental results show that detection performance generally decreases under collaborative writing scenarios compared to the prefix-conditioned generation setting. The authors also observe substantial performance differences across domains, difficulties in identifying human–AI authorship boundaries, and asymmetric generalization across different production conditions.
Critical strengths
- The paper introduces a large-scale benchmark for detecting human–AI collaborative text in Chinese, covering five domains, six generators, 21 detectors, and four collaborative writing scenarios.
- The evaluation is comprehensive, going beyond document-level detection to examine domain sensitivity, boundary localization, and cross-condition generalization.
Critical weaknesses
- While constructing a large-scale detection benchmark for Chinese is valuable, many of the main findings, such as the sensitivity of detection performance to collaborative writing processes and the difficulty of generalizing across them, have already been reported in prior work [1]. As a result, it is unclear what new insights this benchmark provides beyond extending existing evaluations to Chinese. I find the novelty of the empirical findings somewhat limited for ICLR.
- I am not fully convinced that the proposed collaborative scenarios sufficiently represent real-world human–AI collaborative writing. The benchmark mainly focuses on modifying human-written text using LLMs or generating text from human-written sources. However, another common scenario is that humans edit and refine AI-generated text, as studied in prior work [2]. It would be good to have a more thorough discussion of this and how the proposed benchmark differs from existing datasets that cover such scenarios.
- In Section 4.4, the authors select the optimal decision threshold (Youden) using test samples with labels. This setting may lead to overly optimistic generalization performance, as such labels would not be available in practice.
References:
[1] Zhang et al. LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected? NAACL 2024 Findings.
[2] Artemova et al. Beemo: Benchmark of Expert-edited Machine-generated Outputs. NAACL 2025.
Questions for authors
See the weakness part
6 comments
[Response to Weaknesses 2 and Weakness 3] Benchmark Scope and Threshold Selection
Summary of Weaknesses 2 and Weakness3: The benchmark omits human revision of AI-generated drafts, and its distinction from existing collaborative-text datasets needs clarification. In addition, test-set Youden optimization may yield optimistic estimates of cross-domain generalization.
Response to Weakness2: Thank you for highlighting this limitation. We agree that our benchmark starts from human-written texts and does not cover the reverse workflow, represented in Beemo, in which humans revise AI-generated drafts. We response this concern in three parts:
-
Planned data and evaluation extensions. Before finalizing the manuscript, we will add a Chinese subset in which human participants revise AI-generated drafts and evaluate detector performance before and after revision. This evaluation will examine whether the relative performance of detectors observed on our benchmark persists after human revision. We will report the subset construction, results, and analysis in the final manuscript.
-
Textual characteristics of the two collaboration directions. Human editing can correct errors and improve expression while retaining useful content from AI drafts, but revisions focused on wording may leave omissions or reasoning weaknesses unresolved. AI generation or revision can expand or reformulate content using human-authored ideas and context, but may weaken important qualifications or introduce unsupported details. Extensive rewording may preserve content, whereas small edits may alter key facts or arguments; edit magnitude does not directly indicate substantive change. After adding the human-revised subset, we will use these complementary settings to compare original and edited texts and examine how changes in expression and content relate to detector responses.
-
Scale and coverage for systematic evaluation. Beyond the construction differences discussed in our response to Weakness 1, our benchmark contains 245,539 Chinese texts and covers five domains, six generation models, and ten collaboration conditions. The domains range from formally structured legal documents to literature with greater stylistic freedom. This coverage enables us to assess whether detector performance remains consistent across writing styles, generation models, and collaboration conditions within a common evaluation framework. We will clarify these distinctions and the benchmark’s complementary value in the Introduction.
We have added a discussion of Beemo to Related Work and we will conduct a detailed experimental comparison of the two collaboration directions once the human-edited subset is complete.
Response to Weakness3: Thank you for raising this concern. We initially used this setting because ACC at a fixed 5% FPR was too poor to distinguish finer-grained differences in transfer performance. We have replaced the original results in Section 4.4 (Figure 6) of the revised manuscript with these updated AUROC results.
We also replace AIGCdetector-zhv2 with bert-base-chinese in this cross-domain analysis. Bert-base-chinese is based on a BERT model designed specifically for Chinese, making it better suited to evaluating detection across Chinese writing domains.The table reports the updated Figure 6 results. AUROC is macro-averaged over 36 fine-category–generator groups for each train–test pair.
| Train / Test | │ | Law | Literature | News | Encyclopedia | Q&A |
|---|---|---|---|---|---|---|
| Law | │ | 100.0 | 52.9 | 85.6 | 66.8 | 64.6 |
| Literature | │ | 93.8 | 96.7 | 96.6 | 93.2 | 98.8 |
| News | │ | 95.1 | 87.9 | 99.4 | 91.8 | 97.4 |
| Encyclopedia | │ | 78.7 | 72.9 | 89.9 | 100.0 | 88.0 |
| Q&A | │ | 68.4 | 67.4 | 91.7 | 85.9 | 100.0 |
Cross-domain transfer is strongly directional. The Literature-trained detector generalizes well to other domains, with a mean AUROC of 95.6%, whereas detectors trained in other domains achieve a mean AUROC of only 70.3% when tested on Literature. The Law-trained detector also shows weaker cross-domain generalization overall (67.5%), although its AUROC on News remains 85.6%. Its AUROC is 52.9% for transfer from Law to Literature, compared with 93.8% in the reverse direction. This asymmetry may reflect the formulaic nature of legal writing and the greater stylistic diversity of literary texts.
Response for Reviewer qeyA
Dear Reviewer qeyA,
Thank you again for your careful consideration of our work. Your comments have encouraged us to clarify the empirical contribution and scope of the collaboration settings, and to revise the cross-domain evaluation to address your concern about threshold selection. We have provided further discussion and results in our responses and would be grateful for any thoughts you may have when time permits. We would be happy to discuss any remaining concerns and appreciate your time and guidance.
Best regards,
On behalf of the authors of Paper 22168
[Response to Weakness 1] Benchmark Distinctions and New Insights
Response: Thank you for highlighting this issue. We acknowledge that both studies construct human–AI collaborative texts and evaluate detectors, and that MIXSET has reported effects of collaboration operations on detection and difficulties in cross-operation generalization. Within this shared setting, our benchmark further differentiates construction conditions within each collaboration mode to compare detector responses across conditions. We address this concern in three points:
Fine-grained construction conditions. MIXSET covers some of the same modes, but we distinguish conditions within each mode. Content filling crosses source-text character replacement coverage (15–35% or 50–75%) with layout (contiguous or dispersed), yielding four conditions. Prompt generation includes L1–L3 with progressively richer prompts and L3+Ref, which adds a same-domain human reference; machine rewriting includes Light and Deep. These ten conditions enable within-mode comparisons of detector responses.
Paired metrics for detector responses. To compare detector responses within each collaboration mode, we define NSS (Net Score Shift) and MAS (Mean Absolute Shift). Each collaborative text is paired with MGT (the AI-generated baseline without collaboration operations) from the same human source and generation model. With scores oriented so higher values indicate AI-generated text, we normalize paired differences by each detector’s fixed reference standard deviation, SD: Δi,c=sD(yi,c)−sD(yi,MGT),NSSc=SD⟨Δi,c⟩,MASc=SD⟨∣Δi,c∣⟩. NSS measures the mean normalized shift relative to MGT: positive values indicate a net shift toward the AI class, negative values toward the human class, and larger absolute values indicate larger net shifts. MAS measures the mean absolute normalized shift: larger values indicate greater average change regardless of direction. Both are in units of SD.
The table reports NSS / MAS relative to paired MGT for Binoculars (zero-shot) and RADAR (pretrained) across the ten conditions. Bold marks each detector’s most negative NSS and largest MAS. LC/HC denote low/high (15–35%/50–75%) source-text replacement coverage; C/D denote contiguous/dispersed layouts.
Within prompt generation, L1 to L2 lowers NSS from −0.170 to −0.589 for Binoculars and +0.095 to −0.581 for RADAR, with MAS increasing for both. One possible explanation is that richer source information constrains generation and changes the cues available to detectors. Yet L3 has a more negative NSS than L2 for Binoculars but not RADAR, while L3+Ref yields values close to L3. More information does not uniformly strengthen the shift toward the human class. Combining NSS and MAS, our benchmark reveals how the net direction and average magnitude of score changes vary across fine-grained conditions within the same collaboration mode, beyond a single aggregate result per mode.
Linguistic motivation for evaluating Chinese. Chinese typically does not explicitly mark word boundaries. For example, “武汉市长江大桥” allows different segmentations:
Segmentation and interpretation can depend on context. Chinese also relies less on inflectional morphology and more on word order, function words, and context to express grammatical relations. These properties may shape the linguistic cues available to detectors, motivating direct examination of detector responses on Chinese human–AI collaborative texts.
In the revision, we have provided detailed definitions of NSS and MAS in Appendix C.1 and detailed experimental analyses in Appendix D.2, highlighting these changes in blue.