acceptodds
Under review as a conference paper at ICLR 2027

C-HAT-Bench: Benchmarking Chinese AI-Text Detection Beyond Fully Generated Text

Abstract

Large Language Models (LLMs) increasingly participate in writing by modifying or extending human drafts, causing machine involvement to vary in both form and extent. Yet most Machine-Generated Text (MGT) detectors are evaluated only on fully human-written versus fully AI-generated text. Because human–AI collaboration can weaken or redistribute cues associated with machine generation, strong performance under this binary setting may overstate detector reliability. This mismatch remains underexplored in Chinese: detection cues are shaped by tokenization and language-specific text distributions, yet controlled resources spanning production settings, domains, and generators remain limited. To fill this gap, we present a **C**hinese **H**uman-**A**I Collaborative **T**ext Detection **Bench**mark (**C-HAT-Bench**), a unified benchmark that links 5,000 human-written source texts from five domains to more than 240,000 variants produced using six generative models under *Prefix-Conditioned Continuation* as a reference setting and three collaborative production modes. We evaluate 21 detectors through four protocols spanning zero-shot and pretrained supervised document-level detection, boundary localization, and cross-condition generalization. Relative to Prefix-Conditioned Continuation, mean AUROC across document-level detectors is 12.0% lower on the collaborative production modes, with the largest detector-specific relative decrease reaching 44.4%. Transfer across collaborative production modes is also asymmetric, indicating that performance under one production setting does not reliably predict performance under another.

Primary area:
datasets and benchmarks
Keywords:
Machine-Generated Text Detection, Human-AI Collaborative Writing, Boundary Localization, Cross-Condition Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qeyAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces C-HAT-Bench, a large-scale benchmark for evaluating AI-generated text detection under different human–AI collaborative writing scenarios in Chinese. The benchmark contains over 240,000 machine-involved texts derived from 5,000 human-written sources across five domains, six generators, and four collaborative writing scenarios. The authors evaluate 21 detectors, examining document-level detection performance, domain sensitivity, boundary localization, and cross-condition generalization. The experimental results show that detection performance generally decreases under collaborative writing scenarios compared to the prefix-conditioned generation setting. The authors also observe substantial performance differences across domains, difficulties in identifying human–AI authorship boundaries, and asymmetric generalization across different production conditions.

Critical strengths
  • The paper introduces a large-scale benchmark for detecting human–AI collaborative text in Chinese, covering five domains, six generators, 21 detectors, and four collaborative writing scenarios.
  • The evaluation is comprehensive, going beyond document-level detection to examine domain sensitivity, boundary localization, and cross-condition generalization.
Critical weaknesses
  • While constructing a large-scale detection benchmark for Chinese is valuable, many of the main findings, such as the sensitivity of detection performance to collaborative writing processes and the difficulty of generalizing across them, have already been reported in prior work [1]. As a result, it is unclear what new insights this benchmark provides beyond extending existing evaluations to Chinese. I find the novelty of the empirical findings somewhat limited for ICLR.
  • I am not fully convinced that the proposed collaborative scenarios sufficiently represent real-world human–AI collaborative writing. The benchmark mainly focuses on modifying human-written text using LLMs or generating text from human-written sources. However, another common scenario is that humans edit and refine AI-generated text, as studied in prior work [2]. It would be good to have a more thorough discussion of this and how the proposed benchmark differs from existing datasets that cover such scenarios.
  • In Section 4.4, the authors select the optimal decision threshold (Youden) using test samples with labels. This setting may lead to overly optimistic generalization performance, as such labels would not be available in practice.

References:

[1] Zhang et al. LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected? NAACL 2024 Findings.

[2] Artemova et al. Beemo: Benchmark of Expert-edited Machine-generated Outputs. NAACL 2025.

Questions for authors

See the weakness part

6 comments
AuthorsOpenReview ↗

[Response to Weakness 1] Benchmark Distinctions and New Insights

Summary of Weakness 1: What does the benchmark contribute beyond extending existing evaluations to Chinese, and how novel are its empirical findings for ICLR?

Response: Thank you for highlighting this issue. We acknowledge that both studies construct human–AI collaborative texts and evaluate detectors, and that MIXSET has reported effects of collaboration operations on detection and difficulties in cross-operation generalization. Within this shared setting, our benchmark further differentiates construction conditions within each collaboration mode to compare detector responses across conditions. We address this concern in three points:

  • Fine-grained construction conditions. MIXSET covers some of the same modes, but we distinguish conditions within each mode. Content filling crosses source-text character replacement coverage (15–35% or 50–75%) with layout (contiguous or dispersed), yielding four conditions. Prompt generation includes L1–L3 with progressively richer prompts and L3+Ref, which adds a same-domain human reference; machine rewriting includes Light and Deep. These ten conditions enable within-mode comparisons of detector responses.

  • Paired metrics for detector responses. To compare detector responses within each collaboration mode, we define NSS (Net Score Shift) and MAS (Mean Absolute Shift). Each collaborative text is paired with MGT (the AI-generated baseline without collaboration operations) from the same human source and generation model. With scores oriented so higher values indicate AI-generated text, we normalize paired differences by each detector’s fixed reference standard deviation, SDS_D: Δi,c=sD(yi,c)−sD(yi,MGT),NSSc=⟨Δi,c⟩SD,MASc=⟨∣Δi,c∣⟩SD.\Delta_{i,c}=s_D(y_{i,c})-s_D(y_{i,\mathrm{MGT}}),\qquad \mathrm{NSS}_c=\frac{\langle\Delta_{i,c}\rangle}{S_D},\qquad \mathrm{MAS}_c=\frac{\langle|\Delta_{i,c}|\rangle}{S_D}. NSS measures the mean normalized shift relative to MGT: positive values indicate a net shift toward the AI class, negative values toward the human class, and larger absolute values indicate larger net shifts. MAS measures the mean absolute normalized shift: larger values indicate greater average change regardless of direction. Both are in units of SDS_D.

    The table reports NSS / MAS relative to paired MGT for Binoculars (zero-shot) and RADAR (pretrained) across the ten conditions. Bold marks each detector’s most negative NSS and largest MAS. LC/HC denote low/high (15–35%/50–75%) source-text replacement coverage; C/D denote contiguous/dispersed layouts.

    Collaboration conditionBinoculars\quad\textbf{Binoculars}\quadRADAR\qquad\textbf{RADAR}\qquad
    LC-C−1.172 / 1.389−0.451 / 1.068
    LC-D−1.330 / 1.514−0.320 / 1.104
    HC-C−0.650 / 1.063−0.654 / 1.043
    HC-D−1.016 / 1.289−0.638 / 1.131
    L1−0.170 / 0.889+0.095 / 0.887
    L2−0.589 / 1.008−0.581 / 0.947
    L3−0.807 / 1.158−0.569 / 0.925
    L3+Ref−0.792 / 1.157−0.543 / 0.928
    Light−1.207 / 1.430−0.317 / 0.970
    Deep−1.154 / 1.386−0.254 / 0.931

    Within prompt generation, L1 to L2 lowers NSS from −0.170 to −0.589 for Binoculars and +0.095 to −0.581 for RADAR, with MAS increasing for both. One possible explanation is that richer source information constrains generation and changes the cues available to detectors. Yet L3 has a more negative NSS than L2 for Binoculars but not RADAR, while L3+Ref yields values close to L3. More information does not uniformly strengthen the shift toward the human class. Combining NSS and MAS, our benchmark reveals how the net direction and average magnitude of score changes vary across fine-grained conditions within the same collaboration mode, beyond a single aggregate result per mode.

  • Linguistic motivation for evaluating Chinese. Chinese typically does not explicitly mark word boundaries. For example, “武汉市长江大桥” allows different segmentations:

    SegmentationInterpretation
    武汉市 / 长江大桥The Yangtze River Bridge in Wuhan
    武汉 / 市长 / 江大桥Wuhan’s mayor, Jiang Daqiao (a hypothetical name)

    Segmentation and interpretation can depend on context. Chinese also relies less on inflectional morphology and more on word order, function words, and context to express grammatical relations. These properties may shape the linguistic cues available to detectors, motivating direct examination of detector responses on Chinese human–AI collaborative texts.

In the revision, we have provided detailed definitions of NSS and MAS in Appendix C.1 and detailed experimental analyses in Appendix D.2, highlighting these changes in blue.

AuthorsOpenReview ↗

[Response to Weaknesses 2 and Weakness 3] Benchmark Scope and Threshold Selection

Summary of Weaknesses 2 and Weakness3: The benchmark omits human revision of AI-generated drafts, and its distinction from existing collaborative-text datasets needs clarification. In addition, test-set Youden optimization may yield optimistic estimates of cross-domain generalization.

Response to Weakness2: Thank you for highlighting this limitation. We agree that our benchmark starts from human-written texts and does not cover the reverse workflow, represented in Beemo, in which humans revise AI-generated drafts. We response this concern in three parts:

  • Planned data and evaluation extensions. Before finalizing the manuscript, we will add a Chinese subset in which human participants revise AI-generated drafts and evaluate detector performance before and after revision. This evaluation will examine whether the relative performance of detectors observed on our benchmark persists after human revision. We will report the subset construction, results, and analysis in the final manuscript.

  • Textual characteristics of the two collaboration directions. Human editing can correct errors and improve expression while retaining useful content from AI drafts, but revisions focused on wording may leave omissions or reasoning weaknesses unresolved. AI generation or revision can expand or reformulate content using human-authored ideas and context, but may weaken important qualifications or introduce unsupported details. Extensive rewording may preserve content, whereas small edits may alter key facts or arguments; edit magnitude does not directly indicate substantive change. After adding the human-revised subset, we will use these complementary settings to compare original and edited texts and examine how changes in expression and content relate to detector responses.

  • Scale and coverage for systematic evaluation. Beyond the construction differences discussed in our response to Weakness 1, our benchmark contains 245,539 Chinese texts and covers five domains, six generation models, and ten collaboration conditions. The domains range from formally structured legal documents to literature with greater stylistic freedom. This coverage enables us to assess whether detector performance remains consistent across writing styles, generation models, and collaboration conditions within a common evaluation framework. We will clarify these distinctions and the benchmark’s complementary value in the Introduction.

We have added a discussion of Beemo to Related Work and we will conduct a detailed experimental comparison of the two collaboration directions once the human-edited subset is complete.

Response to Weakness3: Thank you for raising this concern. We initially used this setting because ACC at a fixed 5% FPR was too poor to distinguish finer-grained differences in transfer performance. We have replaced the original results in Section 4.4 (Figure 6) of the revised manuscript with these updated AUROC results.

We also replace AIGCdetector-zhv2 with bert-base-chinese in this cross-domain analysis. Bert-base-chinese is based on a BERT model designed specifically for Chinese, making it better suited to evaluating detection across Chinese writing domains.The table reports the updated Figure 6 results. AUROC is macro-averaged over 36 fine-category–generator groups for each train–test pair.

Train / Test│LawLiteratureNewsEncyclopediaQ&A
Law│100.052.985.666.864.6
Literature│93.896.796.693.298.8
News│95.187.999.491.897.4
Encyclopedia│78.772.989.9100.088.0
Q&A│68.467.491.785.9100.0

Cross-domain transfer is strongly directional. The Literature-trained detector generalizes well to other domains, with a mean AUROC of 95.6%, whereas detectors trained in other domains achieve a mean AUROC of only 70.3% when tested on Literature. The Law-trained detector also shows weaker cross-domain generalization overall (67.5%), although its AUROC on News remains 85.6%. Its AUROC is 52.9% for transfer from Law to Literature, compared with 93.8% in the reverse direction. This asymmetry may reflect the formulaic nature of legal writing and the greater stylistic diversity of literary texts.

AuthorsOpenReview ↗

Response for Reviewer qeyA

Dear Reviewer qeyA,

Thank you again for your careful consideration of our work. Your comments have encouraged us to clarify the empirical contribution and scope of the collaboration settings, and to revise the cross-domain evaluation to address your concern about threshold selection. We have provided further discussion and results in our responses and would be grateful for any thoughts you may have when time permits. We would be happy to discuss any remaining concerns and appreciate your time and guidance.

Best regards,

On behalf of the authors of Paper 22168

Reviewer_XNm9OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces C-HAT-Bench, a new machine-generated text (MGT) detection benchmark for Chinese human–AI collaboration. The benchmark comprises 5k source texts across five domains and 240k MGT variants generated by six LLMs. In addition to fully machine-written texts, it includes three production modes: content filling, machine rewriting, and prompt-based generation. The authors evaluate 21 detectors under four protocols, finding lower detection performance for the production modes and asymmetric out-of-distribution (OOD) performance.

Critical strengths
  • Chinese is an understudied language for MGT detection, and this benchmark helps fill this gap.

  • The benchmark is comprehensive, covering multiple domains, production modes, and generators. It also enables evaluation under four protocols and has the potential to support additional settings.

  • The selection of zero-shot and pre-trained detectors is extensive, including recent state-of-the-art detectors.

  • The findings provide a first extensive assessment of the strengths of MGT detectors for Chinese.

Critical weaknesses
  • Line 51: This paragraph would benefit from explaining why Chinese, specifically, is an important setting to study.

  • Detector selection: I understand that the paper evaluates only zero-shot and pre-trained detectors. While it covers a range of such detectors, it omits detectors trained on the benchmark. I believe the most relevant class of detectors includes trained BERT-based detectors (for which a Chinese variant exists: bert-base-chinese). Recent work also shows that representation-space-based detectors can perform well across languages and under OOD settings. Both developments would be interesting to test on this benchmark. In particular, I am thinking of detectors such as RepreGuard (Chen et al., 2025) and MGT probes (Quaremba et al., 2026).

  • Related work: I believe the related-work section, especially the benchmark subsection, would benefit from a brief discussion of the Chinese benchmark data currently available. I am thinking of MultiSocial (Macko et al., 2025) and MULTITuDE (Macko et al., 2024). I also think EditLens (Thai et al., 2026) would be important to include, as it extends the binary-classification setting that the paper also mentions.

  • The findings are broadly in line with what other benchmarks have established (e.g., DetectRL-X (Wu et al., 2026) and TSM, which the paper cites). Why should we expect different findings in Chinese? In what ways do the findings differ from those of prior benchmarks?

  • Clarifications:

    • How are spans selected for the content-filling task?

    • How do you ensure that the source topic, main facts, and domain style are preserved after machine rewriting (line 198)? An additional check would be useful here. Otherwise, extensive changes may go beyond the intended edits.

    • Line 217: Do you have statistics on text length? If possible, could you provide subsets of the data for different text lengths, as in other benchmarks such as DetectRL-X (Wu et al., 2026)?

  • Questions and minor comments

    • Line 160: Use a period instead of a comma.

References (apologies for the inconsistent formatting, but the references itself are correct)

Xin Chen, Junchao Wu, Shu Yang, Runzhe Zhan, Zeyu Wu, Ziyang Luo, Di Wang, Min Yang, Lidia S. Chao, and Derek F. Wong. 2025. RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns. Transactions of the Association for Computational Linguistics, 13:1812–1831.

Dominik Macko, Jakub Kopál, Robert Moro, and Ivan Srba. 2025. MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 727–752, Vienna, Austria. Association for Computational Linguistics.

Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023. MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9960–9987, Singapore. Association for Computational Linguistics.

Thai, Katherine, Bradley Emi, Elyas Masrour, and Mohit Iyyer. "Editlens: Quantifying the extent of ai editing in text." In International Conference on Learning Representations, vol. 2026, pp. 122997-123024. 2026.

Gerrit Quaremba, Hanqi Yan, Elizabeth Black, Denny Vrandecic, and Elena Simperl. 2026. Linear Probing Provides Robust and Efficient Detection of Machine-Generated Text. arXiv preprint arXiv:2608.24780.

Junchao Wu, Yefeng Liu, Chenyu Zhu, Hao Zhang, Zeyu Wu, Tianqi Shi, Yichao Du, Longyue Wang, Weihua Luo, Jinsong Su, and Derek F. Wong. 2026. DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 38247–38294, San Diego, California, United States. Association for Computational Linguistics.

Questions for authors

see above

6 comments
AuthorsOpenReview ↗

[Response to Weakness 1, Weakness 2, and Weakness 3] Chinese-Specific Motivation, Detector Coverage, and Benchmark Context

Summary of Weakness 1, Weakness 2, and Weakness 3: The manuscript does not sufficiently explain which linguistic properties make Chinese distinctive for AI-text detection, does not evaluate representative detectors from supervised training-based and representation-space paradigms, and provides insufficient discussion of existing Chinese benchmarks and EditLens in the Related Work section.

  1. Response to Weakness 1: Thank you for this suggestion. We will add the following explanation to the paragraph at Line 51 identified by the reviewer: Chinese typically lacks explicit word boundaries and relies less on inflectional morphology to express grammatical relations, placing greater emphasis on word order, function words, and context. These properties may shape the linguistic cues used by detectors and motivate direct evaluation of their reliability on Chinese human-AI collaborative text. The corresponding additions and revisions have been highlighted in blue in the revised manuscript.

  2. Response to Weakness 2: Thank you for the reviewer’s suggestion. We additionally evaluated one supervised training-based detector (bert-base-chinese) and three representation-space detectors (RepreGuard, MGT Probes-LLP, and MGT Probes-CLP), and report their AUROC under each collaboration condition using the same test split. All table values are rounded to two decimal places.For the training and fitting configurations, data splits, sample counts, and random seeds used for these additional baselines, please refer to our response to Reviewer iDJX’s Question 1.

    DetectorMGTLC-DLC-CHC-DHC-CLightDeepL1L2L3L3+Ref
    bert-base-chinese0.990.750.850.980.930.970.981.001.001.001.00
    RepreGuard0.960.770.790.840.950.790.920.990.980.940.94
    MGT Probes-LLP0.980.640.810.740.900.720.840.980.920.940.99
    MGT Probes-CLP0.990.840.940.890.950.910.950.990.970.970.98

    The results show that collaboration dimensions affect detection performance, with the magnitude depending on the detector type. In particular, bert-base-chinese performs strongly on the machine-rewriting subset, achieving AUROCs of 0.97 and 0.98 under Light and Deep Rewriting, respectively. On the prompt-based generation subset, its AUROC reaches 0.99 under all four conditions (L1, L2, L3, and L3+Ref), indicating strong and stable detection capability. By contrast, bert-base-chinese shows a clear performance drop under content filling, especially under LC-D. Representation-space detectors generally achieve lower AUROC under content filling and machine rewriting, although their responses vary across methods; MGT Probes-CLP is generally more stable than MGT Probes-LLP. Prompt-based generation also generally yields higher AUROC than content filling and machine rewriting, possibly because it directly produces complete machine-generated texts and therefore preserves more consistent machine-generation features.

    We have supplemented the revised manuscript with the above experimental settings, results, and analysis. All newly added and revised content have been highlighted in blue.

  3. Response to Weakness 3: Thank you for the reviewer’s suggestion. We agree that the original manuscript insufficiently discussed relevant benchmarks. We will add the following works and compare them with C-HAT-Bench in the revised Related Work section:

    • MultiSocial targets multilingual social-media detection, includes Chinese, and mainly performs human-machine binary classification using machine-rewritten human posts. In contrast, C-HAT-Bench distinguishes content filling, machine rewriting, and prompt-based generation in Chinese, multi-domain human-AI collaborative text.

    • MULTITuDE focuses on multilingual news detection and cross-lingual/generator generalization, with Chinese mainly used as an unseen test language. It generates complete news articles from headlines, whereas C-HAT-Bench links human source texts with multiple collaborative variants and controls collaboration types and degrees.

    • EditLens distinguishes human-written, AI-edited, and AI-generated text while estimating editing extent. Although related to machine rewriting, it mainly focuses on English and does not cover C-HAT-Bench’s multi-domain, multi-model, and multi-mode settings.

    Thus, C-HAT-Bench complements these works with source-linked, fine-grained evaluation of Chinese human-AI collaboration. These discussions have been added to Related Work, with new content highlighted in blue.

AuthorsOpenReview ↗

[Response to Weakness 5 and Weakness 6] Data Construction Clarifications and Minor Textual Correction

Summary of Weakness 5 and Weakness 6: The reviewer requests clarification on the construction and placement of content-filling spans, the preservation of source topics, main facts, and domain styles during machine rewriting, and whether text-length-stratified data subsets can be provided. In addition, a punctuation error appears at Line 160, where a comma should be replaced with a period.

Response to Weakness 5: Thank you for these clarification requests. We clarify the construction of spans, semantic preservation, and text-length statistics below:

  1. Span selection and placement. As described in the “Data Construction” section on page 4 of the original manuscript, we define four content-filling configurations, each with a coverage range and permitted span counts. For each sample:

    • Coverage and span-count sampling: Given the assigned configuration, we randomly sample the actual coverage from its corresponding range and select a permitted number of spans. We then determine the length of each span accordingly.
    • Span placement: For the continuous configurations (LC_C and HC_C), we select one contiguous suffix at the end of the original human-written text. For the dispersed configurations (LC_D and HC_D), we select multiple spans across different text regions using a randomized, boundary-aware procedure, subject to boundary, minimum-gap, and non-overlap constraints.
    • Text replacement and reproducibility: The generated text replaces the selected spans, while all remaining parts of the original text remain unchanged. A fixed random seed is used throughout the process to ensure reproducibility.

    We will add these span selection and placement details to Appendix A.5 (Data Construction Details) of the revised manuscript, with the corresponding additions and revisions highlighted in blue.

  2. Preservation of source topic, main facts, and domain style. Thank you for the reviewer's suggestion. In the prompts for machine rewriting, we explicitly require the source text's topic, main facts, and domain style to be preserved. To further examine this constraint, we computed the cosine similarity between the Chinese semantic embeddings of 59,038 machine-rewritten texts and their corresponding human-written source texts. The results are shown below:

    Rewriting ConditionNumber of PairsMean Cosine Similarity ± SD5th Percentile
    Light Rewriting29,6470.9594 ± 0.05140.8449
    Deep Rewriting29,3910.9401 ± 0.06040.8081

    The results show that both Light and Deep Rewriting maintain high overall semantic similarity to the source texts, supporting the general preservation of the original topics and main semantics. We will add this check, the table, and the relevant explanation to the revised manuscript. All newly added and revised content will be highlighted in blue.

  3. Text-length statistics and stratified subsets. Thank you for the suggestion. Tables 5 and 6 in Appendix A.6 report length statistics for human-written and machine-involved texts, respectively. We have now partitioned all 240,539 machine-involved texts into nine subsets by character count, retaining each text in full. We use 100-character bins spanning 900–1,599 characters, plus two bins for lengths below 900 and at least 1,600. The table summarizes the length statistics for each subset.

    RangeMaxMinMeanSDCV (%)Count
    <900899611839.0948.575.7916,370
    900–999999900953.2828.773.0223,867
    1000–1099109910001051.3428.662.7335,211
    1100–1199119911001149.8728.902.5139,567
    1200–1299129912001248.9028.852.3138,891
    1300–1399139913001347.7628.872.1434,461
    1400–1499149914001445.5828.661.9825,498
    1500–1599159915001544.1228.541.8514,055
    ≥1600213316001704.2597.695.7312,619
    Total21336111222.62222.7018.22240,539

    Lengths are measured in characters, including punctuation and whitespace. SD denotes the population standard deviation, and CV is the coefficient of variation (SD/Mean × 100%).

    We have added the partitioning rules and statistics to Appendix A.6 of the revised manuscript and uploaded the length-stratified subsets to our GitHub repository. All additions and revisions to the manuscript have been highlighted in blue.

Response to Weakness6: Thank you for pointing this out. We have replaced the comma with a period at Line 160 in the revised manuscript.

AuthorsOpenReview ↗

[Response to Weakness 4] Consistency with Prior Benchmarks and Chinese-Setting Findings

Summary of Weakness 4: The reviewer notes that our findings are broadly consistent with prior benchmarks, such as DetectRL-X and TSM, and asks why different findings should be expected in Chinese and in what respects our findings differ from those of prior benchmarks.

Response: Thank you for the reviewer’s comment. We acknowledge that C-HAT-Bench is broadly consistent with prior human–AI collaborative benchmarks, our explanation has two aspects:

  • Motivation for evaluating Chinese. Modern Chinese typically lacks explicit word boundaries. Grammatical relations rely less on inflectional morphology and more on word order, function words, and context. Thus, the same character sequence may have different segmentations and interpretations. For example:

    SegmentationInterpretation
    武汉市 / 长江大桥The Yangtze River Bridge in Wuhan
    武汉 / 市长 / 江大桥Wuhan’s mayor, Jiang Daqiao (a hypothetical name)

    This example illustrates that the same Chinese character sequence may admit different segmentations and semantic interpretations, potentially affecting the lexical, syntactic, and contextual cues available to detectors. Beyond this ambiguity, domain-specific writing conventions may also affect the linguistic cues available to detectors. We therefore investigate how these factors influence detection behavior under human-AI collaboration. C-HAT-Bench applies finer-grained control over collaboration modes and their extent to examine their effects on detectors in Chinese.

  • Fine-grained findings in the Chinese setting. Our results reveal interaction effects between collaboration factors and detector types. The revised AUROC analysis also shows substantial cross-domain differences for training-based detectors. A detector trained on formal legal texts achieves an AUROC of 64.6 on Q&A texts, compared with 100.0 on its in-domain test set. In the reverse direction, a detector trained on Q&A texts achieves an AUROC of 68.4 on legal texts, compared with 100.0 in-domain. These results show that writing distributions across Chinese domains affect the cross-domain generalization of training-based detectors. Detailed AUROC results are provided in our response to Reviewer qeyA’s Weakness 3.

Reviewer_iDJXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces C-HAT-Bench, a benchmark for detecting Chinese human–AI collaborative texts. It evaluates existing detectors under diverse collaborative writing settings and provides insights into their performance and generalization.

Critical strengths
  1. The dataset has a relatively comprehensive coverage and of a large scale. Particularly, the "operation (prompt)" aspect is carefully considered in terms of degree.

  2. The evaluated detectors provide relatively comprehensive coverage.

Critical weaknesses
  1. The content-filling procedure clearly states its grouping and assignment rules, but the other three settings do not specify per-source model assignments. A table or paragraph summarizing LLM and operation assignments per setting would clarify dataset construction.

  2. The boundary-localization analysis does not separate the effects of coverage and filling layout. More systematic comparisons would help clarify their individual effects and possible interaction.

Questions for authors
  1. The paper omits key training details, such as learning rates, batch sizes, training epochs, optimizers, and random seeds. And for models requiring training, it does not specify the training-set partitioning or sample sizes. These omissions limit reproducibility.

  2. Could you please explicitly define what “human–AI collaboration” means in this paper?

5 comments
AuthorsOpenReview ↗

[Response to Weakness 1] Clarification of Per-Source Model and Operation Assignments

Summary of Weakness 1: Per-source LLM assignments are not specified for MGT, Machine Rewriting, and Prompt-based Generation. Please summarize the LLM and operation assignments for each setting in a table or paragraph.

Response: Thank you for pointing out that our per-source assignment rules were unclear. All four settings use the same 5,000 human-written source texts (1,000 per domain).

For MGT, every source is assigned to all six generators listed, each producing one continuation from the first 30 source tokens.

For Content Filling, we divide each domain’s sources into three nearly equal groups and assign the six generators in pairs to these groups. The same fixed pair processes every source in its group under all four filling conditions.

For Machine Rewriting, all six generators rewrite every complete source at both Light and Deep levels, producing two planned outputs per generator per source.

For Prompt-based Generation, GPT-5.5 first extracts a shared set of structured fields from each source, which we use to construct the four prompt levels. All six generators, then produce one text per source at each level: L1, L2, L3, and L3+Ref.

The table below summarizes the generator and operation assignments and the planned number of final texts per source.

SettingGenerator and operation assignments per sourcePlanned outputs per source
MGTEach of the six generators continues from the first 30 source tokens once, targeting the source length; the prefix and continuation are concatenated6 × 1 = 6
Content FillingBoth assigned generators apply all four combinations of coverage (low/high) and layout (contiguous/dispersed)2 × 4 = 8
Machine RewritingEach of the six generators rewrites the complete source once at each level: Light and Deep6 × 2 = 12
Prompt-based GenerationEach of the six generators produces one text at each prompt level: L1, L2, L3, and L3+Ref6 × 4 = 24

We have incorporated these assignment rules and the table into Appendix A.5 of the revised manuscript, with the corresponding additions and revisions highlighted in blue.

AuthorsOpenReview ↗

[Response to Question 1] Training Configurations and Data Splits

Summary of Question 1: Please specify the learning rates, batch sizes, training epochs, optimizers, and random seeds, together with data splits and sample sizes for models requiring training, to support reproducibility.

Response: Thank you for highlighting these omissions. We have added the training and fitting configurations, including random seeds, to Appendix C.5, and the data splits and sample counts to Appendix C.6. These additions also cover the benchmark-trained and representation-based baselines introduced during revision. The changes are highlighted in blue.

(1) Training and fitting configurations

For the neural detectors fine-tuned in our experiments, we use AdamW, a learning rate of 1×10−41 \times 10^{-4}, an effective batch size of 32, and two training epochs. DeBERTa–BiGRU–CRF uses OneCycleLR, while IMBD and WCP use cosine annealing. We use seed 42 for training, fitting, source partitioning, and sampling. Appendix C.5 provides the configurations for each method.

(2) Data splits and sample counts.

Sources are partitioned within each domain, with all variants of a source kept in the same training, validation, or test partition. Detectors share source partitions within each evaluation, while datasets are constructed separately across evaluations. Document-level datasets contain equal numbers of human-written and machine-involved texts. All counts below follow training/validation/test order.

EvaluationSource partitionTraining / validation / test counts
Cross-collaboration800/100/100 sources per domain, shared across the four settings8,000/1,000/1,000 texts per setting
Cross-domain800/100/100 sources per domain1,600/200/200 texts per domain
Cross-generator800/100/100 sources per domain, shared across the six generators8,000/1,000/1,000 texts per generator
Boundary localization70%/10%/20% of content-filling source groups within each domain25,779/3,669/7,389 annotated records
Additional experimentsOne human text and one selected machine-involved variant per source, kept in the same partition8,000/1,000/1,000 texts

For generalization experiments, each detector is trained and validated on one setting, domain, or generator, then evaluated without further training on all four settings, five domains (5 × 5 matrix), or six generators (6 × 6 matrix), respectively.

Appendix C.6 details the sampling and preprocessing rules. In cross-collaboration evaluation, 12 sources lack valid rewriting candidates; unused variants from other sources in the same domain and partition fill the shortfall. Cross-domain and cross-generator datasets sample from Machine Rewriting, Prompt-based Generation (L1–L3), and MGT using approximately equal setting quotas. For boundary localization, sentence-level filtering yields 25,643/3,657/7,362 usable sequences, without changing source partitions or WCP fine-tuning pair counts.

Thank you again for your valuable insights, from which we have benefited greatly!

AuthorsOpenReview ↗

[Response to Weakness 2 and Question 2] Boundary-Localization Analysis and Definition of Human-AI Collaboration

Summary of Weakness 2 and Question 2: Please clarify the individual effects of coverage and filling layout on boundary localization, as well as their possible interaction, through systematic comparisons. An explicit definition of human-AI collaboration is also needed.

Response to Weakness2: Thank you for this suggestion. We acknowledge that the original discussion lacked systematic comparisons of coverage, filling layout, and their possible interaction. This was an omission in our analysis. We have added the following observations to Section 4.3 of the revised manuscript and highlight these additions in blue. The comparisons below use the document-level means over each condition’s evaluated samples (Table 12), together with the boundary-count diagnostics in Table 13.

  • At each coverage level, both detectors achieve higher Boundary F1@1 and lower WindowDiff under dispersed filling than under contiguous filling. For WCP, however, sentence-level k is lower under dispersed filling. Its approximately four predicted boundaries per document are closer to the roughly five gold boundaries in dispersed texts than to the single boundary in contiguous texts. This count pattern helps contextualize the higher boundary-matching scores: For WCP, the dispersed-layout advantage in boundary matching does not extend to sentence-level authorship attribution.

  • At higher coverage, WCP performs worse on WindowDiff and sentence-level κ under contiguous filling but better on both under dispersed filling. The corresponding gains in Boundary F1@1 for DeBERTa-BiGRU-CRF are 0.337 and 0.091, respectively. Higher coverage is therefore not consistently associated with better localization across layouts and detectors. Contiguous filling uses one span at both coverage levels, whereas dispersed filling uses 2–3 at low coverage and 3–4 at high coverage. These comparisons remain descriptive: they neither isolate coverage from span count nor establish an independent coverage–layout interaction.

  • The diagnostics further clarify the larger coverage-related gain observed for DeBERTa-BiGRU-CRF under contiguous filling. Under contiguous filling, the proportion of documents with no predicted boundary decreases from 70.90% at low coverage to 31.43% at high coverage. Yet sentence-level k at high coverage remains only 0.005. The large gain in boundary matching therefore coexists with near-chance sentence-level authorship agreement. Higher boundary scores alone would obscure this remaining weakness in recovering the human-AI composition of the text.

Response to Question2: Thank you for highlighting the lack of an explicit definition of human-AI collaboration. In this paper, human-AI collaboration refers to a source-conditioned text-production process in which human-authored material shapes the task or content, while an AI model contributes to producing the resulting text. We have added a clear definition in the Introduction of the revised manuscript and highlight it in blue.

We operationalize this concept through three settings: in Content Filling, the model fills selected spans while the remaining human-written text is retained; in Machine Rewriting, it revises a complete human-written source; and in Prompt-based Generation, it generates new text from content fields automatically extracted from the source, under progressively richer writing constraints. These are controlled source-conditioned settings, rather than records of direct or iterative interaction between the source authors and the models.

MGT also uses human-authored material: the model receives the first 30 tokens of the source and its target length. It is therefore a human-seeded condition, not a zero-human-input condition. In our benchmark, we designate MGT as a prefix-only reference condition, distinct from the three focal collaboration settings, which retained source context with designated gaps, a complete source draft, or source-derived content fields.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers, ACs, SACs, and PCs

We sincerely thank all reviewers as well as ACs, SACs, and PCs for their careful handling of our submission. Based on the constructive suggestions, we provide point-by-point responses:

  1. Motivation and scope of C-HAT-Bench

    • 1.1 Why study Chinese? (Reviewer XNm9). Chinese typically lacks explicit word boundaries. This motivates testing how well detection cues transfer to Chinese writing and how they vary across collaboration conditions.
    • 1.2 Definition of human–AI collaboration (Reviewer iDJX). We study three controlled workflows in which a model fills gaps in a human text, revises a human draft, or writes from source-derived instructions. These workflows span ten conditions. Prefix-conditioned generation (MGT) serves as the reference.
    • 1.3 Relation to existing benchmarks (Reviewers XNm9 and qeyA). MIXSET examines variation across writing operations. C-HAT-Bench also compares levels within each operation using outputs matched by source and generator. Our revised discussion has situated this design alongside multilingual detection benchmarks such as MultiSocial and MULTITuDE and studies of editing extent such as EditLens.
    • 1.4 Human revision of AI drafts (Reviewer qeyA). Beemo studies a complementary workflow in which people edit machine-generated drafts. We plan to add a Chinese subset for this workflow and evaluate detectors on it before final submission.
  2. What the within-operation comparisons reveal

    • 2.1 Document-level detector responses (Reviewers XNm9 and qeyA). For each collaborative output, we compare its detector score with that of an MGT output from the same source and generator. NSS and MAS summarize the direction and magnitude of these changes across filling coverage, rewriting extent, and prompt constraints.
    • 2.2 Coverage and layout in boundary localization (Reviewer iDJX). Increasing coverage raises DeBERTa–BiGRU–CRF’s Boundary F1@1 by 0.337 under contiguous filling and 0.091 under dispersed filling. These are descriptive comparisons because the number of dispersed spans also increases with coverage.
    • 2.3 Sentence-level authorship attribution (Reviewer iDJX). Even after the larger boundary F1 gain under contiguous filling, sentence-level agreement at high coverage remains close to chance (κ = 0.005). The detector thus improves at locating transitions while still struggling to assign authorship to individual sentences.
  3. Detector coverage and cross-domain evaluation

    • 3.1 Additional detectors (Reviewer XNm9). We evaluated benchmark-trained bert-base-chinese, RepreGuard, and the LLP and CLP variants of MGT Probes across all collaboration conditions on a shared test split.
    • 3.2 Cross-domain evaluation protocol (Reviewer qeyA). We reran the cross-domain experiments in Section 4.4 with bert-base-chinese and report AUROC. This evaluates how well detector scores rank texts without selecting a decision threshold from labeled target test samples.
  4. Dataset construction, quality checks, and reproducibility

    • 4.1 Per-source assignments and filling spans (Reviewers iDJX and XNm9). Each source is assigned to two generators for all four filling conditions. For MGT, rewriting, and prompt-based generation, all six generators perform every applicable condition. Appendix A.5 will summarize these assignments, span selection and placement, and planned and retained counts.
    • 4.2 Semantic checks on rewrites (Reviewer XNm9). We assessed source–rewrite semantic similarity across all 59,038 retained pairs, covering both light and deep rewriting.
    • 4.3 Text-length subsets (Reviewer XNm9). We grouped all 240,539 retained machine-involved texts into nine character-length ranges for release, preserving full texts within each subset.
    • 4.4 Data splits and training details (Reviewer iDJX). All variants of a source stay in the same training, validation, or test partition. Appendix C have documented split sizes, training hyperparameters, and detector-fitting settings.

We appreciate your time and careful consideration of our submission. We hope this summary assists you in navigating our rebuttal situation. The revised PDF containing all these improvements has been uploaded.

Best regards,

on behalf of the authors of Paper 22168

AuthorsOpenReview ↗

General Response to Reviewers, ACs, and SACs

Dear Reviewers and Area Chairs,

Thank you for the time and care you have devoted to reviewing our paper. We have now responded to all reviewer comments and uploaded an updated PDF, with revisions highlighted in blue.

Your feedback has guided additional detector experiments, a revised cross-domain evaluation, and more detailed analyses of collaboration conditions. We have also clarified the benchmark’s motivation and scope, dataset construction, quality checks, and reproducibility settings.

When you have an opportunity, we would be grateful for your thoughts on our responses and the updated manuscript. We would be happy to clarify any remaining questions and appreciate your continued consideration.

Best regards,

On behalf of the authors of Paper 22168

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.