- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes CoMa-LD for document-level detection and sentence-level localization of LLM-generated text. It combines conformal evidence with a two-state Human/LLM switching model. Forward aggregation provides document-level alarms, while post-alarm Viterbi decoding estimates sentence labels. The paper provides false-alarm guarantees under stated exchangeability assumptions and introduces document-level calibration to accommodate dependence within documents.
Critical strengths
The shared Human/LLM sequence model provides a principled way to combine document-level detection with sentence-level localization, and the ablations support the contribution of both components. Document-level calibration is a further strength: it allows dependence among sentences while controlling false alarms under exchangeability across documents, an important consideration for authorship assessment.
Critical weaknesses
Limited external validity of the mixed-authorship evaluation. All evaluated mixed documents are constructed by replacing alternating human-written blocks with LLM-generated continuations (Appendix F.1). This leaves an important gap between the evaluation and the motivating applications of revision, rewriting, and human AI collaboration: the results do not establish whether the gains persist when humans select and edit AI suggestions or contributions are intertwined within sentences. Actual collaborative-writing data exists: for example, CoAuthor, although its mixed-authorship sentences require an explicit labeling policy for this binary framework. Without such evaluation, the practical scope of the claimed detection and localization improvements remains uncertain.
Questions for authors
-
Clarification of calibration data. What human reference/calibration data and threshold-selection procedures are used for MCP and Online Betting? In particular, do these baselines receive access to a comparable calibration-data budget to CoMa-LD_DC, which uses 500 additional human documents? This affects whether the reported gains reflect the method or differences in calibration resources.
-
Interpretation of the detection improvement. In Table 5, CoMa-LD reduces missed detections on WritingPrompts while increasing FPR from 0.063 to 0.080. Does the detection advantage persist at comparable false-alarm rates, using thresholds selected on held-out calibration data? This would clarify how much improvement comes from the switching detector versus the operating point.
4 comments
Reply to Reviewer zMV1 (2/2)
- (Q2) “In Table 5, CoMa-LD reduces missed detections on WritingPrompts while increasing FPR from 0.063 to 0.080. Does the detection advantage persist at comparable false-alarm rates, using thresholds selected on held-out calibration data? This would clarify how much improvement comes from the switching detector versus the operating point.”
Thank you for pointing out this important distinction. We have replaced the original detector-ablation table with Figure 7, which compares CTM and CoMa-LD across , using the same 100 human reference sentences and Viterbi localizer. As shown in Figure 7, CoMa-LD achieves lower miss rates and WindowDiff than CTM at comparable empirical FPRs on both datasets. These results suggest that the detection improvement is not solely due to a higher false-alarm operating point. For example, on WritingPrompts, CoMa-LD at achieves a lower miss rate ( vs. ) and lower WindowDiff ( vs. ) than CTM at , while operating at a slightly lower empirical FPR ( vs. ).
The human reference sources are disjoint from the test sources (Appendix F.2). The thresholds are determined by the nominal significance level , rather than fitted using a separate held-out calibration set. We have updated the discussion in Section 4.5 accordingly.
Thank you for the additional experiments, especially on CoAuthor. I appreciate the authors taking the time to address my concern. I still feel that the connection to real-world use is a bit unclear, since people might edit or paraphrase AI suggestions and mix them with their own writing in different ways. The CoAuthor results help, but the detection rate is still quite low, and it is hard to tell much about false positives from the limited human-only data. I appreciate the authors acknowledging these limitations. I will maintain my score.
Thank you for your follow-up and for taking the time to consider our additional experiments. We understand your concerns regarding real-world applicability and false-positive control. Importantly, CoMa-LD achieves the best performance among the evaluated methods, with a document-level TPR of 0.332 (compared with 0.177 for the strongest baseline) and the lowest WindowDiff of 0.351. We believe these results demonstrate meaningful improvements in this challenging collaborative-writing setting, while acknowledging the remaining limitations. Thank you again for your constructive feedback, which helped us strengthen the paper!
Reply to Reviewer zMV1 (1/2)
Thank you for your encouraging comments and constructive feedback. Your comments helped us clarify the calibration budgets, compare detection at similar false-alarm rates, and extend the evaluation to real collaborative writing. We respond to each point below.
Thank you for highlighting this concern and suggesting CoAuthor. We agree that evaluating real human–AI collaborative writing is important beyond our controlled experiments.
Following your suggestion, we evaluated CoMa-LD on CoAuthor collaborative-writing documents, using 100 human reference sentences and Gemma-3-1B-PT for scoring. For mixed-authorship sentences, we adopt a binary labeling policy: a sentence is labeled as AI-positive if it contains any AI-generated contribution, including sentences with both human and AI contributions.
CoMa-LD achieves a higher document-level TPR than Fast-DetectGPT and Online Betting, and a lower WindowDiff than VCP and WCP_len. These results provide further evidence that CoMa-LD's advantages over the evaluated baselines extend to this collaborative-writing setting, although the overall detection sensitivity remains limited.
One limitation of this evaluation is the small number of human-only documents. After filtering, only 37 documents remain, all from a single writer. This limits our ability to reliably assess false positives across different human authors, so we do not report FPR for this experiment.
We have also added experiments on multi-editor Wikipedia text (Section 4.4, Table 5) and LLM-polished human text (Appendix G.6). These experiments further examine the effects of multiple human authors and local AI edits to human-written text. Finally, we have clarified the limitations of binary sentence-level labeling in Appendix H, as it does not distinguish between different amounts of human and AI contributions within a sentence. Finer-grained detection and localization remain directions for future work.
Thank you for asking us to clarify the calibration budgets. MCP uses 500 human documents for threshold calibration, matching CoMa-LD$_{\mathrm{DC}}$. Original Online Betting shares CoMa-LD's 100 human reference sentences and uses its theoretical threshold. Calibration and test sources are disjoint (Appendix F.2).
To ensure a more comparable calibration budget, we have additionally evaluated Online Betting + DC, which uses 500 human documents to calibrate the threshold on the maximum wealth over sentence prefixes at α=0.05. The table below reports TPR at K=1 and human-only FPR, while Appendix G.5, Table 16, provides the complete results for K∈{1,2,3,5}.
Across the full comparison, CoMa-LD$_{\mathrm{DC}}$ has the highest TPR in seven of eight settings and the lowest empirical FPR on both datasets. The only exception in TPR is CNN/DailyMail at K=5, where MCP performs best. The gains therefore persist under equal document-calibration budgets.