- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the theoretical limits of AI-generated text detection using Bayes error and rate–distortion theory.
Critical strengths
The paper addresses an important problem with practical implications. The motivation for uncertainty-aware reporting is reasonable, and I appreciate the authors' willingness to report a negative experimental finding.
Critical weaknesses
W1. Limited novelty and questionable theoretical soundness. The main Bayes-error result is a standard result from statistical hypothesis testing, and the rate–distortion bound follows directly from its definition. And the proposed selective-classification guarantee appears incorrect: a bound on the Bayes-optimal error does not guarantee the error rate of an actual classifier.
W2. Unconvincing experiments and serious citation issues. I found several results are hard to understand, including three detectors achieving exactly 0.500 AUC and rule-based paraphrasing improving AUC from 0.472 to 1.000. More concerningly, I found multiple incorrect references. The arXiv IDs for Fraser et al. (2501.05970), Opara et al. (2502.16105), Przystalski et al. (2502.10218), and Zhang et al. (2502.09861) all correspond to unrelated papers.
Questions for authors
See W1-W2.
We thank the reviewer, and we take W2's citation concern with the seriousness it deserves. We address all points candidly.
Citations (research integrity). The reviewer is correct. We verified the flagged arXiv identifiers and confirm that 2501.05970, 2502.10218, 2502.16105, and 2502.09861 point to unrelated papers. This is a serious error and we take full responsibility. We are conducting a complete audit of every reference, verifying each title, author list, venue, and identifier against the primary source, and replacing or removing any entry we cannot confirm. The claims these four references supported are being re-grounded in verified literature or removed if no correct source exists. We will also correct our AI-use disclosure to accurately reflect how the bibliography was assembled. We are grateful to the reviewer for catching this and are committed to a corrected, fully verifiable reference list. W1 soundness of the selective-risk guarantee. This is a valid and important correction. The reviewer is right that a bound on the Bayes-optimal error does not bound the error of an actual classifier. Our protocol has two distinct components, and we conflated them. (1) The conformal coverage guarantee Pr[y ∈ C(x)] ≥ 1−β is valid, distribution-free, and holds for any detector; it is a statement about the prediction set, not the Bayes floor. (2) The TV-based statement ½(1−TV) ≤ η bounds only the best achievable error on the retained region; it is an information-theoretic floor, not a guarantee on a deployed detector's realized error, and we incorrectly presented it as the latter. We have rewritten §5 to state (1) as the operative guarantee, to label (2) explicitly as a floor on achievable (not realized) error, and to back all deployment claims with measured realized selective accuracy. On a held-out split the protocol leaves 18.5% undecided with 62.6% decided-accuracy at β=0.30, rising to 80.7% undecided / 75.5% accuracy at β=0.05, with empirical coverage meeting 1−β throughout. These are realized numbers, not Bayes bounds. W1 novelty. We agree the Bayes-error result is standard and the rate–distortion inequality follows from its definition; we do not claim either as our contribution. Our intended contributions are the deployable calibrated protocol (now stated with the corrected guarantee) and the pre-registered negative result on the ΔC·ΔS trade-off. We have rewritten the contributions list to avoid overstating the theory.
W2 experiments. The reviewer is right that these were unconvincing. The three detectors at exactly 0.500 were placeholders for zero-shot logit methods requiring white-box generator access we lacked; we have removed them and state the constraint. The rule-based paraphrase result (0.472→1.000, CI [1.000,1.000]) is a degenerate artifact a systematic textual signature from the rule-based transform combined with a very small attack subset and is not a meaningful robustness measurement; we have removed it and retain the semantic paraphrase attacks (DIPPER 0.860; back-translation 0.472), which are the informative ones.
We thank the reviewer for a rigorous review that has materially improved the paper's correctness and integrity.