acceptodds
Desk rejected at ICLR 2027

CATS: Power-Optimized Conformal Selection for AI-Text Detection

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Detecting machine-generated text is critical for manuscript review, academic-integrity enforcement, and content moderation, where each accusation carries direct consequences for the flagged author. However, existing methods predominantly rely on heuristic detection scores without statistical guarantees; even when calibrated to control the false positive rate (FPR), such fixed-threshold rules leave the composition of the selected set unconstrained, risking an output dominated by falsely accused human texts. In this work, we formalize machine-generated text detection as a candidate selection problem and propose Conformal AI-Text Selection (CATS), a distribution-free framework that controls the false discovery rate (FDR) in finite samples, ensuring the expected proportion of human texts within the selected set stays below a user-specified level q. Specifically, CATS estimates the selection threshold from pooled calibration and candidate data through a permutation-equivariant map and learns a score adjustment by maximizing a local surrogate for selection power. Theoretically, we prove that this pooled permutation equivariance preserves the super-uniformity of null conformal p-values, thereby guaranteeing finite-sample FDR control. Extensive experiments across ten benchmark datasets and nine detection scores demonstrate that CATS consistently maintains valid FDR control below the target level q while improving selection power over the unadjusted baseline, with an average relative gain of 38% in the main comparison.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine-generated text detection, Large language models, DetectGPT

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_eGHQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CATS, a conformal-based AI-generated text selection framework that reformulates the conventional binary classification task as a candidate selection problem under false discovery rate (FDR) constraints. Specifically, CATS employs conformal p-values and the Storey–BH procedure to control the expected proportion of falsely flagged human texts within the selected set. Meanwhile, it adaptively adjusts existing detection scores through permutation-equivariant boundary estimation and local selection power optimization, aiming to improve the recall of AI-generated texts under a fixed FDR constraint. The effectiveness of the proposed method is evaluated on 10 data subsets from the MAGE benchmark using 9 different detection scores.

Critical strengths
  1. This paper investigates AI-generated text detection from the perspective of FDR control, an evaluation criterion that may have received relatively limited attention in this research area. This provides a novel perspective on the reliability of AI-generated text detection.

  2. The paper provides relatively comprehensive theoretical analysis and establishes finite-sample FDR control guarantees under specific exchangeability assumptions. Furthermore, the experiments cover multiple datasets and different types of detection methods, demonstrating that CATS can effectively improve selection power while maintaining FDR control.

Critical weaknesses
  1. Motivation and the practical implications of FDR control. Lines 117–127 present a fundamental motivation of this paper: even when the FPR is relatively low, more than half of the flagged texts may still be human-authored, highlighting the necessity of controlling the FDR. However, this argument is based on an imbalanced detection scenario in which human-authored texts substantially outnumber AI-generated texts. If the proportion of AI-generated texts becomes comparable to or exceeds that of human-authored texts, a different concern may arise: even when the FDR appears very low, a substantial number of human-authored texts could still be incorrectly flagged as AI-generated. In other words, a low FDR does not necessarily imply a low FPR, and a high FPR may lead to serious consequences for innocent authors. Therefore, whether FDR control alone sufficiently supports the practical significance emphasized in the first paragraph of the Introduction remains open to discussion.

  2. Limited benchmark coverage. The experimental evaluation could be further strengthened by incorporating additional widely used public benchmarks, such as DetectRL, RAID, and RealDet. Evaluations on these datasets would help provide more comprehensive empirical evidence and strengthen the persuasiveness of the study.

  3. Selection of baseline methods. The baseline comparisons could be expanded to include more recent and representative detection methods. For statistical-based detectors, commonly adopted methods from the past two years, such as Binoculars and DNA-DetectLLM, could be considered. For supervised detectors, additional comparisons with methods such as DeTeCtive and Biscope would help improve the comprehensiveness of the evaluation.

  4. Potential trade-off between FDR and FPR. While the experiments demonstrate improved selection power under controlled FDR, it remains unclear whether the corresponding changes in FPR have been sufficiently examined. Should FPR also be explicitly monitored when optimizing selection power under an FDR constraint? More importantly, does controlling FDR necessarily prevent the FPR from becoming excessively high? Further analysis of the relationship between these two error metrics would help clarify the potential risks of the proposed framework in practical applications.

  5. Robustness under text perturbations and varying input lengths. The experimental evaluation could be further extended to consider adversarial text attacks and different input sequence lengths. Additional robustness experiments under these settings would help provide a more comprehensive assessment of the proposed method.

Questions for authors

See Weakness.

6 comments

Thank you for your valuable time and constructive suggestions. We address each concern below.

1. Machine-text proportion in auditing and the FPR under FDR control [W1]

Thank you for this insightful comment. We would like to first clarify the scenario our work targets. We focus on high-stakes auditing, such as manuscript review and coursework auditing, where downstream actions are applied to the flagged texts. Because the use of machine-generated text is restricted and subject to sanctions in these scenarios, machine-generated texts are expected to form a minority of the candidate pool, as supported by recent large-scale studies of scientific writing [1,2]. The imbalanced setting in Lines 117–127 therefore reflects this auditing scenario, where a low FPR can still leave the selected set dominated by human texts and the auditor instead needs to bound the FDR.

Moreover, the FDR guarantee of our method also bounds the FPR. Consider a candidate pool of size mm containing m0=π0mm_{0}=\pi_{0}m human texts. Since 1∨∣S∣≤m1\vee|\mathcal{S}|\le m in Eq. (4), we have

E[FPR]=E[∑j=1m1{j∈S, H0,j is true}m0]≤E[FDP⋅mm0]=FDRπ0≤qπ0.\mathbb{E}[\mathrm{FPR}]=\mathbb{E}\left[\frac{\sum_{j=1}^{m}\mathbf{1}\lbrace j\in\mathcal{S},\thinspace H_{0,j}\text{ is true}\rbrace}{m_{0}}\right]\le\mathbb{E}\left[\frac{\mathrm{FDP}\cdot m}{m_{0}}\right]=\frac{\mathrm{FDR}}{\pi_{0}}\le\frac{q}{\pi_{0}}.

The expected FPR of our method is therefore at most 2q2q whenever human texts make up at least half of the candidate pool. The reverse does not hold, since a fixed-threshold rule with FPR=α\mathrm{FPR}=\alpha places no bound on the FDR when machine texts are rare, which is the expected case in high-stakes auditing.

When the proportion of machine-generated texts becomes comparable to or exceeds that of human texts, the auditor can control the FPR at level qq for any π0\pi_{0} by omitting the Storey estimator in Eq. (7). The selection threshold qkq∗/mqk_{q}^{\ast}/m then never exceeds qq, so every selected text satisfies pj≤qp_{j}\le q. The super-uniformity of null conformal pp-values in Eq. (31) then gives

E[FPR]=1m0∑j∈H0P(j∈S)≤1m0∑j∈H0P(pj≤q)≤q,\mathbb{E}[\mathrm{FPR}]=\frac{1}{m_{0}}\sum_{j\in\mathcal H_{0}}\mathbb{P}(j\in\mathcal{S})\le\frac{1}{m_{0}}\sum_{j\in\mathcal H_{0}}\mathbb{P}(p_{j}\le q)\le q,

while the FDR remains controlled at π0q≤q\pi_{0}q\le q. Without the Storey estimator, our method thus controls both the FDR and the FPR at level qq, with a more conservative threshold. We hope this discussion addresses your concern.

2. Evaluation on more public benchmarks [W2]

Thank you for the suggestion. We add new experiments on DetectRL [3], RAID [4], and RealDet [5], using the same nine detection scores and settings as the main experiments. The table below reports the selection power over 500 repetitions, averaged over the candidate pools and detection scores of each benchmark (formatted as baseline / our method):

Benchmarkq=0.05q=0.05q=0.10q=0.10q=0.20q=0.20q=0.30q=0.30
DetectRL0.363 / 0.4120.423 / 0.4890.500 / 0.5950.559 / 0.681
RAID0.650 / 0.7110.718 / 0.7830.824 / 0.8970.894 / 0.950
RealDet0.338 / 0.4720.439 / 0.5920.573 / 0.7760.687 / 0.876

Our method improves selection power on all three benchmarks at every target level. The results for each detection score are reported in Table 15 in Appendix D.3 of the revised version.

3.Additional detection scores [W3]

Thank you for the suggestion. We add four recent detection scores, Binoculars [6], DNA-DetectLLM [7], DeTeCtive [8], and BiScope [9], and evaluate them on RealDet [5]. The table below reports the selection power over 500 repetitions (formatted as baseline / our method):

Detection scoreq=0.005q=0.005q=0.01q=0.01q=0.05q=0.05q=0.10q=0.10
Binoculars0.254 / 0.2700.481 / 0.5520.800 / 0.8310.869 / 0.876
DNA-DetectLLM0.259 / 0.2670.479 / 0.5240.801 / 0.8290.874 / 0.886
DeTeCtive0.426 / 0.4910.814 / 0.8430.989 / 0.9890.995 / 0.995
BiScope0.288 / 0.4240.536 / 0.6280.768 / 0.8070.850 / 0.888

Our method improves or matches selection power in all settings, and the empirical FDR stays below qq in all settings. Detailed results are reported in Appendix D.4 of the revised version.

4. Relationship between the FDR and the FPR [W4]

Thank you for this question. Following the notation in our response to W1, the FPR and the FDP count the same falsely selected human texts but divide them by m0m_{0} and 1∨∣S∣1\vee|\mathcal{S}|, respectively, so that FPR=FDP⋅(1∨∣S∣)/m0\mathrm{FPR}=\mathrm{FDP}\cdot(1\vee|\mathcal{S}|)/m_{0}. Since 1∨∣S∣≤m1\vee|\mathcal{S}|\le m, the FDR guarantee bounds the expected FPR:

E[FPR]≤FDRπ0≤qπ0,\mathbb{E}[\mathrm{FPR}]\le\frac{\mathrm{FDR}}{\pi_{0}}\le\frac{q}{\pi_{0}},

which is at most 2q2q whenever human texts make up at least half of the candidate pool. The reverse does not hold, since a fixed-threshold rule with FPR=α\mathrm{FPR}=\alpha places no bound on the FDR when machine texts are rare, which is the expected case in high-stakes auditing. Moreover, the auditor can control the FPR at level qq for any π0\pi_{0} by omitting the Storey estimator in Eq. (7), as shown in our response to W1. We hope this clarifies the relationship between the two error rates.

5. Robustness to adversarial attacks and input length [W5]

Thank you for the suggestion. We add new experiments on adversarial attacks and input length using the same nine detection scores over 500 repetitions. Each table reports the selection power averaged over candidate pools and detection scores (formatted as baseline / our method).

Adversarial attacks. On the candidate pools of RAID [4] and DetectRL [3] in our response to W2, we replace the machine texts with their attacked versions. Each DetectRL row averages the attacks of one category, namely two prompt, three paraphrase, and three perturbation attacks. The results are provided below:

DatasetAttackq=0.05q=0.05q=0.10q=0.10q=0.20q=0.20q=0.30q=0.30
RAIDParaphrase0.376 / 0.4560.469 / 0.5750.589 / 0.7340.687 / 0.822
RAIDSynonym0.402 / 0.4900.501 / 0.6120.641 / 0.7570.752 / 0.842
RAIDHomoglyph0.380 / 0.4460.489 / 0.5660.665 / 0.7330.753 / 0.816
DetectRLPrompt0.332 / 0.3800.399 / 0.4670.476 / 0.5760.537 / 0.658
DetectRLParaphrase0.241 / 0.2830.301 / 0.3660.375 / 0.4730.431 / 0.555
DetectRLPerturbation0.057 / 0.0620.078 / 0.0960.111 / 0.1580.144 / 0.217

Input length. On the varying-length task of DetectRL [3], we group the texts into four candidate pools by word count. The results are provided below:

Length (words)q=0.05q=0.05q=0.10q=0.10q=0.20q=0.20q=0.30q=0.30
<100< 1000.015 / 0.0200.042 / 0.0660.084 / 0.1570.127 / 0.273
100–2000.132 / 0.1540.175 / 0.2450.226 / 0.3870.281 / 0.543
200–3000.222 / 0.2750.276 / 0.3990.386 / 0.5940.489 / 0.686
300–3600.406 / 0.4590.494 / 0.6160.593 / 0.7570.652 / 0.813

Across all attack types and input lengths, our method consistently improves selection power over the baseline at every target level qq. Detailed results are reported in Appendix D.5 of the revised version.

References

[1] "Quantifying large language model usage in scientific papers" (Nature Human Behaviour, 2025)

[2] "Delving into LLM-assisted writing in biomedical publications through excess vocabulary" (Science Advances, 2025)

[3] "DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios" (NeurIPS, 2024)

[4] "RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors" (ACL, 2024)

[5] "Reliably Bounding False Positives: A Zero-Shot Machine-Generated Text Detection Framework via Multiscaled Conformal Prediction" (ACL, 2025)

[6] "Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text" (ICML, 2024)

[7] "DNA-DetectLLM: Unveiling AI-Generated Text via a DNA-Inspired Mutation-Repair Paradigm" (NeurIPS, 2025)

[8] "DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning" (NeurIPS, 2024)

[9] "BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens" (NeurIPS, 2024)

Thank you for the comprehensive response and the additional experiments. However, I still have one major concern: while FDR is undoubtedly an important consideration, CATS does not yet seem to have demonstrated its practical utility. In most experiments, the calibration and test sets are tightly coupled in terms of their underlying distributions. Under this setting, many existing calibration methods can also improve performance while providing statistical guarantees.

In real-world detection scenarios, however, the test distribution is generally unknown in advance. How should practitioners determine the distribution of human-written texts used to construct the calibration set? I consider this question central to the practical applicability of the proposed method and would appreciate further discussion.

Reviewer_KiSXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies AI-generated text detection from a set-selection perspective. The authors argue that conventional detector calibration based on false positive rate (FPR) can be inadequate for high-stakes auditing: when human-written texts dominate the candidate pool, even a detector with a low FPR may produce a selected set containing a large fraction of human texts. The paper therefore formulates AI-text detection as a multiple-testing problem and targets false discovery rate (FDR), i.e., the expected proportion of human-written texts among the selected texts.

The proposed method, CATS, converts an existing detector score into conformal p-values and applies a Storey-BH procedure to obtain an FDR-controlled selected set. To improve utility under a fixed FDR budget, CATS further learns a score adjustment optimized for selection power. An important component is a pooled permutation-equivariant construction over calibration and candidate samples, which is designed to preserve null exchangeability. Under the stated exchangeability assumptions, the paper provides a finite-sample FDR guarantee. Experiments across multiple datasets and detector scores show that CATS generally preserves empirical FDR control while improving selection power over using the original detector scores directly.

Critical strengths

1. A meaningful reformulation of AI-text detection.\textbf{1. A meaningful reformulation of AI-text detection.} I find the distinction between FPR control and FDR control well motivated. For auditing a pool of texts, the quantity of practical interest is arguably not only how often a human text is individually misclassified, but also how reliable the resulting flagged set is. The example in the introduction clearly illustrates that low FPR does not necessarily imply a low proportion of false accusations among selected texts.

Critical weaknesses

1. The methodological novelty relative to existing work.\textbf{1. The methodological novelty relative to existing work.} The methodological novelty relative to existing work on conformal selection and power-optimized conformity scores needs to be made sharper. Conformal p-values combined with BH-type procedures for FDR-controlled selection are established tools, and the paper also discusses prior work that directly optimizes conformity scores for selection power. My understanding is that the main methodological distinction of CATS is that its pooled permutation-equivariant adjustment only requires exchangeability between calibration samples and null candidates, rather than stronger exchangeability across the complete calibration and candidate distributions. If this is the key novelty, I would like the paper to make the comparison more explicit. In particular, it would help to clarify which component or guarantee of CATS cannot be obtained by directly adapting prior optimized conformal-selection methods to this setting.

Questions for authors

1. Difference from optimized conformal selection.\textbf{1. Difference from optimized conformal selection.} Can the authors state more precisely the technical difference between CATS and prior work on optimizing conformity scores for selection power? Which assumption, theorem, or algorithmic component is genuinely enabled by the pooled permutation-equivariant construction and is unavailable in those approaches?

3 comments

Thank you for your positive feedback and insightful question. We address your concern below.

1. Difference from optimized conformal selection [W1, Q1]

Thank you for the summary. We would like to clarify that our method and prior work [1,2] address different settings. Prior work selects test units whose unobserved continuous response exceeds a threshold, i.e., Y>cY > c, and requires the labeled calibration data and the test data to be exchangeable as pairs (X,Y)(X, Y). Our method instead calibrates with human texts, and its guarantee relies on exchangeability between the calibration texts and the human texts in the candidate pool. The table below summarizes the difference.

AspectPrior work [1,2]Our method
Selection targetTest units with Y>cY > cMachine texts in the candidate pool
Calibration dataLabeled data of both classesHuman texts only
Exchangeability for FDR controlLabeled pairs (X,Y)(X, Y) across calibration and test dataCalibration texts and human candidates

In our setting, exchangeability of the pairs (X,Y)(X, Y) would require the calibration set to contain machine and human texts in the same proportion as the candidate pool, which an auditor cannot know. Our method instead learns a score adjustment by maximizing a local surrogate for selection power around the selection threshold of the candidate pool. Because this threshold depends on the candidate pool, we leverage pooled permutation equivariance and estimate it from the unlabeled multiset U=Dcal∪Dtest\mathcal{U} = \mathcal D_{\mathrm{cal}} \cup \mathcal D_{\mathrm{test}}, so that the calibration texts and the human texts in the candidate pool remain exchangeable after the score adjustment. We have clarified this distinction in the Related Work section of the revised version.

References

[1] "Optimized conformal selection: Powerful selective inference after conformity score optimization" (arXiv, 2024)

[2] "Selection by prediction with conformal p-values" (JMLR, 2023)

Thank the authors for their detailed response. My concerns have been addressed, and I maintain my positive rating.

Thank you for taking the time to review our response. We are pleased that it addressed your concerns, and we appreciate your positive assessment.

Reviewer_GmC1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formalizes machine-generated text detection as a candidate selection problem and proposes Conformal AI-Text Selection (CATS), and the theoretical contribution is also meaningful. The experiment results, with an average relative gain of 38% in the main comparison, also show the effectiveness of the proposed method.

Critical strengths

This paper formalizes text detection as selection with FDR guarantees, which is practically meaningful. And experiments across 10 datasets and 9 detection scores demonstrate reliable empirical FDR control. The calibration-set proportion analysis is informative, and the limitations are discussed candidly.

Critical weaknesses
  1. OptCS is cited but not included in the experiment comparisons, so the proposed method’s significance relative to OptCS remains unclear.
  2. Binoculars is missing from the evaluation. Gains may be concentrated on weaker detection scores, so reporting the gains separately for stronger and weaker detection scores would clarify the method’s practical benefit.
Questions for authors

Could you provide an intuitive explanation of why learning the score adjustment from pooled data containing AI-generated texts does not violate exchangeability between calibration texts and human texts in the candidate pool?

3 comments

Thank you for the positive comments and constructive suggestions. We address each concern below.

1. Comparison with OptCS [W1]

Thank you for the helpful comment. We would like to clarify that OptCS [1] is not directly applicable to our auditing setting. It targets the selection of test units whose unobserved continuous response exceeds a threshold [2], i.e., Y>cY > c, and requires that the labeled calibration data and the test data be exchangeable as pairs (X,Y)(X, Y). In our setting, this would require the calibration set to contain machine and human texts in the same proportion as the candidate pool, which is impossible to know in auditing. Our method instead calibrates with human texts, and its guarantee relies on exchangeability between the human calibration texts and the human texts in the candidate pool. We have made this distinction explicit in the Related Work section of the revised version.

2. Evaluation with Binoculars [W2]

Thank you for the suggestion. We add a new experiment with Binoculars [3], using Falcon-7B as the observer model and Falcon-7B-instruct as the performer model. The table below reports the selection power (formatted as baseline / our method):

Domainq=0.05q=0.05q=0.10q=0.10q=0.20q=0.20q=0.30q=0.30
ChangeMyView0.968 / 0.9750.977 / 0.9820.984 / 0.9870.986 / 0.990
WritingPrompts0.944 / 0.9580.959 / 0.9680.970 / 0.9780.973 / 0.982
XSum0.882 / 0.9040.913 / 0.9260.933 / 0.9450.944 / 0.956

Our method improves selection power in all settings, and the empirical FDR stays below qq in all settings. We also report the gains separately for stronger and weaker detection scores at q=0.10q=0.10 by grouping all settings in the main experiment by their baseline selection power:

Baseline power# SettingsBaselineOur methodGain
≤0.3\le 0.3460.0770.191+0.114
0.3–0.6300.4800.614+0.135
0.6–0.8150.7120.806+0.095
>0.8> 0.8170.8870.905+0.018

Our method improves selection power at every level of baseline power, with the largest gain when the baseline power is between 0.3 and 0.6. As the baseline power approaches 1, the gain becomes smaller. These results are reported in Appendix D.7 of the revised version.

3. Intuition for exchangeability under the pooled score adjustment [Q1]

Thank you for the question. To see the role of pooling, it helps to first consider what happens without it: estimating the cutoff from Dtest\mathcal D_{\mathrm{test}} alone treats calibration texts and human candidates asymmetrically. For example, consider a teacher grading essays from two classes, where Class A (Dcal\mathcal D_{\mathrm{cal}}) has only student-written essays and Class B (Dtest\mathcal D_{\mathrm{test}}) also contains some machine-generated essays. If the teacher sets the grading rubric after reading only Class B and then applies it to Class A, the student essays in Class B influence the rubric whereas those in Class A do not. The rubric may then inadvertently favor patterns specific to Class B and break exchangeability between the two classes.

By contrast, pooling removes the set labels before learning the adjustment, so the machine texts in Dtest\mathcal D_{\mathrm{test}} affect calibration texts and human candidates in the exact same way. In this example, the teacher removes the class names, shuffles all essays from Class A and Class B into one pile, sets the rubric from the whole pile, and grades every essay before checking the class names. Although the machine-generated essays influence the rubric, the teacher does not know which student essays came from Class A and which came from Class B, so students of equal ability in both classes are graded by the exact same rule. Swapping a student essay between Class A and Class B leaves the pile and the rubric unchanged, so the two essays simply exchange their scores and remain exchangeable. We hope this example helps convey the intuition.

References

[1] "Optimized conformal selection: Powerful selective inference after conformity score optimization" (arXiv, 2024)

[2] "Selection by prediction with conformal p-values" (JMLR, 2023)

[3] "Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text" (ICML, 2024)

Thank you for the clear response and the additional experiments. The Binoculars results and the breakdown by baseline power have addressed my concern well about gains with stronger detection scores. I also appreciate the clarification of why OptCS is difficult to apply directly in this setting and how the pooled score adjustment preserves exchangeability under the stated assumptions. I will maintain my original rating of 3.

Thank you for taking the time to follow up. We are glad that our response addressed your concerns and appreciate your continued support.

Reviewer_mHcWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work formalizes machine-generated text detection as a candidate selection problem and proposes Conformal AI-Text Selection (CATS), a distribution-free framework that controls the false discovery rate (FDR) in finite samples, ensuring the expected proportion of human texts within the selected set stays below a user-specified level. Specifically, CATS estimates the selection threshold from pooled calibration and candidate data, and learns a score adjustment by maximizing a local surrogate for selection power. Extensive experiments across various benchmarks and detectors demonstrate that CATS consistently maintains valid FDR control below the target level while improving selection power over the baseline, with an average relative gain of 38%.

Critical strengths
  • The paper is overall well written and easy to follow.
  • The paper provides a theoretical guarantee of FDR control under the stated assumptions, i.e., we have human-written calibration texts which are exchangeable to human-written texts in a target pool. The proposed method optimizes detection scores to improve selection power while preserving this guarantee.
  • The paper provides comprehensive experiments, covering diverse domains, generators, and detectors, demonstrating superior classification power via calibration.
  • Various sensitivity analysis supports the validity of the proposed method, including especially the effect of variation in target machine-generated text, impact of calibration data size, and cross-generator transferability.
Critical weaknesses
  • Overall, I see the motivation of the paper seems a bit weak. They claim the superiority of FDR by taking examples where the precision could be low even if we achieve FPR=5%. However, most studies and further especially commercial AI text detectors often set the much lower FPR like 0.1% and high scores on other metrics (e.g., precision, recall, F1) as goals.
  • The proposed framework (CATS) is to identify machine-generated text from candidates, but it seems less practical given that many realistic scenarios could be detecting a single text. I further expect the guarantee for FPR could be weaker in that scenario. Providing a discussion on such other usecases of CATS would strengthen the paper.
  • The CATS framework requires the strong assumption that we have human-written calibration texts which are exchangeable to human-written texts in a target pool. It may not hold in practice, when we don’t have prior knowledge on what text comes. While the paper assesses cross-generator transferability, it does not evaluate distribution shifts in human-written text, which directly affects the validity of its theoretical guarantee.
Questions for authors

See the weakness part

2 comments

Thank you for your thorough review and constructive feedback. We address each concern below.

1. Motivation for FDR control versus low-FPR thresholds [W1]

Thank you for raising this important point. We would like to clarify that controlling the FPR and controlling the FDR target different risks. The FPR measures the overall fraction of falsely flagged texts among all human texts in the candidate pool, whereas the FDR measures the expected fraction of human texts among the texts flagged as machine-generated. In high-stakes auditing, downstream actions such as manuscript rejection or disciplinary sanctions are applied directly to the selected set, so the auditor needs to bound the proportion of wrongful accusations among the flagged texts.

For a candidate pool with machine-text proportion π1\pi_{1}, the FDR under a fixed threshold is given by FDR≈(1−π1)FPR(1−π1)FPR+π1TPR\mathrm{FDR} \approx \frac{(1-\pi_{1})\mathrm{FPR}}{(1-\pi_{1})\mathrm{FPR} + \pi_{1}\mathrm{TPR}}. For example, at FPR=0.1\mathrm{FPR}=0.1% and TPR=90\mathrm{TPR}=90%, 9.99.9% of the flagged texts are human when π1=1\pi_{1}=1%, nearly 100100 times the nominal 0.10.1% FPR level. Since π1\pi_{1} is unknown, the FPR alone cannot determine the expected proportion of wrongful accusations among the flagged texts, whereas our method bounds it at a user-specified level. Moreover, metrics such as precision, recall, F1, or TPR at 0.1% FPR are evaluation metrics used to benchmark a binary classifier on a labeled test set, whereas our method is designed to select a subset from an unlabeled candidate pool with a theoretical guarantee on the FDR. We hope this clarifies the motivation and addresses your concern.

2. Guarantee when testing a single text [W2]

Thank you for the question. We would like to clarify that the guarantee does not become weaker for a single text. With a single text, the multiple testing problem reduces to testing a single hypothesis H0,1H_{0,1}. The FDR then reduces to the Type-I error of this test:

FDR=E[∑j=1m1{j∈S, H0,j is true}1∨∣S∣]=m=1P(1∈S∣H0,1 is true)⏟Type-I error=P(p1≤q∣H0,1 is true)≤q,\mathrm{FDR}=\mathbb{E}\left[\frac{\sum_{j=1}^{m}\mathbf{1}\lbrace j\in\mathcal{S},\thinspace H_{0,j}\text{ is true}\rbrace}{1\vee|\mathcal{S}|}\right]\overset{m=1}{=}\underset{\text{Type-I error}}{\underbrace{\mathbb{P}(1\in\mathcal{S}\mid H_{0,1}\text{ is true})}}=\mathbb{P}(p_{1}\le q\mid H_{0,1}\text{ is true})\le q,

where the second equality holds when H0,1H_{0,1} is true because ∣S∣=1|\mathcal{S}|=1 whenever the text is flagged, and the third holds because we skip the Storey estimator π^0+\widehat\pi_{0}^{+} for a single text. This means that the selection threshold in this case is the same as the threshold that controls the FPR at level qq.

3. Distribution shift in human-written text [W3]

Thank you for raising this point. We agree that the FDR guarantee relies on exchangeability between the final human calibration set and the human texts in the candidate pool, as noted in our Limitations section. In practice, auditors can satisfy this by matching the calibration set to the target audit domain. For example, a university auditing student coursework or a publisher reviewing manuscripts can construct the calibration set from historical human-written archives from the same institution or venue prior to the widespread adoption of LLMs. Importantly, this data requirement is consistent with existing AIGC detectors that calibrate thresholds on a reference set for FPR control [1,2], as well as with supervised, learned-statistic, or two-sample detection methods [3,4,5] that learn from reference data of both human and machine-generated text.

We also add a new experiment on distribution shift in human-written text. Under covariate shift between the calibration and candidate human texts, we can account for the distributional difference by incorporating weighted conformal pp-values [6,7] into our method. Specifically, on WritingPrompts and XSum with GPT-3.5 Turbo, let z(x)z(x) be the standardized log⁡(1+token length)\log(1+\text{token length}). We assign each human text to Dref\mathcal D_{\mathrm{ref}} with probability σsig(a−z(x))\sigma_{\mathrm{sig}}(a-z(x)) (where the intercept aa preserves the reference-to-candidate split ratio) and to Dtest\mathcal D_{\mathrm{test}} otherwise, shifting Dtest\mathcal D_{\mathrm{test}} toward longer texts. Estimating the density-ratio weights w^(x)\widehat{w}(x) via a cross-fitted logistic classifier that separates the unlabeled candidate pool from the human texts in Dref\mathcal D_{\mathrm{ref}} based on z(x)z(x), we replace Eq. (6) with the weighted conformal pp-value:

pjw(sq)=w^(Xjtest)+∑i=1ncalw^(Xical)1{sq(Xical)≥sq(Xjtest)}w^(Xjtest)+∑i=1ncalw^(Xical).p_{j}^{\mathrm{w}}(s_{q}) = \frac{\widehat{w}(X_{j}^{\mathrm{test}}) + \sum_{i=1}^{n_{\mathrm{cal}}} \widehat{w}(X_{i}^{\mathrm{cal}})\mathbf{1}\lbrace s_{q}(X_{i}^{\mathrm{cal}}) \ge s_{q}(X_{j}^{\mathrm{test}})\rbrace}{\widehat{w}(X_{j}^{\mathrm{test}}) + \sum_{i=1}^{n_{\mathrm{cal}}} \widehat{w}(X_{i}^{\mathrm{cal}})}.

The table below reports the empirical FDR of our method over 1000 repetitions (formatted as WritingPrompts / XSum):

Detection scoreq=0.05q=0.05q=0.10q=0.10q=0.20q=0.20q=0.30q=0.30
Log-Likelihood0.035 / 0.0330.095 / 0.0930.194 / 0.1950.295 / 0.295
Rank0.018 / 0.0440.092 / 0.0950.193 / 0.1920.295 / 0.291
Log-Rank0.035 / 0.0430.095 / 0.0950.194 / 0.1940.295 / 0.294
Entropy0.000 / 0.0250.004 / 0.0720.110 / 0.1840.281 / 0.292
DetectLLM-LRR0.028 / 0.0410.093 / 0.0940.193 / 0.1930.293 / 0.294
Fast-DetectGPT0.042 / 0.0430.094 / 0.0920.194 / 0.1900.295 / 0.287
AdaDetectGPT0.042 / 0.0400.092 / 0.0900.193 / 0.1900.294 / 0.287
RoBERTa-base0.018 / 0.0440.088 / 0.0970.192 / 0.1970.291 / 0.298
RADAR0.004 / 0.0190.062 / 0.0830.179 / 0.1950.281 / 0.297

The results show that our weighted method keeps the empirical FDR below the target level qq across all settings, providing empirical evidence consistent with FDR control under this token-length shift. Extending weighted CATS to more complex forms of distribution shift and establishing rigorous theoretical guarantees remain important directions for future work. We have added further experimental details and the full tables of selection power and empirical FDR to Appendix D.6 of the revised version, where our method improves selection power over the baseline under this shift.

References

[1] "Reliably Bounding False Positives: A Zero-Shot Machine-Generated Text Detection Framework via Multiscaled Conformal Prediction" (ACL, 2025)

[2] "Detecting LLM-Generated Text with Performance Guarantees" (arXiv, 2026)

[3] "RADAR: Robust AI-Text Detection via Adversarial Learning" (NeurIPS, 2023)

[4] "AdaDetectGPT: Adaptive Detection of LLM-Generated Text with Statistical Guarantees" (NeurIPS, 2025)

[5] "Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy" (ICLR, 2024)

[6] "Conformal Prediction Under Covariate Shift" (NeurIPS, 2019)

[7] "Model-free Selective Inference under Covariate Shift via Weighted Conformal p-values" (Biometrika, 2025)

Comments on the paper

2 comments

Dear Reviewers,

Could you please review the authors’ rebuttals and follow up with any further comments or questions?

Many thanks for your time and efforts.

Best regards, AC

Dear Reviewers,

As the AC–reviewer discussion phase is about to begin, please highlight any disagreements or inconsistencies across the reviews, as well as any concerns that remain unresolved after the author rebuttal.

Please share your perspectives on these points and engage with one another to help us reach a well-supported assessment of the paper.

Thank you for your time and contributions.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.