- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an entropy-conditioned data-adaptive method for detecting LLM-generated text. The authors argue that existing logit-based detectors overlook contextual uncertainty, as tokens with the same log probability may carry different discriminative evidence under different entropy levels. To address this, they introduce a two-dimensional tensor-spline witness that jointly models token log probability and contextual entropy. The paper also identifies a theoretical issue in the optimization objective of AdaDetectGPT and reformulates detection as maximizing detection power under a fixed human FPR. By applying a martingale decomposition, the authors derive a class-conditional Gaussian approximation for passage-level features. Under a shared-covariance assumption, they show that fixed-FPR detection power, fixed-FNR human TNR, and AUC share the same optimal projection direction, leading to a regularized closed-form solution. An independent human calibration set is further used to provide finite-sample FPR control. Experiments are conducted across five datasets and four source models in both white-box and black-box settings. The proposed method outperforms existing baselines, achieving AUROC scores of 0.8374 and 0.7701 in white-box and black-box settings, respectively. Ablation studies suggest that entropy conditioning contributes most of the performance improvements.
Critical strengths
- Well-motivated problem formulation. The paper identifies limitations in existing logit-based detectors and highlights a theoretical issue in AdaDetectGPT's optimization objective, providing motivation for the proposed reformulation.
Critical weaknesses
- The theoretical results rely on strong assumptions, particularly Gaussianity and shared covariance. However, the experiments only examine the Gaussianity of the final detection scores, without validating the shared-covariance assumption. This raises concerns about whether the theoretical guarantees hold in practice.
- The experiments do not clearly demonstrate the benefits of the proposed theoretical reformulation. Ablation results suggest that most improvements come from entropy conditioning rather than other components. Comparisons with alternative optimization methods using the same features would help clarify the contribution of the theory.
Questions for authors
- Fast-DetectGPT suffers from length sensitivity due to its normalization scheme. This paper removes the passage-specific variance normalization but instead uses an unnormalized sum of token-level evidence, whose expected value may scale linearly with passage length. As a result, longer passages may systematically receive higher or lower detection scores than shorter ones, even when their average token-level evidence is similar, potentially leading to length-dependent false-positive rates under a shared threshold. Since the experiments use length-matched passages of 150–200 tokens, could the authors evaluate performance on mixed-length texts and report length-stratified FPR and detection power under a shared calibration threshold?
- The paper argues that tokens with the same log probability but different contextual entropy may provide different discriminative evidence for human-vs-machine detection. However, this claim is not directly validated by the current experiments. Could the authors provide an analysis showing whether contextual entropy affects discriminative evidence when controlling for token log probability, and visualize how the learned witness varies across entropy levels?
- all evaluated source models were released in 2022–2023, with no experiments on more recent LLMs (close-source LLMs such as GPT6). This raises concerns about whether the proposed method generalizes to modern LLM-generated text, particularly given the rapid improvements in generation quality.
- The baseline evaluation mainly focuses on earlier logit-based methods and overlooks recent detectors such as SurpMark [1] and Uncertainty++ [2]. SurpMark is relevant as both methods leverage labeled human and machine reference text to construct detection statistics, making it a meaningful comparison. Uncertainty++ is particularly relevant to the paper's motivation of exploiting uncertainty information beyond token log probabilities. Comparisons with these methods under matched experimental settings would better establish the competitiveness and novelty of the proposed approach.
- The paper lacks evaluation on established public benchmarks for LLM-generated text detection, such as DetectRL (NeurIPS2024) and RAID (ACL2024). These benchmarks cover more diverse generators, text lengths, decoding strategies, and adversarial settings. Could the authors evaluate their method on these benchmarks to better demonstrate its generalization and robustness in realistic scenarios?
[1] Black-Box Detection of LLM-Generated Text Using Generalized Jensen-Shannon Divergence ICML26 [2] On the Salience of Low-Probability Tokens for AI-Generated Text Detection: A Multiscale Uncertainty Perspective ICML26
4 comments
Weakness 2: What the reformulation and the score each contribute
We agree that the previous experiments cannot show where the gains come from. Here we restate our contribution that the score design accounts for much of the gain. Previously we have done ablation over entropy and here we provide the ablation of document-level covariance modeling, or in other words, removing the passage-specific denominator. The motivation of this change is from the failed attempt of AdaDetectGPT for its theoretical claim. The previous difficulty is to deal with the denominator to form a Gaussian style score. Here we remove it and prove that the new score directly fall in the Gaussian distribution. The means and variances of this new score can therefore determine its Gaussian power. A regularized linear equation solve fits the Gaussian AUROC surrogate and supplies one direction for all FPR targets.
Let denote the score that divides our linear score by the passage-specific reference standard deviation. We keep the fitted direction and the length-matched passages fixed and calibrate the two scores separately on the same human set. The table averages 20 white-box and black-box settings.
| Regime | Score | AUROC | FPR / TPR, nominal 1% | FPR / TPR, nominal 5% |
|---|---|---|---|---|
| White box | Ours | 0.8374 | 0.84% / 17.72% | 4.40% / 40.02% |
| White box | 0.8340 | 0.68% / 14.06% | 4.21% / 37.63% | |
| Black box | Ours | 0.7754 | 1.03% / 10.35% | 4.59% / 27.87% |
| Black box | 0.7721 | 0.86% / 8.42% | 4.34% / 25.35% |
Removing the denominator adds about 0.003 to AUROC in both regimes. At nominal 5% FPR, it raises calibrated TPR by 2.39 percentage points in white box and 2.52 in black box. The TPR gains persist when we compare at exactly 5% FPR on the test ROC: from 41.91% to 43.06% in white box and from 28.91% to 29.61% in black box.
We agree that the optimal projection direction itself is not our main contribution. Our contribution is to show how Gaussian AUROC provides a tractable fitting objective for a single score used at multiple FPR targets. To test the practical value of this reformulation, we compared our covariance-sum fit with -regularized logistic regression and Gaussian Neyman–Pearson (NP) fits using the same features and data splits.
Each entry reports white-box / black-box results, averaged over 20 settings. TPR is measured at matched FPR on the test ROC. Time is the median for model selection and refitting on cached features.
| Fitting method | AUROC | TPR at 1% FPR | TPR at 5% FPR | Fit time (s) |
|---|---|---|---|---|
| Ours | 0.8373 / 0.7753 | 21.13% / 11.33% | 42.97% / 29.79% | 0.35 / 0.64 |
| logistic regression | 0.8371 / 0.7744 | 20.91% / 11.23% | 43.18% / 30.16% | 3.36 / 8.68 |
| Gaussian NP, 1% target | 0.8226 / 0.7657 | 19.78% / 11.19% | 40.91% / 27.60% | 1.49 / 1.52 |
| Gaussian NP, 5% target | 0.8302 / 0.7601 | 19.97% / 9.98% | 42.91% / 27.52% | 1.43 / 1.49 |
Ours and logistic regression yield nearly identical AUROC and similar TPR at both FPR levels. The clearest difference is fitting time: Ours is 9.6× faster in white box and 13.6× faster in black box than logistic regression. The comparison will Gaussian NP leads to similar conclusion. This is the practical benefit of the reformulation: the Gaussian AUROC objective gives an optimal linear direction in closed form, and a single linear solve per fit estimates that direction with regularization while retaining the detection performance.
Question 1: Mixed lengths under a shared threshold
Thank you for raising length sensitivity as a problem. We scored the prompt and fresh continuation and used one human calibration threshold across the three lengths for each method, setting, and nominal FPR. The table reports realized rates across 20 white-box settings. Here uses the same fitted direction as Ours, with the passage-specific denominator.
| Continuation tokens | Method | AUROC | FPR / TPR, nominal 1% | FPR / TPR, nominal 5% |
|---|---|---|---|---|
| All | Ours | 0.8934 | 0.84% / 33.12% | 4.76% / 57.20% |
| 0.8932 | 0.70% / 30.06% | 4.66% / 56.07% | ||
| Ada | 0.8859 | 0.93% / 29.96% | 4.84% / 55.54% | |
| SurpMark | 0.8281 | 0.72% / 14.63% | 4.57% / 37.77% | |
| Uncertainty | 0.7732 | 0.74% / 5.90% | 4.50% / 25.07% | |
| 50–99 | Ours | 0.8252 | 0.30% / 8.68% | 3.29% / 30.32% |
| 0.8218 | 0.57% / 10.46% | 4.46% / 33.94% | ||
| Ada | 0.8143 | 0.67% / 11.77% | 4.97% / 33.13% | |
| SurpMark | 0.7642 | 0.39% / 3.22% | 2.61% / 18.11% | |
| Uncertainty | 0.7020 | 0.95% / 4.05% | 5.43% / 19.48% | |
| 100–149 | Ours | 0.9144 | 0.98% / 31.86% | 4.47% / 60.55% |
| 0.9123 | 0.91% / 28.95% | 4.34% / 58.12% | ||
| Ada | 0.9029 | 1.24% / 28.88% | 4.56% / 57.77% | |
| SurpMark | 0.8392 | 0.45% / 12.35% | 4.00% / 38.17% | |
| Uncertainty | 0.7810 | 0.79% / 5.51% | 4.34% / 24.48% | |
| 150–200 | Ours | 0.9520 | 1.23% / 58.99% | 6.54% / 80.88% |
| 0.9508 | 0.61% / 50.90% | 5.18% / 76.27% | ||
| Ada | 0.9468 | 0.88% / 49.37% | 4.99% / 75.84% | |
| SurpMark | 0.8802 | 1.32% / 28.42% | 7.12% / 57.17% | |
| Uncertainty | 0.8400 | 0.49% / 8.14% | 3.71% / 31.29% |
Across all lengths, Ours has the highest AUROC (0.8934) and the highest TPR at both nominal FPR levels (33.12% at 1% and 57.20% at 5%).
Ours has the highest AUROC in each of the three length bands. At 50–99 tokens, ours has lower TPR than at both nominal FPR levels, but also lower realized FPR. At 100–149 and 150–200 tokens, Ours has the highest TPR at both nominal FPR levels. Therefore, totally our score has the best performance over mixed-length detection.
Question 2: Entropy at comparable token log probabilities
Thank you for suggesting this comparison. Figure 2 stratifies by log probability, relative position, and passage length. The top panels show held-out log odds ratios for machine authorship (P55–P75 vs. P25–P45 entropy); the bottom panels show witness slopes at P35 and P65. Below are the odds ratios and slopes for , with pointwise paired-bootstrap intervals.
| White-box setting | High vs. low entropy odds ratio [95% CI] | Witness slope at P35 | Witness slope at P65 |
|---|---|---|---|
| OPT-2.7B / WritingPrompts | 1.076 [1.024, 1.135] | 0.0222 | -0.0152 |
| OPT-6.7B / WritingPrompts | 1.077 [1.019, 1.134] | 0.0155 | 0.0263 |
| OPT-2.7B / C4 | 1.164 [1.088, 1.247] | 0.0037 | 0.0309 |
Within this log-probability window, the higher-entropy group has 7.6%–16.4% higher odds of machine authorship, and all three pointwise intervals exclude 1. For OPT-2.7B on WritingPrompts, the fitted witness slope switches from positive at P35 to negative at P65. Some lower-probability WritingPrompts windows also show the reverse odds pattern. The effect of entropy thus depends on where we look in the log-probability range. We will also visualize the difference in our essay.
Question 3: More recent generators
Thank you for recommending more recent generators. We evaluated Llama-3.1-8B and Qwen3-8B in white box. Each row averages the results across all five datasets.
AUROC
| Generator | Ours | Fast | Ada | SurpMark | Uncertainty | Uncertainty++ |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | 0.8023 | 0.7230 | 0.7889 | 0.7326 | 0.6830 | 0.7676 |
| Qwen3-8B | 0.9411 | 0.8966 | 0.9051 | 0.8622 | 0.7452 | 0.7896 |
Ours has the highest AUROC on both generators, exceeding Ada, the strongest baseline, by 0.0134 on Llama-3.1-8B and 0.0360 on Qwen3-8B.
TPR at 5% FPR (test ROC)
| Generator | Ours | Fast | Ada | SurpMark | Uncertainty | Uncertainty++ |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | 36.64% | 21.40% | 32.72% | 25.08% | 14.92% | 28.08% |
| Qwen3-8B | 71.08% | 64.76% | 65.92% | 41.80% | 25.52% | 37.68% |
Ours also has the highest TPR at 5% FPR: 3.92 percentage points above Ada on Llama-3.1-8B and 5.16 points above Ada on Qwen3-8B.
Question 4: SurpMark and Uncertainty++
Thank you for suggesting these baselines. Results average 20 settings in each regime.
AUROC
| Regime | Ours | SurpMark | Uncertainty | Uncertainty++ |
|---|---|---|---|---|
| White box | 0.8374 | 0.7698 | 0.7048 | 0.7992 |
| Black box | 0.7701 | 0.6914 | 0.6332 | 0.6953 |
Uncertainty++ is the strongest baseline in both regimes. Ours leads it by 0.0382 AUROC in white box and 0.0748 in black box.
TPR at 5% FPR (test ROC)
| Regime | Ours | SurpMark | Uncertainty | Uncertainty++ |
|---|---|---|---|---|
| White box | 43.06% | 25.68% | 18.05% | 32.58% |
| Black box | 28.80% | 18.24% | 10.74% | 19.89% |
At 5% FPR, Ours improves on Uncertainty++ by 10.48 percentage points in white box and 8.91 points in black box. The gains in overall ranking therefore also appear at this operating point.
Question 5: DetectRL and RAID
Thank you for recommending public benchmarks. We fit on the benchmark training splits and evaluate with fixed proxy models in black box, using 1,000 pairs per condition.
AUROC
| Black-box subset | Ours | Fast | Ada | SurpMark | Uncertainty | Uncertainty++ |
|---|---|---|---|---|---|---|
| DetectRL main (4 generators) | 0.9243 | 0.8731 | 0.8689 | 0.9281 | 0.9310 | 0.8901 |
| RAID subset (16 conditions) | 0.9750 | 0.9023 | 0.9737 | 0.9187 | 0.9188 | 0.9512 |
| Equal-weight mean | 0.9496 | 0.8877 | 0.9213 | 0.9234 | 0.9249 | 0.9207 |
Uncertainty leads on DetectRL, where our mean AUROC is 0.0067 below the best result. On RAID conditions, our method has the highest mean AUROC (0.9750). Our method achieves the highest mean AUROC across the two benchmarks (0.9496) when each benchmark is weighted equally.
Thank you for your careful reading and your advices. Here we restate our contributions, test the covariance assumption and broaden the experimental coverage.
Weakness 1: The covariance assumption and power
Thank you for raising the shared-covariance assumption. Our previous claim of equal covariance is actually unnecessary for the AUROC evaluation. We fit it using the regularized sum of two separately estimated class covariances. Under the Gaussian model, AUROC is the probability that an independent machine score exceeds a human score. Their difference has mean θ⊤(μM−μH) and variance θ⊤(ΣH+ΣM)θ. AUROC increases with the ratio of this mean to its standard deviation. Cauchy–Schwarz gives the maximizing direction θ0∝(ΣH+ΣM)−1(μM−μH) even with unequal covariances.
At fixed FPR α, human variance sets the threshold’s offset from the human mean, and machine variance affects the fraction of machine scores above it. So, still, the AUROC direction can lose TPR when the variance split changes with direction. We bound this loss here:
0≤supθπαG(θ)−παG(θ0)≤O(Δτ2).
Here παG is Gaussian TPR at FPR α, and Δτ is the range of the human share of total projected variance across directions. The bound holds when those shares stay away from 0 and 1 and the standardized mean gap along θ0 is bounded above and away from zero. The loss will be zero as long as the covariances of the scores of human and machine are proportional.
To make things clearer we also do experiments on whether this fitting error will influence our test TPR. Here our setting fits one direction for both targets(human and machine) while Gaussian Neyman–Pearson (NP) uses both class covariances to fit separate linear directions on the training data by optimizing Gaussian power at 1% and 5% FPR. Both use the same cached passage features and data splits in 20 white-box and black-box settings. For each target, we compare Ours and the corresponding NP fit at the same FPR on the test ROC.
We measure covariance mismatch on held-out features. The residual in the table is the relative Frobenius distance after rescaling one covariance to best match the other (zero if proportional); it is distinct from Δτ. The human variance share is the human score variance divided by the sum of the two class score variances along our fitted direction. Each row averages the settings in that regime.
The covariance residual is about 14% in both regimes, but it does not directly measure a loss in power. At matched FPR, Ours leads the NP fits by 0.06–2.27 percentage points across the four comparisons. This means the shared-covariance fitting direction is reasonable.