- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors investigate the issue of "score-ranking inversion" in machine-generated text detectors, which rely on language model probabilities—when watermarking techniques are applied. In such cases, some machine-generated texts receive lower detection scores than human-written texts, rendering detection rules based on a fixed, unidirectional threshold ineffective. The authors propose an asymmetric two-sided testing framework: the upper branch constructs a statistic using temperature adjustment, probability truncation, and context-entropy weighting, while the lower branch retains the original Fast-DetectGPT statistic. Both branches are calibrated using human-written text and then combined to reach a final decision. Experiments span five generative models, twelve datasets, and ten different generation scenarios (including both watermarked and non-watermarked settings).The proposed detector exceeds Fast-DetectGPT and AdaDetectGPT in terms of mean AUROC by 0.1929 and 0.1986.
Critical strengths
1.The research problem is practically important. Figure 1 and Table 3 show that different generation conditions can place machine-generated text on either side of the human-text distribution, exposing the limitations of fixed-direction detection. Addressing this failure mode without prior knowledge of the generation method is therefore valuable.
2.The method is concise and supported by a well-matched baseline. The authors construct a two-sided Fast-DetectGPT baseline using the same surrogate model, calibration sources, and budget allocation. Table 4 shows an AUROC gain of approximately 0.04–0.05 over this baseline, suggesting that the improvement goes beyond simply adopting two-sided detection.
3.The experiments are broad and well analyzed. The paper covers diverse generators, watermarking methods, and text domains, and reports AUROC, rank inversions, challenging-case performance, and low-FPR detection rates. Ablations on position weighting, reference distribution adjustment, and branch contributions further support the reported gains.
Critical weaknesses
-
The score-ordering reversal phenomenon does not fully justify the specific asymmetric construction. Score-ordering reversal motivates considering deviations in both directions, but does not by itself determine which statistic should be used for each tail. The paper uses entropy weighting and reference shaping for the upper tail while retaining the original statistic for the lower tail. Section 5 provides empirical support, but does not isolate the gain from the asymmetric design itself. Comparing the proposed detector with a variant that applies the improved statistic to both tails would clarify whether the asymmetric construction provides additional benefits and under what conditions.
-
The specific design of the reference distribution adjustment lacks sufficient motivation. Section 3 motivates temperature scaling and nucleus truncation by noting that low-probability tokens occur infrequently in both human and machine-generated text. However, low frequency does not imply negligible contributions to the reference mean and variance, and truncation changes both the center and scale of the standardized statistic. The paper shows empirical gains but does not sufficiently explain why this particular reference transformation improves human–machine separation. Ablating temperature scaling, nucleus truncation, and their combination would provide stronger evidence for the proposed design.
-
The stated false-positive guarantee does not cover the complete parameter-selection and calibration procedure. Sections 4.1 and 5.3 indicate that the same human texts are used for hyperparameter selection and threshold calibration, while Proposition 5 requires the scoring rule to be independent of the calibration sample. Thus, the actual experimental pipeline does not fully satisfy the proposition's assumptions.
-
The evaluation lacks a truly held-out test of unseen generation conditions. The same five generators and ten watermarking methods are used for both parameter selection and testing. Although the text samples may be independent, the generation conditions themselves are not held out. Therefore, the experiments do not directly establish whether the selected parameters and statistical construction transfer to unseen generators or watermarking methods.
Questions for authors
- Does the asymmetric design provide additional benefits over applying the improved statistic to both tails?
- Why does the proposed combination of temperature scaling and nucleus truncation improve human–machine separation? Can you provide ablations for the two components?
- Does the current parameter-selection and calibration procedure satisfy the assumptions of Proposition 5? Can you verify the FPR using independent development and calibration sets?
- How does the method perform on a generator or watermarking method that is completely held out during parameter selection?
5 comments
Weakness 3 and Question 3 : Does the calibration procedure satisfy Proposition 5?
We agree that the original procedure did not fully satisfy the assumptions of Proposition 5, as the same 800 human texts were used for parameter selection and calibration. Proposition 5 establishes marginal FPR control when the scoring rules and allocation are fixed independently of the calibration and query samples, and these human samples are exchangeable.
To address this, we retained the original development set and fixed , , , and . We conducted two complementary evaluations: one to verify FPR under exchangeable sampling, and another to assess the corrected calibration procedure on the original benchmark.
1. FPR verification under exchangeable sampling. We constructed a new pool of 4,000 human texts, with 1,000 each from C4, SQuAD, Writing, and XSum, excluding identified overlaps, duplicates, and near-duplicates. Across 1,000 random splits, we uniformly partitioned the pool into 800 calibration and 3,200 test texts without per-corpus quotas. Conditional on the fixed pool and scoring rules, the calibration texts and a uniformly selected test query are exchangeable. Both tails were recalibrated for each split.
| Proxy | Mean FPR at nominal 1% | 95% MC interval | Mean FPR at nominal 5% | 95% MC interval |
|---|---|---|---|---|
| OPT | 0.992% | [0.969%, 1.016%] | 4.953% | [4.898%, 5.008%] |
| Falcon | 0.998% | [0.974%, 1.023%] | 5.010% | [4.956%, 5.065%] |
The observed mean FPRs are consistent with the nominal levels under the exchangeable sampling design, as expected from Proposition 5. The intervals quantify Monte Carlo uncertainty across random splits.
2. Independent calibration on the original benchmark. We further evaluated the corrected calibration procedure while keeping the original detector parameters, proxy models, and 660 test settings unchanged. We replaced the original calibration bank with 800 additional human texts, comprising 200 from each of the four corpora and excluding identified overlaps with development and test sources. Both tails were recalibrated using the new bank.
| Calibration | AUROC (OPT / Falcon) | FPR@1% (OPT / Falcon) | FPR@5% (OPT / Falcon) |
|---|---|---|---|
| Original | 0.8219 / 0.8317 | 1.31% / 2.29% | 6.69% / 6.53% |
| Independent | 0.8175 / 0.8299 | 1.06% / 1.31% | 6.14% / 6.19% |
FPRs are measured on 2,360 unique human test texts at fixed nominal thresholds of 1% and 5%.
Independent calibration largely preserves detection performance, with only minor AUROC decreases of 0.0044/0.0018 (OPT/Falcon). The observed FPR decreases at both nominal levels, particularly for Falcon at 1%, where it drops from 2.29% to 1.31%. At the nominal 5% level, FPR remains slightly elevated (6.14%/6.19%), suggesting some sensitivity to differences between calibration and test-domain distributions.
We will update Sections 4.1 and 5.3 and Appendix C.3 to incorporate the independent-calibration protocol and results and clarify the scope of Proposition 5.
Weakness 4 and Question 4 : How does the method perform on generators or watermarking methods completely held out during parameter selection?
We conducted leave-one-generator-out (LOGO, 5 folds) and leave-one-watermark-out (LOWO, 10 folds) evaluations to assess performance on generation conditions excluded from parameter selection.
In each fold, we excluded the target generator or watermarking method from the machine-generated development data. We selected , , and through five-fold cross-validation grouped by the original 800 development sources, with fixed. The selected parameters were then frozen, and our detector was calibrated using the independent 800-human bank introduced in W3/Q3. Evaluation covered 12 test domains, with 660 LOGO and 600 LOWO settings.
The table reports mean AUROC and TPR at matched observed FPR operating points.
| Detector | LOGO AUROC | LOWO AUROC | LOGO TPR@1% FPR | LOWO TPR@1% FPR | LOGO TPR@5% FPR | LOWO TPR@5% FPR |
|---|---|---|---|---|---|---|
| Likelihood | 0.5295 | 0.5097 | 16.81% | 16.70% | 25.52% | 24.79% |
| Entropy | 0.5449 | 0.5517 | 10.57% | 11.16% | 20.06% | 20.95% |
| Rank | 0.5702 | 0.5490 | 13.28% | 12.33% | 24.30% | 22.64% |
| Log-Rank | 0.5452 | 0.5247 | 17.87% | 17.71% | 26.64% | 25.79% |
| Fast-DetectGPT | 0.6388 | 0.6113 | 29.94% | 28.05% | 38.90% | 36.02% |
| AdaDetectGPT | 0.6331 | 0.6065 | 28.30% | 26.72% | 36.33% | 33.70% |
| Binoculars | 0.6135 | 0.5879 | 22.68% | 21.72% | 32.27% | 30.17% |
| NTS | 0.7341 | 0.7251 | 29.22% | 29.13% | 39.08% | 38.45% |
| Ours (OPT) | 0.7194 | 0.6975 | 26.63% | 23.30% | 40.78% | 37.41% |
| Ours (Falcon) | 0.8298 | 0.8210 | 31.13% | 29.91% | 51.82% | 50.13% |
The Falcon configuration achieves the highest performance across all reported metrics. It outperforms Fast-DetectGPT by 0.1910/0.2097 AUROC and NTS by 0.0957/0.0959 under LOGO/LOWO. At 5% FPR, its TPR reaches 51.82%/50.13%, exceeding Fast-DetectGPT by 12.92/14.11 percentage points. Falcon consistently selected , , and across all folds.
The OPT configuration shows more limited transfer. It improves AUROC over Fast-DetectGPT by 0.0806/0.0862, but remains below NTS. At 5% FPR, its TPR exceeds Fast-DetectGPT under LOGO and LOWO, while its TPR at 1% FPR is lower. Most OPT folds selected , , which also showed weaker performance in our reference-shaping ablations. Reselecting parameters using all development conditions yielded 0.7034/0.6983 AUROC on the corresponding task sets, suggesting sensitivity to parameter selection.
we examined how text length affects its detection statistic. Human texts in the development set had a median length of 168 scored tokens, compared with 21 in HellaSwag, 26 in ROCStories, and 68 in TLDR. These differences matter because, for single-model OPT at , truncation introduces a negative centering term whose magnitude grows approximately with when local mean shifts and variances are comparable (Appendix F.2, Eq. 44).
We therefore fixed in this term at 133.76, its median over the 800 development human texts. We left the texts, reference variances, entropy weights, and lower-tail statistic unchanged. Both versions used and were separately calibrated on the same independent set of 800 human texts. Fixing the length factor lowered mean AUROC from 0.8061 to 0.7896 over the 220 development settings, but raised it from 0.7034 to 0.8081 over the 660 test settings.
These results support parameter transfer for the Falcon configuration across held-out benchmark conditions, while showing greater configuration sensitivity for OPT. We will incorporate the held-out evaluations and parameter-selection results into the updated version.
Thank you for the detailed and constructive comments. We provide additional structural comparisons, reference-shaping ablations, independent calibration results, and held-out generation-condition evaluations.
Throughout, two-sided FD denotes Fast-DetectGPT using its original statistic in both tails.
Weakness 1 and Question 1 : Why use an asymmetric design rather than the modified statistic in both tails?
Thank you for highlighting this missing comparison. We have added a symmetric variant that applies the modified statistic to both tails to directly evaluate the additional benefit of our asymmetric design. Both configurations use the same 660 evaluation settings, proxy models, 800 human calibration texts, and allocation γ=0.5, with each configuration separately calibrated.
Our asymmetric design improves mean AUROC by approximately 0.046/0.058 (OPT/Falcon), with paired 95% confidence intervals of [0.0329, 0.0587]/[0.0481, 0.0701]. The comparison isolates the benefit of our asymmetric design.
The gains are particularly pronounced on CGW (+0.1486/+0.1966) and HeavyWater (+0.1226/+0.1720), with additional improvements on SemStamp (+0.0838/+0.1136), while remaining negligible on SynthID. These results show that the asymmetric construction provides substantial additional benefits under conditions with informative lower-tail evidence.
We will include this comparison and the complete condition-level results in Section 5 and the appendix.
Weakness 2 and Question 2 : Ablations for temperature scaling and nucleus truncation components
We agree that low-probability tokens can contribute substantially to the reference mean and variance. Our motivation for reference shaping is to adjust the centering and normalization of the detection statistic to improve its discrimination relative to the human score distribution.
Specifically, reference shaping changes the mean and variance used for standardization while keeping the observed-token log probabilities unchanged. Proposition 3 shows that increasing temperature and reducing nucleus mass shift the reference center in opposite directions in the single-model setting, while both affect the normalization scale. Proposition 4 further characterizes when these changes improve detection relative to the human-calibrated rejection boundary. These properties motivate exploring the combined transformation, whose effectiveness we evaluate empirically.
Component ablations. We fixed the entropy weights, lower-tail statistic, and γ=0.5, and separately calibrated each configuration on the same independent set of 800 human texts, disjoint from development and test data. Results are macro-averaged over the original benchmark’s 660 test settings (12 datasets × 5 generators × 11 generation conditions), reported as OPT / Falcon.
The combined transformation outperforms either component alone across all three metrics. Compared with the original reference, it improves AUROC by 0.0025/0.0167 (OPT/Falcon), with paired 95% confidence intervals of [−0.0037, 0.0110]/[0.0052, 0.0298]. The gain is clearer for Falcon, while the original reference retains higher TPR@1% FPR for both proxies and higher TPR@5% FPR for OPT.
Benefits also vary across watermarking conditions. Averaged within each pair of methods, AUROC gains are 0.0298/0.0674 for ITSEdit/SemStamp, compared with 0.0047/0.0213 for CGW/HeavyWater (OPT/Falcon). In the OPT analysis (Appendix A.7), ITSEdit and SemStamp exhibit weaker score reversal, with original proxy-score medians closer to the human median. This suggests that reference shaping may provide greater benefits when machine-generated text is less clearly separated from human text. These condition-level differences are consistent with the smaller aggregate gain observed for OPT.
We selected T=1.2 and c=0.8 on the development set. We will refine the motivation in Section 3 and clarify the component ablations, condition-specific effects, and parameter sensitivity in Section 5.2.