- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when LLM-generated text remains detectable after editing. Under a Markov model with Huber contamination, it derives a detection boundary and identifies conditions under which clipped tests succeed while raw tests fail. Experiments show that clipping can improve robustness, although the gains vary substantially across detectors and editing settings.
Critical strengths
The theoretical analysis identifies a detection boundary and conditions, helping explain when clipping can improve robustness.
Critical weaknesses
- The raw-versus-clipped comparisons do not establish an advantage over existing alternatives. Missing comparisons include kNNProxy’s lower-tail clipping component and trimmed means.
- The main RAID results rely on attack- or rate-specific fitting, which requires information usually unavailable at detection time. With a universal rule, gains are concentrated in Rank and LRR, while other detectors improve little. Some gains also reduce clean-text detection and increase human false positives. Thus, the proposed method’s practical value as a general robustness enhancement is uncertain.
Questions for authors
- How do Theorems 3.1 and 4.1 extend existing robust-testing results for composite hypotheses and history-dependent contamination?
- Without attack or modification-rate information, under the same scoring model and tuning budget, how does universal clipping compare with kNNProxy’s clipping component and trimmed means?
- How much of the RAID TPR gain remains after matching human FPR and limiting clean TPR loss, particularly for Binoculars?
2 comments
Thanks for the response. For Weakness1 and Qusetion2, the newly added comparison experiments address my concerns about the missing comparison. However, for Question1, I think the theoretical novelty remains unclear. Actually, the existing work (e.g. Saha and Ramdas) also covers fixed sample-sizes, so the fixed-length setting alone is not novel enough. Given the limited gains from universal clipping, the theoretical improvement is key to evaluating the paper’s novelty. Thus, I keep my original rating.
We thank the reviewer for the feedback and the constructive criticisms.
1. Theoretical contribution. As mentioned in Section~1.1, Huber and Strassen study minimax testing for composite hypotheses. Saha and Ramdas address composite hypotheses and adaptive contamination in a sequential setting. We ask a fixed-length question: For a fixed sample-size, how much contamination makes reliable detection impossible? To our knowledge, Theorem 3.1 gives the first exact boundary for unwatermarked LLM detection under a composite class of human Markov sources and history-dependent Huber contamination. Theorem 4.1 distinguishes this limit from failure of a particular detector: under its assumptions, clipping an additive score can preserve consistency even when the raw test loses worst-case power at vanishing FPR. This framework provides a benchmark for studying other detection rules.
2. Universal clipping versus alternatives. We compared clipping and trimming on the same scores with equal tuning budgets, without attack or rate labels at prediction time. At matched 5% human FPR, rank TPR rose from 37.75% raw to 56.79% with clipping and 58.16% with trimming. For LRR, the two were nearly tied (62.02% vs. 62.11%). For others, the two are nearly tied as well. Clipping therefore does not outperform trimming in the full-universial comparison. On the same Falcon log probabilities, kNNProxy’s fixed-floor component yielded 52.15% TPR versus 52.33% for both raw and tuned clipping. We also mention here that our analysis goes beyond log-likelihood: it shows why clipping can help a broader class of additive scores.
3. Binoculars and clean-performance cost. After limiting clean tuning TPR loss to one percentage point, no full-universal gain remained for Binoculars at matched 5% human FPR: attacked TPR was 72.81% raw and 72.72% clipped. At independently calibrated thresholds, clean test TPR increased by 0.10 points and human FPR by 0.02 points. We will revise the paper to reflect this limited benefit.
We agree with the reviewer that, although clipping improves detection in some settings, our experiments do not support it as a general solution for detecting contaminated LLM-generated text. However, we want to emphasize that, our main contribution is, to our knowledge, the first framework to characterize an exact detection boundary for unwatermarked LLM. Clipping arises naturally from the test that achieves this boundary; it is one way to improve existing detectors. We see the framework as a stepping stone for developing other rules and will revise the paper to include the trimmed-mean comparisons.