acceptodds
Under review as a conference paper at ICLR 2027

Local Entropy Fluctuations for Black-Box LLM-Generated Text Detection

Abstract

Zero-shot detection of machine-generated text is vital for mitigating the risks of large language models (LLMs) misuse. While state-of-the-art dual-model and resampling-based detectors outperform single-model statistical detectors on unprompted continuations, they suffer a catastrophic breakdown on instruction-conditioned text, especially in the black-box setting where the source model is unknown. We analyze this failure and trace it to poor prompt conditioning for resampling-based methods and diminished cross-model discriminability for dual-model detectors. Motivated by recent study that human writing maintains a steady, uniform information flow, whereas machine-generated text exhibits sharp, localized entropy fluctuations, we propose a new metric, the Relative Peak Prominence Rate (RelPPR) to capture this structural contrast along the entropy trajectory and further leverage it as a signal to for LLM-generated text detection. Extensive evaluations across diverse model families and datasets demonstrate that RelPPR achieves state-of-the-art detection performance in the black-box setting.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-Generated Text Detection, Entropy, Prominence Rate

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_LxjXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates zero-shot LLM-generated text detection under instruction-conditioned generation in the black-box setting. The authors first analyze the performance degradation of existing dual-model and resampling-based detectors, attributing it to reduced cross-model discriminability and the lack of access to the original instruction prompt, respectively. Motivated by the hypothesis that machine-generated text exhibits stronger local entropy fluctuations than human-written text, the paper proposes Relative Peak Prominence Rate (RelPPR), a detection metric that measures local peak prominence along token-level entropy trajectories, normalized by the sequence-level mean entropy. The proposed method requires only a single evaluator model and does not involve additional training or resampling.

Critical strengths
  1. The paper clearly motivates the need to study LLM-generated text detection under instruction-conditioned generation, a realistic yet underexplored setting.
Critical weaknesses
  1. While the paper identifies why existing detectors struggle with instruction-conditioned generation, these findings do not directly motivate the use of local entropy fluctuations as an alternative detection signal. It remains unclear why entropy fluctuations should remain discriminative under the same conditions that undermine existing methods, leaving a gap between the problem analysis and the proposed solution.
  2. While the Uniform Information Density hypothesis provides a reasonable motivation for studying information fluctuations, it does not directly establish that human-written text should exhibit smoother predictive entropy trajectories when evaluated by a surrogate language model. Similarly, prior findings on high-entropy forking tokens do not establish that these fluctuations are systematically stronger in machine-generated text than in human writing. The paper would benefit from a more direct empirical validation of this proposed distinction.
  3. Previous works, such as LastDE and SurpMark [1], have already explored token-level probability dynamics as signals for machine-generated text detection. The paper does not sufficiently explain why peak prominence offers a more informative characterization of these dynamics than simpler fluctuation statistics, such as variance, adjacent differences, or transition-based measures.

[1] Black-Box Detection of LLM-Generated Text Using Generalized Jensen Shannon Divergence, ICML 2026

Questions for authors
  1. Why does RelPPR perform substantially worse in the white-box setting? Could the authors further investigate this behavior?
  2. The experiments use fixed task-specific instruction templates for each dataset. Would the observed entropy fluctuation patterns and detection performance remain consistent under different instruction formulations, such as stricter formatting requirements or different writing styles?
2 comments

Thank you for your detailed review. We appreciate that you recognize the realistic yet underexplored setting we study. We address your remaining concerns below.

[W1]: From the failure analysis to local entropy fluctuations

We state the motivation of our method in Section 4.1 and briefly summarize it here. According to the Uniform Information Density principle, humans keep the rate of information transmission steady, which yields a smooth entropy trajectory. In contrast, LLM generation contains forking tokens with high entropy, which alternate with predictable stretches and produce higher-amplitude local fluctuations. We further conjecture that instruction conditioning accentuates the peaks at forking tokens. We therefore detect machine text by quantifying the fluctuations of the entropy trajectory.

[W2]: UID hypothesis and forking tokens

UID is a hypothesis about human writing, and forking tokens are a phenomenon observed in the literature. Both motivate the study of fluctuations in the entropy trajectory. However, to our knowledge, no prior work has proposed an effective quantity that captures this kind of fluctuation in the entropy trajectory. We propose RelPPR as such a quantity, and it achieves state-of-the-art performance on this very challenging detection task.

[W3]: Peak prominence versus simpler statistics

Unlike LastDE and SurpMark, which operate on the log-probability trajectory, RelPPR operates on the entropy trajectory. To examine whether peak prominence characterizes these dynamics more informatively than simpler fluctuation statistics, we compare RelPPR with two simpler statistics of the entropy trajectory, total variation, i.e., the sum of absolute differences between adjacent entropy values, ∑t∣Ht−Ht−1∣\sum_t |H_t − H_{t−1}|, and standard deviation, each with and without mean-entropy normalization. We use OLMo-3.1-32B-Instruct as the source model and Qwen3.5-2B as the evaluator. As shown in the table below, RelPPR achieves the best average AUROC (0.887), ahead of the best simple statistic (0.820), and is best on six of the seven datasets. Mean-entropy normalization substantially improves both simpler statistics, raising the average AUROC of total variation from 0.528 to 0.818 and that of the standard deviation from 0.408 to 0.820. This confirms that the benefit of mean-entropy normalization is not limited to PPR, but extends to statistics based on entropy trajectory in general, since normalization removes the bias caused by the typically higher entropy of human text as we analyze in Section 4.2.

XSumSQuADWikiTextWritingPromptsCNN/DMELI5PubMedQAMean
Total variation, w/o norm0.6180.4840.5110.7150.5570.3900.4230.528
Total variation, w/ norm0.8620.8490.8180.8980.8360.9430.5230.818
Std, w/o norm0.4980.3550.3190.6320.3860.2880.3810.408
Std, w/ norm.0.8680.8590.7760.8940.8320.9450.5690.820
RelPPR (ours)0.9450.9340.9290.9490.9180.9800.5560.887

[Q1]: White-box setting

In the white-box setting, one of the two evaluator models used by dual-model detectors is the source model itself, which has low uncertainty about the text, while the other model remains uncertain. This restores the cross-model discrepancy that dual-model detectors rely on, and these detectors therefore achieve strong performance in the white-box setting.We would like to emphasize that the white-box setting is not practical, since real detectors usually have access to neither the source model nor the prompt.

[Q2]: Instruction templates

To evaluate the sensitivity to the instruction template, we construct two alternative templates in addition to the default one. The strict template adds formatting requirements to the instruction, and the style template asks the model to write with a persona fitted to the dataset. We regenerate the machine text under each template and compare the detection methods across all three. As shown in the table below, RelPPR still achieves the best performance under every template, which shows its robustness to the instruction template. We have added this experiment to Section 5 of the revised paper, with template details in Table11 in the appendix.

Methodsdefaultstrictstyle
Entropy0.8370.8210.845
Likelihood0.7940.7670.800
LogRank0.7890.7680.797
LRR0.7400.7310.756
LastDE0.7810.7690.783
Fast-DetectGPT0.5150.4910.503
Binoculars0.5290.5040.515
DNA-DetectLLM0.5150.5000.506
IRM0.6580.6460.659
RelPPR (Ours)0.8870.8770.892

We hope you find our response with additional experiments and further clarification to be helpful, and consider turning the verdict into an accept.

Thank you for the additional experiments and clarifications. The new comparisons in W3 are helpful, showing that RelPPR outperforms normalized total variation and standard deviation. The additional experiments in Q2 also provide useful evidence of robustness across different instruction templates.

However, my main concerns remain partially unresolved. For W1 and W2, the authors still rely on UID and forking tokens as motivation without directly validating that machine-generated text exhibits stronger local entropy fluctuations than human-written text under instruction-conditioned generation. The new W3 experiments demonstrate the effectiveness of RelPPR, but do not validate this underlying hypothesis.

Overall, I appreciate the additional experiments, but the central methodological motivation remains insufficiently established. I therefore maintain my original assessment.

Reviewer_rvYCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the issue of severe performance degradation in existing zero-shot LLM text detectors such as dual-model detectors and resampling detectors under instruction-conditioned generation scenarios. Through an information-theoretic analysis, the authors demonstrate that the failure of dual-model detectors stems from the collapse of the KL divergence between models under instruction-conditioned conditions, whilst the failure of resampling detectors arises from the prompt conditioning gap. Based on the observation that human writing follows uniform information density (UID), whilst LLM generation exhibits high entropy peaks at semantic forking tokens, the paper proposes a detection metric that requires no training and only a single forward pass: Relative Peak Prominence Ratio (RelPPR). This method captures the local, sharp fluctuations characteristic of machine-generated text by calculating the topological prominence of the entropy trajectory produced by a surrogate model and normalising it by the sequence’s average entropy. Extensive experiments across seven datasets and five source models demonstrate that RelPPR achieves state-of-the-art (SOTA) performance under black-box instruction conditions, whilst exhibiting exceptional robustness against low False Positive Rates (FPR), varying text lengths, decoding strategies and rewriting attacks.

Critical strengths

The motivation is profound and the failure analysis is convicing: rather than blindly proposing new complex networks, the paper provides an extremely rigorous theoretical and empirical deconstruction of the failures of existing state-of-the-art methods such as Fast-DetectGPT, Binoculars and DNA-GPT in instruction-conditioned scenarios. The analysis of diminished cross-model discriminability and the ‘prompt-conditioned gap’ hits the nail on the head and offers significant guidance to the AI safety community.

The method is minimalist and computationally efficient: RelPPR relies solely on entropy sequences extracted from a single forward pass of the evaluation model, requiring no perturbations, no resampling and no fine-tuning. This parameter-free method with O(L) complexity is of immense practical value in black-box deployment and streaming detection scenarios.

The experimental design is comprehensive and closely aligned with real-world deployment: the paper not only reports AUROC but also places particular emphasis on TPR at low FPR. Furthermore, robustness tests regarding text length, decoding randomness (Temperature/Top-p) and T5 rewriting attacks demonstrate that this signal is not a fragile statistical shortcut.

Critical weaknesses

Tokenizer mismatch and entropy trajectory alignment. The paper claims to use a small model (such as Qwen3.5-2B) as an evaluator in a ‘black-box’ setting to evaluate source models of other families or scales (such as OLMo, Gemma and Mistral). However, there are significant differences between the tokenizer of different models. The forking tokens generated by the source model are determined based on the semantic boundaries defined by its own tokeniser; when the entire generated text is re-encoded using Qwen’s tokeniser to calculate conditional entropy, high-entropy tokens from the source model may be split into multiple subwords or merged with adjacent words. This misalignment of token boundaries can cause the peak entropy positions calculated by the evaluator to be severely shifted, smoothed out, or even disappear. The paper does not discuss the disruptive impact of this tokenizer mismatch on the core signal of RelPPR, nor does it provide any analysis of the alignment of entropy trajectories across different tokenizer.

Prior work [1] demonstrates that human texts do not constitute a unified distribution; their extremely high diversity and domain-specificity mean that any binary classification method attempting to characterise the essence of humanity using a single statistic, such as a smooth entropy trajectory, is bound to lack generalisability. The core theoretical foundation of RelPPR is the Uniform Information Density (UID) hypothesis in psycholinguistics, which posits that the entropy trajectories of human texts are smooth. These viewpoints are contradictory. Do the authors have any comments on this?

Evaluation on harder benchmarks and more baselines. The paper relies solely on seven custom datasets and a single T5 rewriting attack. Could the authors provide the results on this benchmark? Moreover, some important baselines [2, 3, 4] are missing.

I am happy to raise my score if the authors can address my concerns.

[1] Zeng C, Tang S, Chen Y, et al. Human texts are outliers: detecting LLM-generated texts via out-of-distribution detection[J]. Advances in Neural Information Processing Systems, 2026, 38: 163483-163513.

[2] Guo X, Zhang S, He Y, et al. Detective: Detecting ai-generated text via multi-level contrastive learning[J]. Advances in Neural Information Processing Systems, 2024, 37: 88320-88347.

[3] Bao G, Zhao Y, He J, et al. Glimpse: Enabling white-box methods to use proprietary models for zero-shot llm-generated text detection[C]//International Conference on Learning Representations. 2025, 2025: 51844-51865.

[4] Zeng C, Tang S, Yang X, et al. Dald: Improving logits-based detector without logits from black-box llms[J]. Advances in Neural Information Processing Systems, 2024, 37: 54947-54973.

Questions for authors

See weakness

4 comments

Thank you for your detailed review. We are preparing a full response. Under "Evaluation on harder benchmarks and more baselines," you ask for results "on this benchmark," but the review doesn't seem to name a specific one. Could you tell us which benchmark you have in mind?

Thank you for your supportive review! We are glad that you appreciate the motivation and failure analysis of existing detectors, the simplicity and efficiency of RelPPR, and our deployment-oriented experiments, including the robustness analyses. We provide detailed responses below.

[W1]: Tokenizer mismatch and entropy trajectory alignment

First, RelPPR does not require the forking tokens of the evaluator to align with those of the source model, and the source model is inaccessible in our setting.

Nevertheless, we examine how well the entropy peaks of the evaluator (Qwen3.5-2B-Base) align with those of the source model (OLMo-3.1-32B-Instruct) on the same machine-generated text, as shown in the table below. Because the two models use different tokenizers, we assign each peak to the whitespace-separated word it falls on, and call the resulting words the peak words. We report peak recall, i.e., the proportion of the source model's peak words that are also peak words of the evaluator, averaged over texts. The evaluator never sees the prompt, whereas the source model is run under two conditions: with the prompt and without it.

XSumWikiTextWritingPromptsELI5SQuADCNN/DMPubMedQA
Source model w/ instruction0.6480.6840.6840.6220.6660.6460.548
Source model w/o instruction0.7000.7300.7410.6610.7140.6950.746

The table shows that the source model and the evaluator share a large proportion of their peak words. Additionally, because RelPPR measures the relative vertical elevation of a peak against its adjacent valleys, a horizontal shift of a peak by 1–2 subwords does not erase its prominence.

Finally, RelPPR performs stably across source models with different tokenizers and across evaluators with different tokenizers as shown in Table 2 in Section 5 and Table 8 in appendix of the revised paper, which also suggests that tokenizer differences have little effect.

[W2]: Relation to Zeng et al. (2026) and the UID motivation

Zeng et al. (2026) argue that a binary classifier trained with cross-entropy can reach near-optimal training loss yet incur a high loss on on unseen human text (Theorem 2 and 6), and illustrates this with the distances between samples in an embedding space(Figure 1). However, our study is under the zero-shot setting and there is no classifier and no human sample to overfit. Moreover, the UID hypothesis motivates us to operate on the entropy trajectory, not on an embedding space. Since the settings and the signals differ, the work of Zeng et al. (2026) and the UID hypothesis do not conflict.

[W3]: Harder benchmarks and additional baselines

Thank you for the suggestion. We would like to point out that DeTeCtive and DALD require training, whereas we focus on the zero-shot setting, where there is no training set, and that Glimpse is not a detection method. The details are as follows. DeTeCtive is a supervised detector that requires labeled human and LLM training data, which is outside our zero-shot setting. DALD fine-tunes the surrogate model on outputs of advanced LLMs, so it also requires training data. Glimpse does not propose a new detection method. It introduces a way for detectors such as Entropy, Rank, LogRank, and Fast-DetectGPT to use API-only models as evaluator models.

We hope you find our response with additional experiments and further clarification to be helpful, and consider turning the verdict into an accept.

Reviewer_rvYCOpenReview ↗

Response to Authors

Thank you for your rebuttal. My major concerns are addressed. Authors are encouraged to add discussion about the difference with Zeng et al. (2026). For the harder benchmarks, I think RAID[1] is sufficient.

[1] Dugan L, Hwang A, Trhlík F, et al. Raid: A shared benchmark for robust evaluation of machine-generated text detectors[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 12463-12492.

AuthorsOpenReview ↗

Additional Experiments on RAID

Thank you for the follow-up and for the positive assessment. We are glad that your major concerns are addressed.

We add Zeng et al. (2026) to Section 2 in the revised paper. We also run our method and all baselines of the main table on RAID. To make the results as convincing as possible, we choose GPT-4 as the source model, the strongest source model in RAID.

The RAID test set has no public labels, and scores have to be submitted to the public leaderboard through a pull request on GitHub. We are concerned that this could violate the anonymity policy, and that we might not receive our scores in time. We therefore build the evaluation set from the labeled training split:

  • We use all 13,371 human documents of the training split.
  • For each human document, we take the GPT-4 text generated for the same document, which has the same title and domain, with sampling decoding and no repetition penalty. This gives 13,371 human and 13,371 machine texts, paired one to one.
  • Every detector uses the black-box setting and the arguments of the main table, with Qwen3.5-2B-Base as the evaluator for all baselines.

The table below reports AUROC for each of the eight RAID domains. Avg is the mean over domains. RelPPR reaches the highest average AUROC and is the best method in five of the eight domains.

MethodAbstractsBooksNewsPoetryRecipesRedditReviewsWikiAvg
Entropy0.4890.9130.8770.7290.9820.9510.9450.8250.839
Likelihood0.4860.8600.8700.6450.9800.9400.9320.8220.817
LogRank0.5100.8430.8640.6010.9790.9290.9100.8400.810
LRR0.5750.7440.7930.4660.9560.8640.7530.8590.751
GECScore0.7010.6450.5920.6620.8750.8160.8110.6080.714
LastDE0.3560.7920.8700.6430.9640.9140.9000.8980.792
LastDE++0.2400.2540.4330.4220.3390.5480.4550.7120.425
DNA-GPT0.4040.5800.5910.4450.6560.6060.4870.8390.576
Fast-DetectGPT0.5870.3710.6550.3650.7760.6330.5030.6080.562
Binoculars0.5870.3660.6520.3620.7840.6270.4990.6070.561
DNA-DetectLLM0.5710.4100.6140.5100.7210.5640.5920.6060.573
IRM0.9450.4020.7000.4350.4160.3890.3760.1870.481
RelPPR (Ours)0.4780.9580.8960.9210.9810.9540.9940.8460.879

We hope that our additional results on RAID can fully address your remaining concerns, and we would appreciate it if you could consider updating your score accordingly.

Reviewer_Ym13OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies black-box LLM-generated text detection in instruction-conditioned settings. The authors first analyze the performance degradation of existing dual-model and resampling-based detectors in this setting. They then propose RelPPR, a detection method based on local entropy fluctuations. The method measures the prominence of local peaks in the entropy trajectory and normalizes it by the mean entropy. It requires no additional training for text detection. Experiments show that RelPPR achieves good detection performance across multiple datasets and source models.

Critical strengths
  1. The paper focuses on instruction-conditioned black-box detection, which is a practical and meaningful research problem. The analysis of performance degradation in existing detection methods also provides valuable insights.

  2. The proposed method is simple. It requires no additional training or resampling and uses only one evaluator model, offering certain advantages in computational efficiency.

  3. The experiments cover multiple datasets and models and include robustness analyses under different generation settings. RelPPR achieves competitive results in the main black-box experiments.

Critical weaknesses

Weaknesses

1. The discussion of related work and the evaluation of the proposed method's unique contribution could be further strengthened.

Some existing studies have explored using token-level information fluctuations for AI-generated text detection. For example, GPT-who (Venkatraman et al., NAACL 2024 Findings) is based on the Uniform Information Density principle and uses features such as the variance of token-level surprisal and differences between adjacent tokens. FourierGPT (Xu et al., EMNLP 2024) uses spectral analysis of relative likelihood sequences to capture dynamic changes in token probabilities. These works are related to the motivation of this paper and would be worth discussing in the Related Work section. Although this paper uses predictive entropy and Topological Peak Prominence, which differ from the above methods, it is still unclear how much additional benefit this peak-based statistic provides compared with simpler local fluctuation statistics. In particular, the original PPR performs substantially worse than the normalized RelPPR. Therefore, I suggest that the authors include comparisons with simpler statistics, such as normalized standard deviation and total variation, to better demonstrate the necessity and unique contribution of Topological Peak Prominence.

2. The relationship between the core hypothesis and the experimental results deserves further analysis.

One of the main motivations of this paper is that machine-generated text exhibits more pronounced local fluctuations in its entropy trajectory than human text. However, the original PPR, which directly measures local peak prominence, does not show consistent discriminative performance across different settings. The authors point out in Section 4.2 that human text with higher overall entropy may also have larger raw PPR values, and they introduce RelPPR through mean entropy normalization to address this issue. This explanation is reasonable, but it raises a question worth further clarifying: to what extent does the improvement in detection performance come from local entropy fluctuations themselves, and to what extent does it depend on normalization by mean entropy? In addition, the white-box results suggest that the effectiveness of the relative fluctuation signal may depend on the evaluation setting. Furthermore, Figures 4 and 7 mainly compare the PPR distributions of machine-generated text with and without instructions, rather than directly comparing the fluctuations between human- and machine-generated text. I suggest that the authors further present the distributions of PPR, mean entropy, and RelPPR for both human- and machine-generated text. This would help clarify why the method works and which detection signals contribute most to its performance.

3. The connection between the forking-token motivation and the actual detection statistic could be further clarified.

The paper uses the forking-token phenomenon in LLM generation as a motivation for detecting local entropy fluctuations. However, the method computes predictive entropy using an evaluator model without access to the original instruction, rather than using the entropy of the source model during generation. Since the source and evaluator models may differ and use different conditioning contexts, their entropy trajectories may not necessarily be consistent. Therefore, it remains unclear to what extent the local peaks observed by the evaluator model reflect the forking-token phenomenon during source model generation. I suggest that the authors further discuss this difference or compare the local peak characteristics of the two entropy trajectories when the source model and original instruction are accessible.

4. The method shows some sensitivity to the choice of evaluator model.

The paper evaluates RelPPR using multiple evaluator models and achieves good overall results. However, Appendix Table 6 also shows that the relative performance of RelPPR and existing methods can vary noticeably across different evaluator configurations. The authors acknowledge that different model families can affect detection performance, but this observation deserves further analysis. In particular, the performance degradation of existing dual-model detectors in instruction-conditioned black-box settings may depend, to some extent, on the choice of evaluator models. Further discussion of these differences would help clarify the conditions under which the proposed method is most effective and how different detection methods depend on evaluator configurations.

5. The analysis of the prompt conditioning gap could be further supported by more direct experiments.

The paper identifies the prompt conditioning gap as an important factor contributing to the performance degradation of resampling-based detectors. When the detector cannot access the original instruction, the distribution used for resampling may differ from the actual generation distribution. This explanation is reasonable, but the current analysis mainly relies on the conditional distribution mismatch and the observed changes in detection performance, without a direct controlled experiment to evaluate the specific impact of this factor. In addition, the existing work Learn-to-Distance (Zhou et al., ICLR 2026) also discusses the distribution mismatch caused by unknown generation prompts and analyzes Fast-DetectGPT under known-prompt and unknown-prompt settings. Although its focus is not exactly the same as that of this paper, the two studies are related, and I suggest that the authors include some discussion of this work. A more direct experiment would be to provide the original instruction to resampling-based detectors and examine whether their detection performance improves. Furthermore, this paper uses fixed instruction templates designed for different datasets. Testing different prompt formulations for the same task would also help assess whether RelPPR's detection performance is sensitive to the specific instruction templates.

Questions for authors
  1. Have the authors considered comparing RelPPR with simpler entropy fluctuation statistics, such as normalized standard deviation and total variation?

  2. Could the authors further show the distributions of PPR, mean entropy, and RelPPR for human and machine-generated text to clarify the contributions of local fluctuations and normalization to detection performance?

  3. Could the authors further explain the performance differences across evaluator configurations reported in Appendix Table 6?

  4. Have the authors considered providing the original instruction to directly compare detection performance under known-prompt and unknown-prompt settings? Additionally, how sensitive is RelPPR to different instruction templates?

3 comments

Thank you for your detailed and constructive review. We are happy that you appreciate the practicality of our problem setting, our analysis of existing detectors, the simplicity and efficiency of RelPPR, and our comprehensive experiments, including the robustness analyses. We provide detailed responses below.

[W1 & Q1]: Comparison with simpler fluctuation statistics

GPT-who requires supervised training, and FourierGPT requires either supervised training or paired data. Both are therefore not applicable to our zero-shot setting. To address your question, we compare RelPPR with two simpler statistics of the entropy trajectory, total variation and standard deviation, each with and without mean-entropy normalization. We use OLMo-3.1-32B-Instruct as the source model and Qwen3.5-2B as the evaluator. As shown in the table below, when all statistics are normalized, RelPPR performs the best, with an average AUROC of 0.887 against 0.818 for total variation and 0.820 for the standard deviation, and it is best on six of the seven datasets. This shows that the gain does not come from normalization alone, and that peak prominence is itself a more discriminative statistic than total variation and standard deviation.

XSumSQuADWikiTextWritingPromptsCNN/DMELI5PubMedQAMean
Total variation, w/o norm0.6180.4840.5110.7150.5570.3900.4230.528
Total variation, w/ norm0.8620.8490.8180.8980.8360.9430.5230.818
Std, w/o norm0.4980.3550.3190.6320.3860.2880.3810.408
Std, w/ norm.0.8680.8590.7760.8940.8320.9450.5690.820
RelPPR (ours)0.9450.9340.9290.9490.9180.9800.5560.887

[W2 & Q2]: Contribution of local fluctuation and mean-entropy normalization

To separate the contribution of local fluctuation from that of the entropy level, we design a controlled experiment that holds the mean entropy within a narrow range. For each dataset, we pool the human and machine texts, sort them by mean entropy, and split them into five bins, each containing 20% of the samples. We then compute the AUROC of Entropy, PPR and RelPPR within each bin. We use OLMo-3.1-32B-Instruct as the source model and Qwen3.5-2B as the evaluator, and we take XSum and WritingPrompts as examples, as shown in the table below.

DatasetDetectorBin 1, lowest entropyBin 2Bin 3Bin 4Bin 5, highest entropy
XSumEntropy0.5660.6460.6720.5790.576
XSumPPR0.7950.8860.8540.9140.903
XSumRelPPR0.8160.9100.8760.9150.902
WritingPromptsEntropy0.6280.6020.5560.5620.524
WritingPromptsPPR0.8840.8930.9300.9220.905
WritingPromptsRelPPR0.9390.9170.9320.9220.912

Entropy does not work well in this setting, since the human and machine texts within each bin have similar entropy. PPR and RelPPR remain strong in each bin, which shows that local entropy fluctuation carries discriminative information beyond the entropy level itself. This also indicates that the detection signal does not come only from normalization, and that PPR already separates human from machine text once the entropy level is controlled.

RelPPR remains robust in the white-box setting. Our focus is the practical black-box, instruction-conditioned setting, where neither the source model nor the prompt is accessible, and RelPPR achieves state-of-the-art performance in it. In the revised paper, we show the distributions of RelPPR in Figures 3 and 12, of PPR in Figure 13, and of mean entropy in Figure 14, with Figure 3 in Section 4 and Figures 12 to 14 in the Appendix. The mean-entropy and RelPPR distributions of human and machine text are clearly separated. In particular, the figures show that machine text has higher RelPPR than human text, which indicates that machine text has stronger relative fluctuations in its entropy trajectory.

[W3]: Alignment between the evaluator’s and the source model’s entropy peaks

Prior studies show that forking tokens, at which a different choice would lead to a substantially different continuation, arise at semantic branching points of the text, such as narrative branching points [1] and decision points in reasoning [2]. These branching points are determined largely by the content of the prefix x<ix_<i rather than by the instruction or the identity of the source model, so an evaluator that reads the same prefix faces the same semantic choice at these positions, even without access to the instruction.

As an empirical check, we examine how well the entropy peaks of the evaluator (Qwen3.5-2B-Base) align with those of the source model (OLMo-3.1-32B-Instruct) on the same machine-generated text, as shown in the table below. Because the two models use different tokenizers, we assign each peak to the whitespace-separated word it falls on, and call the resulting words the peak words. We report peak recall, i.e., the proportion of the source model's peak words that are also peak words of the evaluator, averaged over texts. The evaluator never sees the prompt, whereas the source model is run under two conditions: with the prompt and without it. The result indicates that the source model and the evaluator share a large proportion of their peak words.

XSumWikiTextWritingPromptsELI5SQuADCNN/DMPubMedQA
Source model w/ instruction0.6480.6840.6840.6220.6660.6460.548
Source model w/o instruction0.7000.7300.7410.6610.7140.6950.746

[W4 & Q3]: Robustness across evaluator models

Table 6 ((Table 8 in the revised paper) shows that RelPPR is robust and generalizable across evaluator models. Across four model families, both dense and MoE architectures, and sizes from 2B to 35B, RelPPR achieves the best AUROC on seven of the nine evaluators and the best average. The dual-model detectors remain far below RelPPR on all of these evaluators, so their degradation under instruction conditioning is not specific to one evaluator. The only exception is Falcon-7B, an older 2023 model that is poorly matched to modern instruction-tuned text.

[W5 & Q4]: Direct evidence for the prompt conditioning gap and sensitivity to instruction templates Thank you for pointing out the work of Zhou et al. (2026). Although both Zhou et al. (2026) and our paper address the distribution mismatch caused by unknown generation prompts, their method requires training, whereas we focus on the zero-shot setting.

To directly test the effect of the prompt conditioning gap, we compare DNA-GPT and LastDE++ with and without instruction cc, in both the black-box and white-box settings, using OLMo-3.1-32B-Instruct as the source model. In the white-box setting, the evaluator is the source model itself, so supplying cc isolates the prompt conditioning gap alone. As shown in the table below, Either supplying cc or setting the evaluator to the source model improves detection performance, which confirms our analysis in Section3.3. We have added this experiment to Section 3.3 of the revised paper.

MethodSettingw/o ccw/ cc
LastDE++black-box0.4420.602
LastDE++white-box0.4190.682
DNA-GPTblack-box0.4750.499
DNA-GPTwhite-box0.7650.977

To evaluate the sensitivity to the instruction template, we construct two alternative templates in addition to the default one. The strict template adds formatting requirements to the instruction, and the style template asks the model to write with a persona fitted to the dataset. We regenerate the machine text under each template and compare the detection methods across all three. As shown in the table below, RelPPR still achieves the best performance under every template, which shows its robustness to the instruction template. We have added this experiment to Section 5 of the revised paper, with template details in Table11 in the appendix.

Methodsdefaultstrictstyle
Entropy0.8370.8210.845
Likelihood0.7940.7670.800
LogRank0.7890.7680.797
LRR0.7400.7310.756
LastDE0.7810.7690.783
Fast-DetectGPT0.5150.4910.503
Binoculars0.5290.5040.515
DNA-DetectLLM0.5150.5000.506
IRM0.6580.6460.659
RelPPR (Ours)0.8870.8770.892

We hope you find our response with additional experiments and further clarification to be helpful, and consider turning the verdict into an accept.

[1] Bigelow E, Holtzman A, Tanaka H, et al. Forking paths in neural text generation. ICLR, 2025.

[2] Wang S, Yu L, Gao C, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. NeurIPS, 2026.

I thank the authors for their detailed response and additional experiments. I appreciate their efforts, but I remain concerned that some fundamental issues have not been fully addressed. Although I regret this, I intend to maintain my original rating.

In particular, the additional results do not convincingly establish the connection between the proposed detection statistic and the underlying forking-token mechanism without the comparison. Moreover, the substantial performance variations across evaluator models raise concerns about the method's robustness and the generality of the authors' explanations. While I acknowledge the promising empirical results, I believe these concerns affect the paper's central claims and remain insufficiently resolved. Therefore, I maintain my original rating.

Reviewer_2M3KOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates zero-shot detection of LLM-generated text under instruction-conditioned, black-box settings. The authors first analyze why existing dual-model and resampling-based detectors experience substantial performance degradation when machine-generated text is conditioned on instructions that are unavailable to the detector.

Motivated by the hypothesis that machine-generated text exhibits stronger local fluctuations in token-level entropy than human-written text, the authors propose Relative Peak Prominence Rate. The method computes the entropy trajectory using a surrogate language model, measures the prominence of local entropy peaks, and normalizes the resulting statistic by the sequence-level mean entropy.

Critical strengths
  1. The proposed RelPPR metric is intuitive and requires only a single surrogate language model without additional model training or resampling.

  2. The decomposition of cross-entropy into entropy and KL divergence provides a useful perspective on the behavior of dual-model detection scores. Similarly, the analysis of prompt-conditioning mismatch offers a plausible explanation for why resampling-based methods may struggle when the original instruction is unavailable.

Critical weaknesses
  1. The central hypothesis is not convincingly supported by the empirical evidence. The paper's central scientific claim is that machine-generated text exhibits stronger localized entropy fluctuations than human-written text. However, the experimental evidence does not convincingly establish this claim. In Table 1, the unnormalized PPR metric achieves an average AUROC of only 0.597, substantially below the Entropy baseline 0.812. Its performance is even below 0.5 on PubMedQA and ELI5. The major performance improvement emerges only after introducing sequence-level entropy into the detection score.

  2. The theoretical motivation conflates distinct notions of information and uncertainty. The paper motivates the proposed method using the Uniform Information Density hypothesis and findings concerning high-entropy forking tokens in LLM generation. However, UID primarily concerns the surprisal of observed linguistic units, whereas RelPPR measures the predictive entropy of a surrogate language model. These are not interchangeable quantities. And, the paper does not provide a theoretical result or a controlled experiment establishing that the entropy peaks observed by the evaluator correspond to genuine decision points in the source model.

  3. The paper does not adequately engage with recent research on probability-trajectory fluctuations for training-free machine-generated text detection. SpecDetect (https://doi.org/10.1609/aaai.v40i38.40510) establishes a single-surrogate, training-free detector based on spectral properties of token-level log-probabilities. WaveDetect (https://doi.org/10.18653/v1/2026.findings-acl.424) further investigates time-frequency representations for generated-text detection. Unveiling Spectral Mechanisms in Training-Free LLM Text Detection (https://doi.org/10.48550/arXiv.2608.25944) provides additional theoretical and empirical analysis of fluctuation-based detection and its limitations. Other recent work, including When AI Settles Down ( https://doi.org/10.48550/arXiv.2601.04833), directly investigates local volatility as a detection signal.

  4. The experimental protocol does not fully isolate the claimed failure mechanism or eliminate potential confounding factors. The evaluation assumes that the source model has access to task-specific instructions while the detectors do not. This is a meaningful setting, but it inherently disadvantages methods that depend on matching the source generation context.

Questions for authors

Q1

A closely related study, some papers reports that human-written text exhibits stronger fluctuations in token-level log-probability trajectories than machine-generated text, attributing this phenomenon to the broader and less predictable token choices made by human writers. In contrast, the present paper argues that machine-generated text exhibits stronger local entropy fluctuations than human-written text, using this observation as the central motivation for RelPPR. These findings appear to suggest opposite conclusions regarding the fluctuation characteristics of human and machine text. How do the authors reconcile these apparently contradictory observations?

Q2

The paper's central hypothesis is that machine-generated text has more pronounced local entropy peaks. However, Table 1 shows that the unnormalized PPR achieves only 0.597 average AUROC, compared with 0.812 for the standard Entropy baseline. On ELI5 and PubMedQA, PPR even performs below 0.5 AUROC. The substantial improvement emerges only after incorporating sequence-level mean entropy, with RelPPR achieving 0.871 AUROC. How can the authors conclude that stronger local entropy fluctuations are the primary discriminative signal when the direct fluctuation metric itself performs poorly?

Q3

Novelty Relative to Existing Fluctuation-Based Detectors.Several prior studies, including SpecDetect (https://doi.org/10.1609/aaai.v40i38.40510), WaveDetect (https://doi.org/10.18653/v1/2026.findings-acl.424), have already investigated temporal or frequency-domain fluctuations in language-model probability signals for machine-generated text detection. What is the fundamental methodological contribution of RelPPR beyond replacing token log-probabilities with predictive entropy and replacing spectral statistics with peak prominence? Can the authors provide a controlled comparison applying both spectral-energy and peak-prominence metrics to both log-probability and entropy trajectories under identical experimental settings?

4 comments

Thank you for your time and feedback. We are glad that you appreciate the simplicity and efficiency of RelPPR, our cross-entropy decomposition for analyzing dual-model detectors, and our analysis of the prompt conditioning gap. We provide detailed responses below.

[W1 & Q2]: Role of PPR and mean-entropy normalization

We do not claim that machine-generated text exhibits stronger localized entropy fluctuations than human-written text. This is our hypothesis, which motivates us to use the degree of fluctuation of the entropy trajectory as a detection signal, and PPR is the statistic used to capture this signal. Our final detector, RelPPR, builds on the analysis in Section 4.2: human text typically exhibits higher entropy than machine-generated text under the evaluator model, so a high-entropy human text can attain a large raw PPR even when its fluctuations are mild relative to its own entropy level. We would like to point out that the concepts of hypothesis and claim are conflated in the first weakness.

Accordingly, we do not conclude that raw local fluctuation alone is the primary discriminative signal. Normalizing by the mean entropy is itself one of our contributions and novelty, since other trajectory-based detectors such as SpecDetect and LastDE do not normalize their score by the statistic of the trajectory itself. Nevertheless, we design a controlled experiment that holds the mean entropy within a narrow range to separate the contribution of local fluctuation from that of the entropy level. For each dataset, we pool the human and machine texts, sort them by mean entropy, and split them into five bins, each containing 20% of the samples. We then compute the AUROC of Entropy, PPR and RelPPR within each bin. We use OLMo-3.1-32B-Instruct as the source model and Qwen3.5-2B as the evaluator, and we take XSum and WritingPrompts as examples, as shown in the table below.

DatasetDetectorBin 1, lowest entropyBin 2Bin 3Bin 4Bin 5, highest entropy
XSumEntropy0.5660.6460.6720.5790.576
XSumPPR0.7950.8860.8540.9140.903
XSumRelPPR0.8160.9100.8760.9150.902
WritingPromptsEntropy0.6280.6020.5560.5620.524
WritingPromptsPPR0.8840.8930.9300.9220.905
WritingPromptsRelPPR0.9390.9170.9320.9220.912

Entropy does not work well in this setting, since the human and machine texts within each bin have similar entropy. PPR and RelPPR remain strong in each bin, which shows that local entropy fluctuation carries discriminative information beyond the entropy level itself. This indicates that the detection signal does not come only from normalization, and that PPR already separates human from machine text once the entropy level is controlled.

[W2]: UID, surprisal, and entropy

Both entropy and surprisal are information-theoretic measures of the model’s uncertainty about the next token. Entropy at a position is the expected amount of information at that position, and the strong form of UID [1] implies that the conditional entropy is constant across positions in expectation. Therefore, UID is a well-founded motivation for our method RelPPR.

RelPPR does not rely on the evaluator's peaks corresponding to decision points in the source model, which is inaccessible in our setting. Nevertheless, we examine how well the entropy peaks of the evaluator (Qwen3.5-2B-Base) align with those of the source model (OLMo-3.1-32B-Instruct) on the same machine-generated text, as shown in the table below. Because the two models use different tokenizers, we assign each peak to the whitespace-separated word it falls on, and call the resulting words the peak words. We report peak recall, i.e., the proportion of the source model's peak words that are also peak words of the evaluator, averaged over texts. The evaluator never sees the prompt, whereas the source model is run under two conditions: with the prompt and without it. The result indicates that the source model and the evaluator share a large proportion of their peak words

XSumWikiTextWritingPromptsELI5SQuADCNN/DMPubMedQA
Source model w/ instruction0.6480.6840.6840.6220.6660.6460.548
Source model w/o instruction0.7000.7300.7410.6610.7140.6950.746

[W3 & Q3]: Novelty relative to existing fluctuation-based detectors

Thank you for pointing out these related works. We have added them into Section 2 in the revised paper.

Compared with SpecDetect, WaveDetect, and TSD (When AI Settles Down), RelPPR captures a different signal in a different domain and requires no additional data to train a classifier or tune hyperparameters. Specifically, SpecDetect captures the signal only in the frequency domain, by applying a discrete Fourier transform to the log-probability trajectory. WaveDetect combines the frequency and temporal domains, but requires training a CNN. TSD works in the temporal domain, but has hyperparameters to tune. In contrast, RelPPR measures the prominence of the peaks of the entropy trajectory relative to its mean entropy, and has no hyperparameter to tune.

Besides, RelPPR normalizes the prominence by the sequence's own mean entropy, which corrects PPR for higher entropy human text. To our knowledge, neither SpecDetect nor TSD applies such a normalization, and we regard it as another novelty of our method.

To compare RelPPR with SpecDetect and TSD, and to test the role of mean normalization, we evaluate them in our black-box setting, with OLMo-3.1-32B-Instruct as the source model and Qwen3.5-2B as the evaluator (TSD with window size 20). We further apply spectral energy and peak prominence to both the log-probability and entropy trajectories. For each combination, we evaluate the statistic with and without mean normalization. When normalized, the statistic is divided by the mean statistic of the trajectory it is applied to. As shown in the table below, RelPPR outperforms SpecDetect and TSD. Moreover, for spectral energy on the entropy trajectory, mean-entropy normalization raises the average AUROC from 0.593 to 0.818, which also exceeds vanilla SpecDetect. This further shows the importance of the normalization for these fluctuation-based methods.

MethodXSumSQuADWikiTextWritingPromptsCNN/DMELI5PubMedQAAvg
Existing methods
SpecDetect0.7450.8370.8200.6190.7790.9290.5600.756
TSD0.7150.8250.7860.6680.6950.8260.6030.731
Spectral energy
log-probability, w/o mean normalization (= SpecDetect)0.7450.8370.8200.6190.7790.9290.5600.756
log-probability, w/ mean normalization0.7440.7180.6280.7030.6930.8630.5670.702
entropy, w/o mean normalization0.5050.6460.6840.3690.6160.7140.6160.593
entropy, w/ mean normalization0.8660.8570.7730.8930.8300.9440.5650.818
Peak Prominence Rate (Ours)
log-probability, w/o mean normalization0.3930.2620.3550.4990.3740.1040.4400.347
log-probability, w / mean normalization (= RelPPR-S, Ours)0.8450.7940.8350.8210.8350.8800.5360.792
entropy, w/o mean normalization (= PPR, Ours)0.7080.5310.5800.7590.6260.3900.4290.575
entropy, w / mean normalization (= RelPPR, Ours)0.9450.9340.9290.9490.9180.9800.5560.887

[W4]: Experimental protocol and the instruction asymmetry

Thank you for recognizing that our setting is meaningful. We consider a practical and challenging setting, black-box and instruction-conditioned. A detector sees only the candidate text and has no access to the source model. In this setting, existing detectors suffer substantially as shown in Figure 1a. As analyzed in Section 3, resampling-based detectors cannot regenerate text conditioned on the prompt the source model saw, so the resampled text drifts away from the source distribution, and dual-model detectors lose the cross-model discrepancy they rely on. RelPPR, which needs only a single evaluator model and requires no resampling, achieves state-of-the-art performance in this setting.

Q1: Reconciling log-probability and entropy fluctuations

These two observations are not contradictory, because surprisal and entropy are different quantities that capture different signals, and both can hold simultaneously. Surprisal is the negative log-probability of the token that was actually written, and measures how surprising the token xix_i is to the model given the prefix x<ix_{<i}. Entropy, in contrast, measures the evaluator's uncertainty about the token xix_i given the same prefix x<ix_{<i}. Empirically, LastDE and our RelPPR are built on these two observations respectively, and they achieve average AUROCs of 0.762 and 0.871 in our experimental setting. This shows that both observations hold simultaneously, and that the fluctuation signal in the entropy trajectory is more discriminative.

We hope you find our response with additional experiments and further clarification to be helpful, and consider turning the verdict into an accept.

[1] Ferrer-i-Cancho R, Dębowski Ł, Moscoso del Prado Martín F. Constant conditional entropy and related hypotheses. Journal of Statistical Mechanics. Theory and Experiment, 2013.

I respectfully maintain my original rating. I feel so sorry about that, but despite the authors' thoughtful rebuttal and additional experiments. While the new results strengthen the evidence, but they do not fully validate the central hypothesis regarding local entropy fluctuations. The theoretical justification also remain insufficiently established.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.