- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates zero-shot LLM-generated text detection under instruction-conditioned generation in the black-box setting. The authors first analyze the performance degradation of existing dual-model and resampling-based detectors, attributing it to reduced cross-model discriminability and the lack of access to the original instruction prompt, respectively. Motivated by the hypothesis that machine-generated text exhibits stronger local entropy fluctuations than human-written text, the paper proposes Relative Peak Prominence Rate (RelPPR), a detection metric that measures local peak prominence along token-level entropy trajectories, normalized by the sequence-level mean entropy. The proposed method requires only a single evaluator model and does not involve additional training or resampling.
Critical strengths
- The paper clearly motivates the need to study LLM-generated text detection under instruction-conditioned generation, a realistic yet underexplored setting.
Critical weaknesses
- While the paper identifies why existing detectors struggle with instruction-conditioned generation, these findings do not directly motivate the use of local entropy fluctuations as an alternative detection signal. It remains unclear why entropy fluctuations should remain discriminative under the same conditions that undermine existing methods, leaving a gap between the problem analysis and the proposed solution.
- While the Uniform Information Density hypothesis provides a reasonable motivation for studying information fluctuations, it does not directly establish that human-written text should exhibit smoother predictive entropy trajectories when evaluated by a surrogate language model. Similarly, prior findings on high-entropy forking tokens do not establish that these fluctuations are systematically stronger in machine-generated text than in human writing. The paper would benefit from a more direct empirical validation of this proposed distinction.
- Previous works, such as LastDE and SurpMark [1], have already explored token-level probability dynamics as signals for machine-generated text detection. The paper does not sufficiently explain why peak prominence offers a more informative characterization of these dynamics than simpler fluctuation statistics, such as variance, adjacent differences, or transition-based measures.
[1] Black-Box Detection of LLM-Generated Text Using Generalized Jensen Shannon Divergence, ICML 2026
Questions for authors
- Why does RelPPR perform substantially worse in the white-box setting? Could the authors further investigate this behavior?
- The experiments use fixed task-specific instruction templates for each dataset. Would the observed entropy fluctuation patterns and detection performance remain consistent under different instruction formulations, such as stricter formatting requirements or different writing styles?
2 comments
Thank you for the additional experiments and clarifications. The new comparisons in W3 are helpful, showing that RelPPR outperforms normalized total variation and standard deviation. The additional experiments in Q2 also provide useful evidence of robustness across different instruction templates.
However, my main concerns remain partially unresolved. For W1 and W2, the authors still rely on UID and forking tokens as motivation without directly validating that machine-generated text exhibits stronger local entropy fluctuations than human-written text under instruction-conditioned generation. The new W3 experiments demonstrate the effectiveness of RelPPR, but do not validate this underlying hypothesis.
Overall, I appreciate the additional experiments, but the central methodological motivation remains insufficiently established. I therefore maintain my original assessment.
Thank you for your detailed review. We appreciate that you recognize the realistic yet underexplored setting we study. We address your remaining concerns below.
[W1]: From the failure analysis to local entropy fluctuations
We state the motivation of our method in Section 4.1 and briefly summarize it here. According to the Uniform Information Density principle, humans keep the rate of information transmission steady, which yields a smooth entropy trajectory. In contrast, LLM generation contains forking tokens with high entropy, which alternate with predictable stretches and produce higher-amplitude local fluctuations. We further conjecture that instruction conditioning accentuates the peaks at forking tokens. We therefore detect machine text by quantifying the fluctuations of the entropy trajectory.
[W2]: UID hypothesis and forking tokens
UID is a hypothesis about human writing, and forking tokens are a phenomenon observed in the literature. Both motivate the study of fluctuations in the entropy trajectory. However, to our knowledge, no prior work has proposed an effective quantity that captures this kind of fluctuation in the entropy trajectory. We propose RelPPR as such a quantity, and it achieves state-of-the-art performance on this very challenging detection task.
[W3]: Peak prominence versus simpler statistics
Unlike LastDE and SurpMark, which operate on the log-probability trajectory, RelPPR operates on the entropy trajectory. To examine whether peak prominence characterizes these dynamics more informatively than simpler fluctuation statistics, we compare RelPPR with two simpler statistics of the entropy trajectory, total variation, i.e., the sum of absolute differences between adjacent entropy values, ∑t∣Ht−Ht−1∣, and standard deviation, each with and without mean-entropy normalization. We use OLMo-3.1-32B-Instruct as the source model and Qwen3.5-2B as the evaluator. As shown in the table below, RelPPR achieves the best average AUROC (0.887), ahead of the best simple statistic (0.820), and is best on six of the seven datasets. Mean-entropy normalization substantially improves both simpler statistics, raising the average AUROC of total variation from 0.528 to 0.818 and that of the standard deviation from 0.408 to 0.820. This confirms that the benefit of mean-entropy normalization is not limited to PPR, but extends to statistics based on entropy trajectory in general, since normalization removes the bias caused by the typically higher entropy of human text as we analyze in Section 4.2.
[Q1]: White-box setting
In the white-box setting, one of the two evaluator models used by dual-model detectors is the source model itself, which has low uncertainty about the text, while the other model remains uncertain. This restores the cross-model discrepancy that dual-model detectors rely on, and these detectors therefore achieve strong performance in the white-box setting.We would like to emphasize that the white-box setting is not practical, since real detectors usually have access to neither the source model nor the prompt.
[Q2]: Instruction templates
To evaluate the sensitivity to the instruction template, we construct two alternative templates in addition to the default one. The strict template adds formatting requirements to the instruction, and the style template asks the model to write with a persona fitted to the dataset. We regenerate the machine text under each template and compare the detection methods across all three. As shown in the table below, RelPPR still achieves the best performance under every template, which shows its robustness to the instruction template. We have added this experiment to Section 5 of the revised paper, with template details in Table11 in the appendix.
We hope you find our response with additional experiments and further clarification to be helpful, and consider turning the verdict into an accept.