- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes ExpStabilityDet, a training-free approach for detecting LLM-generated text by analyzing token-level log-probability sequences. The central idea is to identify the largest adjacent change in token log-probabilities and characterize the variability of the sequence following this transition. The resulting features are combined with global likelihood statistics to produce a detection score. Experiments are conducted across multiple datasets, source models, decoding strategies, text lengths, and languages. In the primary black-box evaluation, ExpStabilityDet achieves an average AUROC of 0.8877, compared with 0.8785 for SpecDetect. ExpStabilityDet++ achieves an average AUROC of 0.9157, compared with 0.8881 for SpecDetect++.
Critical strengths
Strength 1
The paper investigates whether the behavior following a prominent change in token log-probabilities provides useful information for detecting LLM-generated text. The proposed features are simple, interpretable, and can be computed without updating the parameters of the proxy language model. This offers a potentially useful alternative to more computationally involved detection methods.
Strength 2
The authors evaluate the proposed methods across multiple source models, datasets, decoding strategies, and languages. In the primary black-box evaluation, ExpStabilityDet improves the average AUROC over SpecDetect from 0.8785 to 0.8877, while ExpStabilityDet++ improves over SpecDetect++ from 0.8881 to 0.9157. These results suggest that the proposed feature combination can be effective across diverse generation settings.
Critical weaknesses
Weakness 1
The current experiments do not sufficiently establish whether the proposed transition-aware features capture genuinely complementary information or primarily provide correlated measurements of existing statistical properties.
Weakness 2
The paper states that the feature weights and score parameters are calibrated using a held-out calibration set. However, the procedure used to determine these weights is not adequately explained. In particular, it is unclear whether the weights are manually specified, estimated through an unsupervised procedure, or optimized using labeled human-written and machine-generated examples.
Weakness 3
The lack of detail also makes it difficult to determine whether the reported improvements arise from the proposed statistical features or from the particular feature weighting procedure.
Weakness 4
The abstract is relatively lengthy and repeatedly emphasizes the transition-aware motivation without adding substantial information. Similar redundancy appears throughout the introduction and discussion. More importantly, some claims appear overstated relative to the technical contributions. The manuscript uses expressions such as Appendix 10.3, although the proposed method primarily relies on first-order differences and variance statistics, without a corresponding theoretical framework.
Questions for authors
Q1
Section 3.3 describe a weighted combination of six normalized features, with weights determined through calibration. Could the authors clarify how these weights are obtained and provide their exact values?
Q2
The central novelty lies in the dominant transition magnitude J and post-transition variability C. But, Appendix Table 9 shows that the standard deviation feature alone achieves an AUROC of 0.9799, compared with 0.9820 for the full method. What evidence demonstrates that J and C capture genuinely complementary information rather than correlated measures of likelihood variability?
Q3
This method primarily consists of first-order likelihood differences and variance statistics. Is there a theoretical justification for interpreting these features as measures of stability or underlying generation dynamics? A clearer explanation or controlled comparison would help substantiate the claimed methodological novelty.
3 comments
Weakness 3: Contribution of the Features versus the Weighting Procedure
To distinguish the contribution of the feature representation from that of the weighting strategy, we compared several weighting procedures under the same evaluation setting.
| Weighting strategy | Labels required | Mean AUROC |
|---|---|---|
| Manual weighting | No | 0.851 |
| SC-MD (ours) | No | 0.844 |
| PCA | No | 0.805 |
| GMM | No | 0.712 |
| LDA (reference) | Yes | 0.866 |
SC-MD provides competitive performance among label-free weighting strategies. However, manual weighting achieves slightly higher mean AUROC, while supervised LDA provides a reference comparison.
These results indicate that SC-MD offers a label-free mechanism for combining the features but is not itself responsible for a unique performance advantage. The detector's predictive performance is primarily associated with likelihood-variability statistics, particularly .
We will therefore separate the claims regarding the calibration mechanism from those concerning the statistical features and avoid attributing the overall performance exclusively to the proposed weighting procedure.
Weakness 4: Presentation and Overstated Claims
We will revise the manuscript to improve conciseness and ensure that the terminology accurately reflects the mathematical formulation and experimental evidence.
Specifically, we will:
- Shorten the abstract and remove repeated motivation.
- Consolidate redundant discussions of transition-aware analysis in the Introduction and Discussion.
- Remove unsupported stability- and generation-dynamics-related interpretations, including the relevant discussion in Appendix 10.3.
- Replace mechanism-oriented terminology with descriptive statistical terminology.
- Clarify that and are first-order likelihood-difference and variance descriptors, without claiming that they establish a distinct underlying generation mechanism.
The revised contribution will focus on the construction and empirical evaluation of transition-conditioned likelihood statistics.
Q2: Do and Capture Genuinely Complementary Information?
Our additional analyses directly examine this question.
First, their correlations with existing likelihood-variability statistics indicate substantial association. Second, residual analysis shows that neither feature independently discriminates human-written from machine-generated text after controlling for the existing features. Third, cross-validated linear probing does not show improvement from their inclusion, and removing them from the six-feature score slightly improves performance.
The ablation results in Appendix Table 9 further show that the full configuration achieves 0.9820 AUROC, compared with 0.9799 using alone.
Although the features may contribute to particular combined configurations, the available evidence does not establish that their contribution arises from genuinely complementary transition-specific information. We will consequently revise our contribution statements to avoid making this claim.
Q3: Theoretical Justification and Controlled Analyses
The proposed features are statistical descriptors, and their mathematical properties do not independently establish a stability or generation-dynamics mechanism.
For a stationary likelihood sequence with variance and lag-one autocorrelation , the variance of first-order differences satisfies
This relationship demonstrates that first-order differences depend on both likelihood dispersion and temporal correlation. Similarly, is a variance statistic calculated over the segment selected by the dominant transition. Consequently, these features may reflect statistical properties of the likelihood trajectory without necessarily representing underlying generation dynamics.
We additionally conducted temporal-order perturbation experiments using random shuffling, phase-randomized surrogates, and IAAFT surrogates. These transformations did not degrade detection performance. Moreover, the post/pre-transition variance ratio achieved an AUROC of 0.502, indicating no discriminative evidence for the proposed transition-specific variance relationship in this experiment.
These findings motivate a more conservative interpretation. We will remove the stability and generation-dynamics framing and describe and as statistical descriptors of likelihood variation. The revised manuscript will emphasize their mathematical construction and empirical behavior rather than assigning them an unsupported mechanistic interpretation.
We thank the reviewer for prompting these additional analyses and clarifications, which have helped us improve the transparency, precision, and scientific rigor of the manuscript.
Thank you for the rebuttal and additional analyses. I appreciate the authors' efforts to address my concerns. However, I regretfully maintain my original rating.
Firstly, I kindly suggest that the authors carefully check the formatting of their responses before submission, as several mathematical expressions and symbols appear to have been copied without proper rendering.
Secondly, the new results further confirm my original concerns: the proposed features do not demonstrate clear independent predictive value, and the claimed stability mechanism lacks sufficient theoretical and empirical support.
We sincerely thank the reviewer for their valuable feedback. Below, we address the key concerns raised:
W1: Complementarity of the Proposed Features
The reviewer raises an important question: whether dominant transition magnitude (J) and post-transition variability (C) capture information beyond existing likelihood-fluctuation statistics.
Relationship with existing statistics. We examined the correlations of J and C with SpecDetect and Lastde across 57 evaluation settings.
The moderate correlations indicate that the proposed features are related to, but are not numerically identical to, existing local-fluctuation and spectral statistics. However, correlation alone cannot establish complementary predictive information.
Incremental predictive-value analysis. We initially evaluated augmenting existing detectors with J and C.
Although these results show improvements for the combined configurations, further analyses indicate that such improvements should not be interpreted as evidence of independent transition-specific information.
The component-wise ablation in Appendix Table 9 provides consistent evidence: the standard deviation feature alone achieves an AUROC of 0.9799, whereas the full configuration achieves 0.9820, a difference of only 0.0021.
A systematic investigation and characterization of transition-conditioned likelihood statistics, including an empirical evaluation of their relationship to existing likelihood-based features. We therefore distinguish the statistical construction of the proposed feature from the claim that they provide independent predictive information. While they define a transition-conditioned characterization of likelihood trajectories, our experiments do not establish strong independent complementarity.
We will revise the manuscript to reflect this finding and moderate our claims concerning the contribution of these features.
W2 and Q1: Calibration Procedure and Exact Weight Estimation
We clarify that the feature weights are obtained using an unsupervised Self-Contrastive Mean Difference (SC-MD) procedure, rather than manual specification or supervised optimization using human-versus-machine labels.
The procedure is as follows:
The procedure does not require manual weight selection, supervised optimization, or human-versus-machine labels.
The estimated weights (mean ± standard deviation across different calibration sets) are:
Calibration sensitivity. We further examined the sensitivity of the detector to changes in the calibration distribution.
Across 52 calibration sources, the mean absolute AUROC change was 0.003, with a 95th-percentile change of 0.011. These results indicate relatively stable ranking performance across the examined calibration settings.
Threshold calibration is separate from weight estimation. We use 75 in-domain human-written texts to determine the operating threshold and estimate the false-positive rate. No machine-generated examples are used for this step, and threshold calibration does not affect AUROC.
We will include the full SC-MD procedure, normalization details, exact feature weights, and the distinction between weight and threshold calibration in Section 3.3 and Appendix 7.2.