acceptodds
Under review as a conference paper at ICLR 2027

What Happens After the Shift? Training-Free Detection of LLM-Generated Text

Abstract

The increasing use of large language models (LLMs) has created a growing need for reliable methods to distinguish human-written from LLM-generated text. Training-free detectors are attractive because they do not require labeled training data or access to the source generator, but existing approaches primarily rely on global likelihood statistics, temporal fluctuations, or spectral characteristics of token probability sequences. In this work, we investigate whether local transition behavior in token log-probability sequences provides an additional discriminative signal. We identify the dominant token-to-token transition and characterize the variability of the likelihood trajectory following this transition. Based on these observations, we propose ExpStabilityDet, a training-free detector that combines global likelihood statistics with transition-aware features, including the dominant transition magnitude and post-transition variability. We further introduce ExpStabilityDet++, a sampling-based extension that evaluates the observed text relative to conditionally generated contrast texts. Across multiple source models, datasets, response lengths, decoding strategies, and languages, our approach consistently demonstrates the usefulness of transition-aware likelihood characteristics. In the primary black-box evaluation, ExpStabilityDet achieves an average AUROC of 0.8877, while ExpStabilityDet++ achieves 0.9157, compared with 0.8785 and 0.8881 for SpecDetect and SpecDetect++, respectively. Additional experiments show that the proposed representation remains informative under cross-lingual and paraphrasing settings and provides a favorable performance–runtime trade-off. These results suggest that the relationship between a salient likelihood transition and its subsequent trajectory provides complementary information for training-free LLM-generated text detection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI text detection, training-free, detection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Jkc5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ExpStabilityDet, a training-free approach for detecting LLM-generated text by analyzing token-level log-probability sequences. The central idea is to identify the largest adjacent change in token log-probabilities and characterize the variability of the sequence following this transition. The resulting features are combined with global likelihood statistics to produce a detection score. Experiments are conducted across multiple datasets, source models, decoding strategies, text lengths, and languages. In the primary black-box evaluation, ExpStabilityDet achieves an average AUROC of 0.8877, compared with 0.8785 for SpecDetect. ExpStabilityDet++ achieves an average AUROC of 0.9157, compared with 0.8881 for SpecDetect++.

Critical strengths

Strength 1

The paper investigates whether the behavior following a prominent change in token log-probabilities provides useful information for detecting LLM-generated text. The proposed features are simple, interpretable, and can be computed without updating the parameters of the proxy language model. This offers a potentially useful alternative to more computationally involved detection methods.

Strength 2

The authors evaluate the proposed methods across multiple source models, datasets, decoding strategies, and languages. In the primary black-box evaluation, ExpStabilityDet improves the average AUROC over SpecDetect from 0.8785 to 0.8877, while ExpStabilityDet++ improves over SpecDetect++ from 0.8881 to 0.9157. These results suggest that the proposed feature combination can be effective across diverse generation settings.

Critical weaknesses

Weakness 1

The current experiments do not sufficiently establish whether the proposed transition-aware features capture genuinely complementary information or primarily provide correlated measurements of existing statistical properties. 

Weakness 2

The paper states that the feature weights and score parameters are calibrated using a held-out calibration set. However, the procedure used to determine these weights is not adequately explained. In particular, it is unclear whether the weights are manually specified, estimated through an unsupervised procedure, or optimized using labeled human-written and machine-generated examples. 

Weakness 3

The lack of detail also makes it difficult to determine whether the reported improvements arise from the proposed statistical features or from the particular feature weighting procedure.

Weakness 4

The abstract is relatively lengthy and repeatedly emphasizes the transition-aware motivation without adding substantial information. Similar redundancy appears throughout the introduction and discussion. More importantly, some claims appear overstated relative to the technical contributions. The manuscript uses expressions such as Appendix 10.3, although the proposed method primarily relies on first-order differences and variance statistics, without a corresponding theoretical framework.

Questions for authors

Q1

Section 3.3 describe a weighted combination of six normalized features, with weights determined through calibration. Could the authors clarify how these weights are obtained and provide their exact values?

Q2

The central novelty lies in the dominant transition magnitude J and post-transition variability C. But, Appendix Table 9 shows that the standard deviation feature alone achieves an AUROC of 0.9799, compared with 0.9820 for the full method. What evidence demonstrates that J and C capture genuinely complementary information rather than correlated measures of likelihood variability?

Q3

This method primarily consists of first-order likelihood differences and variance statistics. Is there a theoretical justification for interpreting these features as measures of stability or underlying generation dynamics? A clearer explanation or controlled comparison would help substantiate the claimed methodological novelty.

3 comments

We sincerely thank the reviewer for their valuable feedback. Below, we address the key concerns raised:

W1: Complementarity of the Proposed Features

The reviewer raises an important question: whether dominant transition magnitude (J) and post-transition variability (C) capture information beyond existing likelihood-fluctuation statistics.

Relationship with existing statistics. We examined the correlations of J and C with SpecDetect and Lastde across 57 evaluation settings.

FeatureLastdeSpecDetect
J0.4480.499
C0.5940.525

The moderate correlations indicate that the proposed features are related to, but are not numerically identical to, existing local-fluctuation and spectral statistics. However, correlation alone cannot establish complementary predictive information.

Incremental predictive-value analysis. We initially evaluated augmenting existing detectors with JJ and CC.

Starting feature setBefore adding (J,C)After adding (J,C)
SpecDetect0.87830.9580
Lastde0.87550.9381

Although these results show improvements for the combined configurations, further analyses indicate that such improvements should not be interpreted as evidence of independent transition-specific information.

The component-wise ablation in Appendix Table 9 provides consistent evidence: the standard deviation feature alone achieves an AUROC of 0.9799, whereas the full configuration achieves 0.9820, a difference of only 0.0021.

A systematic investigation and characterization of transition-conditioned likelihood statistics, including an empirical evaluation of their relationship to existing likelihood-based features. We therefore distinguish the statistical construction of the proposed feature from the claim that they provide independent predictive information. While they define a transition-conditioned characterization of likelihood trajectories, our experiments do not establish strong independent complementarity.

We will revise the manuscript to reflect this finding and moderate our claims concerning the contribution of these features.

W2 and Q1: Calibration Procedure and Exact Weight Estimation

We clarify that the feature weights are obtained using an unsupervised Self-Contrastive Mean Difference (SC-MD) procedure, rather than manual specification or supervised optimization using human-versus-machine labels.

The procedure is as follows:

  1. Calibration data: We use an unlabeled calibration set that is disjoint from the evaluation data.
  2. Self-contrastive generation: For each calibration text, five sequences are generated using the frozen proxy language model.
  3. Feature normalization: The six features are z-normalized using statistics computed from the calibration set.
  4. Weight estimation: Each feature weight is estimated from the mean difference between the observed and self-generated feature statistics.
  5. Weight normalization and orientation: The resulting weights are L1-normalized and oriented toward the hypothesized human-text direction, such that larger scores correspond to human-written text.

The procedure does not require manual weight selection, supervised optimization, or human-versus-machine labels.

The estimated weights (mean ±\pm standard deviation across different calibration sets) are:

Featurebar{\ell}sigma_\ellbar{\Delta}JCP_m)
SC-MD−0.10±0.02-0.10\pm0.02+0.31±0.01+0.31\pm0.01+0.15±0.02+0.15\pm0.02+0.21±0.02+0.21\pm0.02+0.19±0.03+0.19\pm0.03+0.03±0.01+0.03\pm0.01

Calibration sensitivity. We further examined the sensitivity of the detector to changes in the calibration distribution.

Calibration settingMean AUROC
In-distribution0.850
Cross-domain0.844
Unseen generator0.843

Across 52 calibration sources, the mean absolute AUROC change was 0.003, with a 95th-percentile change of 0.011. These results indicate relatively stable ranking performance across the examined calibration settings.

Threshold calibration is separate from weight estimation. We use 75 in-domain human-written texts to determine the operating threshold and estimate the false-positive rate. No machine-generated examples are used for this step, and threshold calibration does not affect AUROC.

We will include the full SC-MD procedure, normalization details, exact feature weights, and the distinction between weight and threshold calibration in Section 3.3 and Appendix 7.2.

Weakness 3: Contribution of the Features versus the Weighting Procedure

To distinguish the contribution of the feature representation from that of the weighting strategy, we compared several weighting procedures under the same evaluation setting.

Weighting strategyLabels requiredMean AUROC
Manual weightingNo0.851
SC-MD (ours)No0.844
PCANo0.805
GMMNo0.712
LDA (reference)Yes0.866

SC-MD provides competitive performance among label-free weighting strategies. However, manual weighting achieves slightly higher mean AUROC, while supervised LDA provides a reference comparison.

These results indicate that SC-MD offers a label-free mechanism for combining the features but is not itself responsible for a unique performance advantage. The detector's predictive performance is primarily associated with likelihood-variability statistics, particularly σℓ\sigma_\ell.

We will therefore separate the claims regarding the calibration mechanism from those concerning the statistical features and avoid attributing the overall performance exclusively to the proposed weighting procedure.

Weakness 4: Presentation and Overstated Claims

We will revise the manuscript to improve conciseness and ensure that the terminology accurately reflects the mathematical formulation and experimental evidence.

Specifically, we will:

  • Shorten the abstract and remove repeated motivation.
  • Consolidate redundant discussions of transition-aware analysis in the Introduction and Discussion.
  • Remove unsupported stability- and generation-dynamics-related interpretations, including the relevant discussion in Appendix 10.3.
  • Replace mechanism-oriented terminology with descriptive statistical terminology.
  • Clarify that JJ and CC are first-order likelihood-difference and variance descriptors, without claiming that they establish a distinct underlying generation mechanism.

The revised contribution will focus on the construction and empirical evaluation of transition-conditioned likelihood statistics.

Q2: Do JJ and CC Capture Genuinely Complementary Information?

Our additional analyses directly examine this question.

First, their correlations with existing likelihood-variability statistics indicate substantial association. Second, residual analysis shows that neither feature independently discriminates human-written from machine-generated text after controlling for the existing features. Third, cross-validated linear probing does not show improvement from their inclusion, and removing them from the six-feature score slightly improves performance.

The ablation results in Appendix Table 9 further show that the full configuration achieves 0.9820 AUROC, compared with 0.9799 using σℓ\sigma_\ell alone.

Although the features may contribute to particular combined configurations, the available evidence does not establish that their contribution arises from genuinely complementary transition-specific information. We will consequently revise our contribution statements to avoid making this claim.

Q3: Theoretical Justification and Controlled Analyses

The proposed features are statistical descriptors, and their mathematical properties do not independently establish a stability or generation-dynamics mechanism.

For a stationary likelihood sequence with variance σ2\sigma^2 and lag-one autocorrelation ρ\rho, the variance of first-order differences satisfies

Var⁡(ℓt+1−ℓt)=2σ2(1−ρ).\operatorname{Var}(\ell_{t+1}-\ell_t) =2\sigma^2(1-\rho).

This relationship demonstrates that first-order differences depend on both likelihood dispersion and temporal correlation. Similarly, CC is a variance statistic calculated over the segment selected by the dominant transition. Consequently, these features may reflect statistical properties of the likelihood trajectory without necessarily representing underlying generation dynamics.

We additionally conducted temporal-order perturbation experiments using random shuffling, phase-randomized surrogates, and IAAFT surrogates. These transformations did not degrade detection performance. Moreover, the post/pre-transition variance ratio achieved an AUROC of 0.502, indicating no discriminative evidence for the proposed transition-specific variance relationship in this experiment.

These findings motivate a more conservative interpretation. We will remove the stability and generation-dynamics framing and describe JJ and CC as statistical descriptors of likelihood variation. The revised manuscript will emphasize their mathematical construction and empirical behavior rather than assigning them an unsupported mechanistic interpretation.

We thank the reviewer for prompting these additional analyses and clarifications, which have helped us improve the transparency, precision, and scientific rigor of the manuscript.

Thank you for the rebuttal and additional analyses. I appreciate the authors' efforts to address my concerns. However, I regretfully maintain my original rating.

Firstly, I kindly suggest that the authors carefully check the formatting of their responses before submission, as several mathematical expressions and symbols appear to have been copied without proper rendering.

Secondly, the new results further confirm my original concerns: the proposed features do not demonstrate clear independent predictive value, and the claimed stability mechanism lacks sufficient theoretical and empirical support.

Reviewer_qc8vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ExpStabilityDet, a training-free method for detecting AI-generated text. It uses token-level log probabilities and focuses on two features: dominant transition magnitude and post-transition variability. Experiments on multiple datasets show promising results.

Critical strengths

The paper examines local changes in token-level log-probability sequences, offering a different perspective from existing likelihood-based and frequency-based detection methods. The method does not require a separately trained supervised detector, making it potentially useful when labeled training data are unavailable. The experiments cover different source models, domains, languages, and generation settings, with comparisons against several training-free baselines.

Critical weaknesses

The main contribution is the introduction of dominant transition magnitude and post-transition variability. However, the differences between these features and existing measures of local fluctuations or spectral statistics are not sufficiently clear. More evidence is needed to show that the proposed features capture genuinely complementary information. The detector relies on feature normalization and calibrated scoring parameters. The paper should clarify what data and supervision are required for calibration and examine how sensitive the method is to changes in the calibration distribution. The main evaluation uses only 150 human-written samples and their corresponding LLM continuations for each dataset. This relatively small sample size raises concerns about the stability of the reported improvements. Confidence intervals, statistical significance tests, or evaluations on larger independent datasets would make the conclusions more convincing. Although the experiments cover multiple models and languages, the method's performance on short texts, heavily paraphrased content, and cases where the source model differs substantially from the proxy model remains unclear. Further evaluation under these settings would help establish the method's robustness and practical applicability.

Questions for authors

Please refer to the weaknesses mentioned above and address these concerns in the revised manuscript.

2 comments

We sincerely thank the reviewer for their valuable feedback. Below, we address the key concerns raised:

W1: Difference from existing local fluctuation and spectral statistics

Response:

Comparison dimensionSpecDetectLastdeOurs
Mathematical foundationDFT-based spectral energyTime-delay embedding and Diversity EntropyFirst-order likelihood differences and transition-conditioned variance
Primary statisticsSpectral energyMultiscale Diversity EntropyDominant transition magnitude (J) and post-transition variability (C)
Temporal characterizationGlobal frequency-domain statisticsMultiscale local fluctuationsExplicit dominant-transition localization
Transition analysisNo explicit transition localizationAggregates local patterns across windowsIdentifies t^* and measures subsequent variability
Computational complexityO(n\log n)Depends on embedding and window configurations(O(n))

Table 1: Comparison of SpecDetect, Lastde, and Our.

Incremental contribution of J and C:

Starting feature setBefore J & CAfter J & C
\SpecDetect0.87830.9580
Lastde0.87550.9381

These results suggest that adding JJ and CC can improve combined performance, although their independent contribution remains unclear. We will incorporate these analyses and interpret the findings cautiously.

W2: Calibration and sensitivity to the calibration distribution

Response: As described in Section 3.3 and Appendix 7.2, ExpStabilityDet uses unsupervised Self-Contrastive Mean Difference (SC-MD) calibration. Feature weights are estimated by contrasting observed-text features with those obtained from sequences sampled using a frozen proxy language model. Feature normalization statistics are computed from an unlabeled calibration set disjoint from the test data. Neither procedure requires human-versus-machine labels or supervised classifier training.

We further clarify that threshold calibration is a separate step from feature-weight estimation. For threshold-based classification, we use 75 in-domain human-written reference texts to set the operating threshold and estimate the false-positive rate (FPR), without requiring machine-generated examples. We also evaluate sensitivity to calibration distribution and size. Across cross-domain, unseen-generator, and in-distribution calibration settings, we obtain mean AUROCs of 0.844, 0.843, and 0.850, respectively. Across 52 calibration sources, the mean absolute AUROC change is only 0.003, with a 95th percentile of 0.011. Furthermore, using 10, 25, 50, and 100 calibration texts per class yields AUROCs of 0.839, 0.848, 0.845, and 0.846, respectively, indicating relatively stable performance with varying calibration-set sizes. For threshold calibration, the 75 in-domain human-written references produce an FPR of 6.2% against a target of 5%.

We will clarify the calibration requirements, weight estimation, normalization, threshold calibration, and sensitivity analysis in Section 3.3 and Appendix 7.2.

W3:Small evaluation size and statistical significance

Response: We would like to clarify that the evaluation is not restricted to 150 samples throughout the paper.

First, the primary controlled evaluation uses 150 human-written and 150 machine-generated samples per dataset, following the evaluation protocol adopted by several existing baseline studies to ensure a consistent comparison. However, we additionally investigate the effect of evaluation size in Figure 3(b), Section 4.4 of the main paper, and Appendix Section 11.1 (Table 12), where we report performance on XSum, WritingPrompts, and Reddit using different sample sizes (50, 100, and 150).

Second, we evaluate ExpStabilityDet on larger existing benchmarks, as reported in Appendix Section 12.2 (Table 15). These evaluations include HC3 with 1,000 samples, RAID with 5,000 samples, and M4 with 10,000 samples. Thus, our evaluation also includes substantially larger datasets beyond the 150-sample controlled setting.

W4:Short texts, paraphrasing, and source–proxy model mismatch

Response:

  1. Short texts: Beyond the length analysis already reported in Figure 3(a), Appendix 11.1, and Table 11, we evaluated inputs of 16, 32, 64, 128, and 256 tokens, obtaining AUROCs of 0.703, 0.747, 0.817, 0.862, and 0.876, respectively.

  2. Heavy paraphrasing: Beyond the existing results in Figure 5(b), we evaluated heavily paraphrased XSum samples generated using GPT-4 Turbo. ExpStabilityDet achieves an AUROC of 0.632, highlighting the challenge posed by aggressive paraphrasing.

  3. Source–proxy mismatch: The main paper evaluates five source–proxy model pairs in Figure 5(a). We will expand this analysis with additional model pairs to further examine the impact of source–proxy mismatch.

We will incorporate the additional experiments and discuss these limitations in the revised paper.

Thank you for the authors' detailed responses and clarifications. After carefully considering the rebuttal and the content of the paper, I am inclined to maintain my initial evaluation.

Reviewer_7wghOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ExpStabilityDet, a training-free method for detecting LLM-generated text. In addition to examining global statistics of token log-probability sequences, the method explicitly analyzes the most salient local transitions in the sequences and the stability of the subsequent trajectories to distinguish human-written text from LLM-generated text.

Critical strengths

The authors used five datasets covering three languages.

Critical weaknesses

The study examines text continuations rather than texts generated entirely by LLMs, thereby introducing limitations in its applicability.

Questions for authors

How would the performance be affected if a number of tokens other than 30 were used as input?

1 comment

We thank the reviewer for highlighting the limitations of using human-written prefixes to construct machine-generated continuations and for asking about the effect of varying the prompt length. Comment 1: Evaluation of continuations rather than entirely LLM-generated texts

Response: We would like to clarify that our primary evaluation focuses on machine-generated continuations conditioned on human-written prefixes, following the experimental protocols adopted in prior baseline works, including Lastde and SpecDetect. This choice ensures a fair and consistent comparison with existing methods.

However, our evaluation is not restricted to continuation-based texts. We have also evaluated our proposed approach on existing benchmark datasets containing fully LLM-generated texts without human-written tokens in the generated portion. As already mentioned in the manuscript (Line 363), additional benchmark evaluations against existing methods on HC3, RAID, and M4 are provided in Appendices 12.1 and 12.2. In particular, Appendix Table 15 reports the performance comparison on these existing benchmark datasets.

We conducted additional experiments using fully LLM-generated texts without human-written tokens in the generated portion.

Evaluation SettingσℓLastdeExpStabFusionLastde++ExpStab++
XSum, Qwen2.5-7B-Instruct0.8140.7650.7880.8380.8370.846
XSum, Mistral-7B-Instruct0.9550.9420.9390.9660.9520.966
DetectAIRev Review–Deepseek0.9230.9300.9500.9530.9510.960

Table 2: Performance comparison of likelihood-based baselines and proposed methods (ExpStabilityDet and ExpStabilityDet++) on fully LLM-generated texts across XSum and DetectAIRev benchmark settings.

The results demonstrate that likelihood-based detection remains effective when evaluating fully machine-generated texts.

Comment 2: Effect of using different prompt lengths instead of 30 tokens Response: We thank the reviewer for this insightful question. We initially adopted a 30-token human-written prefix to maintain consistency with the experimental protocols used in prior baseline studies, including Lastde and SpecDetect, thereby ensuring a fair comparison with existing methods.

Nevertheless, to investigate the sensitivity of our approach to prompt length, we conducted additional experiments by varying the human-written prefix length across 10, 20, 30, and 50 tokens. The results are reported below.

Prompt Length10203050
GPT-2 XL: σℓ0.9000.8690.8370.884
GPT-2 XL: ExpStabilityDet0.9020.8590.8350.883
GPT-2 XL: ExpStabilityDet++0.9510.9230.8940.870
Qwen2.5-7B: σℓ0.9660.9600.9350.927
Qwen2.5-7B: ExpStabilityDet0.9650.9630.9480.937
Qwen2.5-7B: ExpStabilityDet++0.9610.9750.9590.986

Table 2: Performance comparison of σℓ\sigma_\ell, ExpStabilityDet, and ExpStabilityDet++ under varying human-written prompt lengths (10, 20, 30, and 50 tokens) for GPT-2 XL and Qwen2.5-7B.

The results indicate that detection performance varies with prompt length across different models and detection methods. While some configurations exhibit a gradual performance decline with increasing prompt length, others show different trends. Notably, ExpStabilityDet++ achieves the best performance at 10 tokens for GPT-2 XL (0.951) and at 50 tokens for Qwen2.5-7B (0.986).

These findings suggest that the influence of prompt length is model- and method-dependent rather than strictly monotonic. We will incorporate these additional experiments into the revised manuscript and discuss the sensitivity of detection performance to different prompt lengths.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.