acceptodds
Under review as a conference paper at ICLR 2027

Mary Had a Little [What?]: Decoding Can Matter More Than the Model Itself in Zero-Shot Detection

Abstract

Likelihood-based zero-shot detectors seek to identify the source of a text by comparing its observed tokens with what an evaluator LLM expects to see. We find that this relationship can depend less on who wrote the text than on how it was sampled, which is a routine generation choice. Across six evaluator LLMs and eight domains, applying standard truncation (top-k/top-p) to an evaluator's own generation moves its liketropy score about twice as far as switching to another LLM, and more than ten times as far as switching to a human writer. Calibrated to accept 95% of the evaluator's raw-sampled text, liketropy rejects 78% of its truncated text yet accepts 88% of human text as its own. To explain this, we derive a decomposition of the expected score for a family of log-likelihood-based detectors. It combines an entropy-based configuration term, which is highly sensitive to how narrowly the generator samples, and an identity term, which depends on the KL divergence from the generator to the evaluator. Consistent with this mechanism, in a complementary experiment involving five detectors, switching the generator from truncated to full-vocabulary sampling drags AUROCs as high as 0.99 down to chance or below, while a supervised classifier outside the family still separates the same texts with AUROC 0.84.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine-generated text detection, zero-shot detection, LLM text attribution, decoding strategy, truncated sampling, KL divergence, entropy, statistical hypothesis testing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MSzGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is centered on the zero-shot text detection problem. It finds that the influence of decoding settings can be even larger than that of model identity. The authors decompose the expected score of a family of likelihood-based detectors and show that the score is influenced by both the entropy term and the KL term. In experiments, the authors find that changing decoding settings, such as truncation, can substantially affect detection performance, while changing model identity has a smaller effect. Overall, this paper provides an interesting perspective on the high sensitivity of these detection methods to decoding settings and provides insights for related research.

Critical strengths

I summarize the core strengths of the paper as follows.

  1. The research topic is practical and meaningful. Decoding is a normal setting of large language model generation. The paper evaluates whether the detector judgment would change explicitly while the decoding settings change, which would be beneficial for understanding whether the evaluation results of the generation setting are generalizable.

  2. The theoretical analysis and controlled experiments are clear. The authors decompose the expected score into entropy and KL terms and execute different controlled experiments. This would make the analysis not only stay at the performance change, but also explain where the change comes from.

  3. The paper provides validation beyond a single method and contains different types of comparisons. Besides liketropy, the authors also test some other methods, such as rank-based and supervised detectors. The supervised detectors retain some distinguishability on the same text, which would help explain that the observed failure mode is related to the detector methods and cannot be attributed to the text itself.

Critical weaknesses

I summarize the core weaknesses of the paper as follows.

  1. This paper is more like an expansion of known phenomena and theory, with incremental contributions. As acknowledged by the authors, the fact that changing decoding settings can influence detection has been pointed out by prior works [1, 2]. Meanwhile, the entropy–KL decomposition underlying liketropy also appears in prior work [3]. The core contribution of this paper is to extend the decomposition to a family of likelihood-based detectors and empirically test the influence of decoding settings and generator differences on the scores. Thus, the novelty of the paper may be relatively limited, though the authors have stated the differences from prior work. Further clarification of which conclusions come from prior work and what the new conclusions imply would make the contribution of the current work clearer.

  2. The generalizability and applicability are relatively limited. The paper mainly covers a family of likelihood-based zero-shot detectors. Meanwhile, there are other zero-shot methods, such as the black-box version of DNA-GPT [4] and DetectGPT [5], which are not evaluated in the paper. Though the authors conduct experiments on rank-based methods, these methods are not covered by the theoretical analysis of the paper. Thus, it is still unclear whether the mechanism described in the paper is also applicable to a broader range of zero-shot detectors.

  3. It is unclear whether the failure persists as model size increases. The authors clearly state in the limitations that hardware constraints limit model size to around 8 billion parameters. However, due to this constraint, the current evidence is still insufficient to judge whether the failure mode reported in the paper would persist, become weaker, or become stronger as model size increases.

[1] https://arxiv.org/abs/1911.00650

[2] https://proceedings.mlr.press/v258/kempton25a.html

[3] https://arxiv.org/abs/2501.02406

[4] https://arxiv.org/abs/2305.17359

[5] https://arxiv.org/abs/2301.11305

Questions for authors
  1. Is the conclusion of this paper also applicable to the other zero-shot detector methods?
  2. Could the authors provide some small-scale experiments on models with larger sizes?
2 comments

We thank you for the careful reading. We begin with the new experiment on larger models.

Experiments on Larger Models. We followed your suggestion and reran the full protocol with Gemma-2-27B and Qwen2.5-32B as evaluators, which are significantly larger than the six original ones. Both large evaluators reproduce the failure of zero-shot detection, and every statistic falls within or close to the range of the six original evaluators. The results are summarized below.

EvaluatorOwn truncated text rejectedHuman text acceptedDecoding vs model shift (2×2)Config. term / identity term
Original77.4% (51% to 93%)88.3% (78% to 95%)2.285.45
Gemma-2-27B69.3%87.0%1.725.37
Qwen2.5-32B57.4%87.5%1.955.15

As can be seen, the qualitative findings remain unchanged. The five conditions keep the same order by distance from zero (A < Human < B’’ < B < A’’). The identity term still makes up only about 16% of the combined size of the two terms. Finally, changing the decoding still moves the score further than changing the generator. We will add these results as a new appendix.

Positioning of our contributions. The paper exposes an intrinsic limitation of likelihood-based zero-shot detection: it is so sensitive to what are usually regarded as minor implementation details (e.g., the difference between a language model and a mildly truncated version of it, which are routinely treated as the same model) that detection itself can fail. In this sense, the paper does more than report that “decoding setting influences the score.” Instead, it shows (i) that the effect is large enough to break the detector and (ii) why this happens.

  1. Failure of the detectors. Under raw sampling, the test accepts 88% of human text as the model’s own. In comparison, truncation alone (top-k 40, top-p 0.96, as configured in [3]’s pipeline) pushes the score of the model’s own text far from zero, so the test rejects 77% of it as not self-generated. (Note that [3]’s Theorem 1(a) covers only full-vocabulary sampling, yet [3] evaluates on truncated text and labels it as the model’s own.) The separation [3] reports (with AUROC up to 0.99) therefore comes mainly from the decoding difference, not from recognizing who wrote the text since under raw sampling, AUROC falls drastically to 0.36–0.56.

  2. The mechanism. We decompose the expected score of every detector in the log-likelihood family into a configuration term and an identity term (Proposition 1). The configuration term, which [3] folds into an error term, becomes 3.7× to 7.4× the identity term on the evaluator’s own truncated text. Hence it is the sensitivity of this term on model decoding configuration that ultimately leads to the failure of the detector.

Scope. We agree that our theory focuses on likelihood-based tests, and the paper states these boundaries explicitly; see Equation 2 and Table 2. DetectGPT’s T5 perturbation step falls outside that form, but its analytic successor Fast-DetectGPT is inside it and falls from 0.99 to 0.53; see Table 3. Rank scores, the second half of GLTR, are reported in Appendix J as a robustness check outside the theory, and fail in the same way. Black-box methods such as DNA-GPT do not use token probabilities and lie outside the scope of this paper. In the camera-ready version, we will further clarify those points.

We hope these results and clarifications address your concerns. If so, we would be grateful if you would take them into account in your assessment.

AuthorsOpenReview ↗

Gentle Follow-up

Dear Reviewer MSzG,

Thank you again for your time and constructive review. We wanted to gently follow up to let you know that we have uploaded a revised version of the paper incorporating our responses, with all changes highlighted in blue.

In response to your comments, the revision adds experiments with two larger language models, Gemma-2-27B and Qwen2.5-32B (Appendix O), and clarifies our positioning relative to prior work and the scope of the theory.

We would be grateful to hear whether our response and the revision address your concerns, and we are happy to clarify anything further before the discussion period closes.

Best regards, The authors

Reviewer_SW5JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the impact of decoding strategies on likelihood-based zero-shot LLM-generated text detection. The authors show that changing the decoding configuration, such as applying top-k or top-p sampling, can affect detection scores more than switching between different source models. Using Liketropy as the main case study, they demonstrate that a detector may reject text generated by its own evaluator model while accepting text from other models or human writers. To explain this phenomenon, the paper decomposes the expected detection score into a configuration term related to entropy and an identity term based on KL divergence. The authors further identify two failure modes: over-sensitivity to decoding truncation and insufficient sensitivity to differences between generators. Experiments across multiple models, datasets, and detection methods suggest that these issues extend beyond Liketropy to other likelihood-based detectors. The paper concludes by discussing the limitations of these methods and the challenges of obtaining reliable detection guarantees without access to the generator's distribution.

Critical strengths
  1. The paper systematically disentangles the effects of generator identity and decoding configuration across multiple evaluator models and text domains.
Critical weaknesses
  1. The paper's most striking results rely on comparing truncated decoding with raw full-vocabulary sampling, which may not adequately reflect typical deployment settings. The evaluation covers a limited range of decoding configurations, and the top-k sweep primarily examines changes in the detection score rather than end-to-end detection performance across diverse practical settings. It remains unclear how broadly the reported failures generalize to commonly used decoding strategies.
  2. Although the paper demonstrates the sensitivity of likelihood-based detectors to decoding configurations, it does not sufficiently explore whether these failures can be mitigated under commonly used sampling strategies.
Questions for authors

My main concern is that, from a sampling distribution perspective, the finding does not seem particularly new or surprising, since changing decoding strategies naturally changes the underlying distribution. The framing around model identity makes the problem more interesting, but I'm still not sure how much new insight the paper adds. Even under this framing, I think it is important to evaluate how much detection performance actually degrades when texts generated with different, commonly used decoding strategies are mixed together, as this would better demonstrate the practical significance of the finding.

4 comments

New experiment: common decoding settings and their mixture. Following your suggestion, we have generated text under six new common decoding settings and pooled them with the paper's two settings into one mixture of all eight. All six evaluators and eight datasets are used, and the metrics are end-to-end, as you requested.

Decoding settingMean liketropyOwn text acceptedHuman acceptedAUROC
Paper's settings
raw−0.0395%88%0.56
top-k 40, top-p 0.96+0.5811%100%0.04
Combined
llama.cpp defaults (top-k 40, top-p 0.95, min-p 0.05, T 0.8) + rep. penalty 1.1+0.4434%99%0.16
T 0.7, top-p 0.95+0.862%100%0.01
Single
top-k 50 (Hugging Face default)+0.4430%100%0.12
top-p 0.90+0.4616%99%0.06
top-p 0.95+0.2966%97%0.23
T 0.7+0.802%100%0.01
Mixture of all eight+0.4832%100%0.15

In the table above, Own text accepted corresponds to the percentage of text accepted as self-generated at the threshold which accepts 95% of the model’s own raw text. Human accepted corresponds to the percentage of human text accepted as Model A’s generation when the threshold instead accepts 95% of the model’s own text generated under current row’s setting. In the mixture row, the threshold is recalibrated on the model’s own text pooled over all eight settings. AUROC represents the ability to distinguish the model’s own text from human text for the test’s two-sided rule. (Below 0.5, the test finds human text more self-generated than the model’s own.) T means temperature. The Combined group applies several samplers at once, whereas the Single group changes one at a time. All rows exclude the 30-token human-written prompt so that only generated text is scored.

As can be seen, the paper’s findings hold across these common decoding settings. Every setting other than raw pushes the liketropy score of the model’s own text far from zero, while human text stays near zero. As a result, the model’s own texts are often rejected while the human texts are almost always accepted. Relatedly, the AUROCs between the model’s own text and human text are well below 0.5, except for under raw decoding.

In the camera-ready version, we will add these robustness checks as a new appendix.

Raw sampling in deployment. Raw sampling is not an edge case. OpenAI's API defaults to temperature 1 and top-p 1, and Anthropic's to temperature 1 with top-k and top-p optional, while open-source toolkits truncate by default. Deployed text therefore mixes both regimes. We will also add a clarification in Section 4.1, which mentions only the open-source defaults.

On novelty and positioning. We agree that “decoding settings change the output distribution” is previously known and, on its own, unsurprising. We believe the paper is better read as exposing an intrinsic limitation of likelihood-based zero-shot detection. Such detectors are so sensitive to what are usually treated as minor implementation details that detection itself can fail. One example is the difference between a language model and a mildly truncated version of it, which are routinely regarded as the same model. The paper therefore goes beyond reporting that decoding changes the distribution. It (i) shows that the effect is large enough to break the detector altogether, and (ii) sheds light on why this happens.

In the camera-ready version, we will further clarify the positioning of the paper.

We hope these results and clarifications address your concerns. If so, we would be grateful if you would take them into account in your assessment.

AuthorsOpenReview ↗

Gentle Follow-up

Dear Reviewer SW5J,

Thank you again for your time and constructive review. We wanted to gently follow up to let you know that we have uploaded a revised version of the paper incorporating our responses, with all changes highlighted in blue.

In response to your comments, the revision adds six common decoding settings and their mixture with end-to-end metrics (Appendix N), clarifies the deployment defaults (Section 4.1) and our positioning.

We would be grateful to hear whether our response and the revision address your concerns, and we are happy to clarify anything further before the discussion period closes.

Best regards, The authors

The low AUROCs under common decoding settings are accompanied by high reversed AUROCs (0.77–0.99, and 0.85 for the mixture), suggesting that the discriminative signal may still be present but the original decision rule is inverted. This raises the possibility of mitigating the failures through a reversed or adaptive decision rule, which I think deserves investigation.

More broadly, I am not fully convinced that the current contribution is sufficiently strong. The paper identifies an interesting failure mode but stops short of addressing it. Deriving and validating a mitigation method based on these observations would make the contribution substantially stronger.

AuthorsOpenReview ↗

A Follow-up reply

Thank you for this point. The low AUROC does mean that reversing the decision rule contains signals. However, that is only about the configuration term we define in Proposition 1, not the overall model identity. To confirm this point, we replaced the human text with Model A's own raw sampled text (top_k = 0, top_p = 1, temperature 1). With both samples from the same model, the reversed AUROC is 0.84 to above 0.99 (original: below 0.01 to 0.16). The AUROC numbers are similar to those against the human text, regardless of reversed rule or not; see more details below.

Decoding settingvs human text (original / reversed)vs Model A's raw text (original / reversed)
raw0.56 / 0.44reference
top-k 40, top-p 0.96 (this paper)0.04 / 0.960.02 / 0.98
llama.cpp default0.16 / 0.840.14 / 0.86
T 0.7, top-p 0.950.01 / 0.99<0.01 / >0.99
top-k 50 (Hugging Face default)0.12 / 0.880.08 / 0.92
top-p 0.900.06 / 0.940.03 / 0.97
top-p 0.950.23 / 0.770.16 / 0.84
T 0.7, no truncation0.01 / 0.99<0.01 / >0.99
Mixture of all eight settings0.15 / 0.850.09 / 0.91

Model A's own text under each setting against the text in each column, all scored by Model A, averaged over six evaluators. Reversed AUROC is 1 − original. For the mixture in the last column, the raw texts are split in half so that no text appears on both sides.

As can be seen, it is relatively easy to differentiate the narrowly distributed text (due to decoding configuration) from the raw text, or from the human text. In comparison, to differentiate the raw text from human text is much harder, with AUROC being 0.56 in the original rule and 0.44 in the reversed rule, both closer to random guesses. An adaptive rule meets the same challenge, since the fundamental limit lies in the score (as reported in the paper), not the decision rule or threshold. Since the only identity term in liketropy is near zero for human text, we see this as a structural limit of rules built on liketropy. That limit is the main contribution of the paper.

We will add this clarification and the discussion to the revision.

Reviewer_77GNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how decoding choices affect likelihood-based zero-shot text detection. It reports that changing the sampling configuration can shift detection scores more than changing the generator, and interprets this through a decomposition involving entropy differences and KL divergence.

Critical strengths
  1. The paper addresses a meaningful robustness issue: strong results under one decoding configuration may not generalize to others.
  2. The controlled experiments cover multiple models, domains, and truncation levels, providing useful evidence of decoding sensitivity.
Critical weaknesses
  1. The mechanism interpretation is insufficiently justified. The paper labels the entropy difference as a “configuration term” and the KL divergence as an “identity term,” but both depend on model identity and decoding choices. The decomposition therefore does not disentangle these factors. In particular, entropy changes caused by instruction tuning are themselves model differences; detecting them does not necessarily mean succeeding for the “wrong reason.”

  2. The conclusions appear too strong. Decoding is part of the generation process, and differences introduced by truncation can still provide valid detection signals. Moreover, showing that truncation strongly affects detection scores does not rule out other human–AI differences. The results therefore demonstrate sensitivity to decoding choices, rather than a fundamental limitation of likelihood-based detection.

Questions for authors
  1. What evidence supports interpreting the two terms specifically as configuration and identity, beyond the algebraic decomposition?
  2. Can the authors distinguish failure to generalize across decoding configurations from the broader claim that these methods do not capture meaningful human–AI differences?
2 comments

Thank you for the careful reading. We will respond to your comments below.

Why "configuration" and "identity"? We agree that, in principle, both terms are affected by both model identity and decoding choices. However, they respond in drastically different ways. The configuration term responds to decoding configurations (i.e., truncation and temperature) much more strongly than model identity. The (unobserved) identity term, i.e., KL divergence, on the other hand, is more sensitive to such a model change, as it should be. That is where the names of “configuration” and “identity” terms come from. Under decoding changes, the configuration term moves far more than the identity term. Hence, part of the empirical insight of this paper is that the zero-shot detector breaks down because of the hypersensitivity of the configuration term to decoding setting, which overwhelms the response of the identity term to model identity change.

To make this point clearer, we summarize below how the configuration and identity terms respond to the decoding configuration and model identity respectively.

Change from the evaluator's own raw samplingΔ configurationΔ identityRatio
Decoding: top-k 40, same model (Fig. 2)+0.5450.1005.45
Decoding: top-k 640, same model (Fig. 2)+0.1590.02157.39
Model: Llama-3 → 3.1 (Table 4)−0.0140.0550.25
Model: Mistral v0.1 → v0.3 (Table 4)+0.0060.0360.17
Model: four instruction-tuned relatives (Table 4)+0.23 to +0.920.08 to 0.361.6 to 2.8

Ratio: |Δ configuration| / |Δ identity|; terms in nats.

Truncation moves the configuration term 3.7 to 7.4 times as much as the identity term, even when it narrows the distribution only mildly. (It moves the identity term too, but far less.) This effect ultimately leads to the breakdown of the detector. This observation stands in contrast to Radvand et al., who treat the configuration term merely as an error term.

In the camera-ready version, we will add this table to the paper and further clarify the roles of those terms.

Instruction tuning. Thank you. We agree that post-training is a genuine model difference, and will remove "for the wrong reason" for further clarity. The more precise point is that the score registers this difference through the same channel as truncation, since both lead to the narrowing of distribution and the breakdown of the detector (see Appendices E and I).

Decoding sensitivity and human–AI distinction. Decoding settings, such as truncation, have usually been treated as minor implementation details of LLMs, and many places use truncations by default, such as top-k 50 in Hugging Face and top-k 40 and top-p 0.95 in llama.cpp. On top of that, users can also change the settings freely. A language model and a mildly truncated version of it are therefore routinely regarded as the same model. This is also consistent with the literature. For example, Radvand et al. build their theories on raw sampling, whereas their tests use truncated text (configured as top-k 40 and top-p 0.96). Our study of zero-shot detectors follow the same convention. In this context, our surprising finding is that these detectors are so sensitive to such configuration details that detection itself can fail.

Note that a zero-shot likelihood statistic observes only the combination of the configuration and identity terms. Hence the sensitivity to the configuration term inevitably influences the detector's ability to differentiate human-generated texts from AI-generated ones. This is why we call the limitation intrinsic.

Additional robustness experiment. To check that our finding is not restricted to the paper’s setting, we tested six further decoding settings that are common, with all six evaluators on all eight datasets. Every setting pushes the model’s own text away from zero. Recalibrating the threshold to each decoding setting does not help: it accepts 97% to 100% of human text. In short, each of these common, freely editable settings is enough to mislead liketropy: the score moves with the decoding even when the writer is fixed. We will add these results as a new appendix.

We hope this addresses your questions.

AuthorsOpenReview ↗

Gentle Follow-up

Dear Reviewer 77GN,

Thank you again for your time and constructive review. We wanted to gently follow up to let you know that we have uploaded a revised version of the paper incorporating our responses, with all changes highlighted in blue.

In response to your comments, the revision adds Table 5, which shows how each term responds to decoding and to model changes, and states more precisely what we do and do not claim.

We would be grateful to hear whether our response and the revision address your concerns, and we are happy to clarify anything further before the discussion period closes.

Best regards, The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal Comments

We thank all reviewers for their careful reading and constructive feedback. We uploaded a revision that addresses the following main questions:

  1. Does the failure persist for larger models? (MSzG) Yes. We repeated the protocol with Gemma-2-27B and Qwen2.5-32B in the new Appendix O. They reject 69.3% and 57.4% of their own truncated text yet accept 87.0% and 87.5% of human text, within the range of the original six evaluators. The configuration term also remains dominant, at about five times the identity (KL divergence) term.

  2. Does the failure also hold under commonly used decoding settings and their mixture? (SW5J, 77GN) Yes. We added six common settings, including the Hugging Face and llama.cpp defaults, and a mixture of all settings (new Appendix N). Every setting other than raw sampling pushes the model's own text far from zero. The AUROC against human text falls to 0.01–0.23, and recalibrating the threshold to each setting accepts 97–100% of human text.

  3. What is the main contribution relative to prior work? (all reviewers) We clarified the positioning in Section 1. While it has been reported that decoding changes the output distribution and can affect detection, our contribution is to show (i) that the effect is large enough to break likelihood-based detectors altogether, even between a model and a mildly truncated version of it, and (ii) the underlying reason, through the analysis of a decomposition of a broad log-likelihood family into a configuration term and an identity term. We find that the failure can be explained by the overwhelming sensitivity of the former term to routine decoding implementation.

For ease of reading, the changes are now highlighted in blue. We will change those to black in the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.