acceptodds
Withdrawn from ICLR 2027

Provable LLM Tampering Detection: Catching Model Provider Tampering of Open-Weights LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prior theory-driven methods for detecting black-box LLM configuration tampering do not preserve their guarantees of a low false-positive rate (FPR) when the values of multiple common sampling parameters—such as temperature, token logit bias, and top-p—are hidden. Such sampling parameters can unpredictably and nonlinearly distort LLM output distributions without changing the underlying model configuration, making benign variation in sampling parameters difficult to statistically distinguish from genuine model configuration mismatches such as those arising from model substitution, secret fine-tuning, prompt injection, and weight quantization. To our knowledge, our paper establishes the first tamper-detection method for black-box LLM configurations that formally guarantees a low long-run FPR in the presence of a broad set of hidden standard sampling parameters. To this end, we develop a general theoretical framework for tamper-detection in this setting, and then construct two concrete classes of tests—the Temperature Test and the Logit Bias Delta Test—each provably achieving a long-run FPR of 0%. In our follow-up demo spanning four major tampering classes—model substitution, secret fine-tuning, system prompt injection, and weight quantization—the Temperature Test achieves 100% true-positive rate (TPR) and 0% FPR, while the Logit Bias Delta Test achieves 92.9% TPR and 0% FPR. Estimating the cost of each test in the demo at the prevailing rates of currently popular open-weight model APIs places the upper bound between 2.6, with our adaptive early-stopping technique further reducing the average estimated cost of a positive test to 0.6.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM auditing, black-box model verification, model tampering detection, model fingerprinting, sampling parameters, theoretical guarantees, formal guarantees, statistical identifiability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wqjLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to detect when an API provider silently changes an open-weight LLM’s configuration, even when the provider hides its sampling parameters. The key observation is that the probability ratio between two chosen first tokens is unaffected by truncation-style samplers and repetition penalties. It depends only on temperature and the bias difference between the two tokens. Querying two prompts with the same token pair therefore gives two estimates of temperature, or of the bias difference, that should agree when the configuration is untampered. The test flags tampering when a 99% bootstrap confidence interval of their gap excludes zero, checked every 6,000 queries up to 30,000. On Qwen2.5-14B-Instruct with 28 tampered variants covering substitution, fine-tuning, system-prompt injection and quantization, the Temperature Test reaches 100% TPR and the Logit Bias Delta Test 92.9% TPR, both at 0% FPR, at an estimated $0.1–2.6 per test.

Critical strengths
  1. The invariance result is simple and useful: it reduces about 15 hidden samplers to two unknowns that two prompts can cancel. It needs only output strings and max_tokens=1.

  2. The tests are cheap: positive tests stopped after 6,778 queries on average.

Critical weaknesses
  1. FPR guarantee assumes exact logit agreement: The test treats any departure from the locally computed logits z_j(t) as tampering. Yet the FPR trials (App. B.1) use the same local stack as both “expected” and “served,” so 0% FPR holds by construction, and no real provider is tested. In Fig. 9, the 6-bit and 5-bit variants end at 0.018 and 0.036, smaller in magnitude than the untampered model’s own −0.056, yet both are flagged. The test therefore reacts to perturbations at the scale of one quantization step. Different kernels, GPUs, batching nondeterminism (which DiFR, cited by the paper, studies), or routing across replicas could cause shifts of that size. A mixture of replicas would also break the single-softmax form behind Cor. 5. Thms 1–3 are standard consistency and delta-method results, and the actual decision rule (bootstrap, five interim checks) has no finite-sample FPR control.

  2. Narrow evaluation and no baselines: Only temperature, top-p and logit bias are hidden, with one random draw per regime: T in 0.98–1.10, top-p in 0.88–0.95. In three of the four regimes the bias is equal on “0” and “1”, so Δb = 0 and logit bias, the stated core difficulty, never changes the measured ratio. There is no head-to-head with IRIS, RESF, Model Equality Testing or the rank-based uniformity test, although the motivation rests on IRIS’s 30% FPR. Configurations that never produced both tokens were excluded (A.2.2), but how many were excluded is not reported. Only one base model is tested.

Questions for authors
  1. What is the FPR when the expected logits come from a stack with the same nominal configuration but different software or hardware (vLLM vs. Transformers, different GPU, batched serving, a different 8-bit scheme)?

  2. How many candidates were disqualified in A.2.2? How do both tests behave at T ≤ 0.5 and with asymmetric biases on the probe tokens?

  3. How do IRIS and Model Equality Testing perform under the same hidden parameters and 30,000-query budget?

Reviewer_d1YyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies model-configuration tampering detection in black-box LLM APIs using returned text and the expected model’s open weights while accounting for hidden sampling parameters. It exploits probability-ratio invariance in ρ\rho-regular first-token experiments to construct the Temperature Test and Logit Bias Delta Test, which assess cross-experiment consistency of parameter estimates. Under stated assumptions, the paper establishes estimator consistency, asymptotic normality, and asymptotically vanishing false-positive rates for a specified threshold rule.

Critical strengths

A useful and economical construction for handling multiple hidden sampling parameters. This method reduces the relevant nuisance parameters to quantities that can be eliminated or compared across contexts. The probability-ratio invariance for surviving tokens allows the same framework to accommodate several truncation mechanisms beyond temperature changes. Also it turns this observation into concrete tests using accessible reference weights and returned text. This is the paper's strongest technical contribution. This method does not require additional training of detectors, fine-tuning of models, or access to privileged information of providers, so it has a relatively low threshold for implementation under the condition of open weights and black-box query capabilities.

Critical weaknesses
  1. Theorem 2 proves asymptotically vanishing FPR for a specific threshold rule, whereas the experiments repeatedly inspect 99% bootstrap intervals with early stopping. The overall false-positive probability across these checks is not calibrated, and the theoretical guarantee is not established for this procedure. Observing no false positives in limited trials does not resolve this gap, leaving the reliability of the practical decision rule insufficiently supported.

  2. The evaluation uses one reference checkpoint, three fixed prompts, the token pair "0"/"1", and four sampling configurations. Three configurations have zero bias difference between the selected tokens, while broader sampling combinations remain empirically unexplored. Configurations unable to generate both tokens are excluded, leaving the method’s operational coverage unclear. Furthermore, up to 30,000 queries per test, averaging 6,778 for positive tests, raises deployment questions concerning request quotas and latency that token-cost estimates do not address. Although adversarial spoofing is discussed, no adaptive evaluation establishes resistance to an informed provider recognizing the finite probe set and selectively returning compliant behavior.

Questions for authors
  1. All the tampered configurations in the experiment are from the same model family of Qwen2.5, sharing the tokenizer and base distribution. What will happen to the detection performance of the method if the service provider is replaced with a completely different model family and the output text form is matched by post-processing? Is there a validation plan across model families?

  2. Please specify how many candidate configurations were considered and excluded in Appendix A.2.2. Does “could not be generated” mean a verified zero probability, or merely that a token was not observed within a finite sampling budget? If the latter, please state the budget and exclusion criterion. In addition, beyond temperature, top-p, and logit bias, were other standard sampling mechanisms or their combinations evaluated?

  3. How do detection performance and query requirements vary across diverse prompts and token pairs beyond the binary-output setup? Have you evaluated adversarial scenarios in which a provider knows the detection method and uses targeted fine-tuning or other adaptive modifications to evade the first-token consistency tests while preserving task utility?

  4. Did you compare with at least one close prior method under shared access conditions and matched query budgets, and explicitly distinguish applicability and the FPR–TPR tradeoff relative to RESF and IRIS, which both address temperature changes in black-box auditing? If so, how do the results compare with those of your method?

Reviewer_H549OpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to detect whether an LLM API provider is serving the expected open-weight model when its sampling parameters are unknown. The authors propose a general detection framework and develop two tests, the Temperature Test and the Logit Bias Delta Test. Both use repeated API queries to estimate hidden sampling parameters from first-token output frequencies and check whether the estimates agree across different prompts. The paper proves that the false-positive rate of these tests approaches zero as the number of queries increases. Experiments on Qwen2.5-14B-Instruct cover 28 modified configurations across model substitution, fine-tuning, system prompt injection, and weight quantization.

Critical strengths

Strength 1: The paper addresses a practical problem of verifying whether an LLM API provider serves the expected model configuration.

Strength 2: The proposed tests have clear mathematical derivations and provide theoretical guarantees on the asymptotic false-positive rate.

Strength 3: The experiments cover four types of model tampering and show strong detection results with relatively low query costs.

Critical weaknesses

Weakness 1: In Section 6.1 (Theorem 8) and Appendix A.2.1, the paper presents the Temperature Test and the Logit Bias Delta Test as two separate methods. However, in the three-experiment setup, their statistics satisfy ϕ^Δb=γ2ϕ^T\hat{\phi}_{\Delta b}=\gamma_2\hat{\phi}_T. When γ2≠0\gamma_2\neq0, both tests check the same condition. This also makes the difference in their reported TPRs difficult to interpret.

Question: Do these two tests provide different detection capabilities in the current experimental setup?

Weakness 2: The estimators in Sections 5.2–5.3 cannot detect certain model changes. For example, if the served model's logit differences satisfy Δz~j=aΔzj+c\Delta\tilde z_j=a\Delta z_j+c across the tested contexts, both tests can give consistent estimates even though the model has changed. The experiments in Section 7 do not examine whether the tested model substitutions, fine-tuning, or quantization changes produce similar relationships.

Question: Do any of the tested model modifications produce logit differences close to this relationship, and how does that affect the detection results?

Weakness 3: Appendix A.2.1 uses three prompts that all ask for random binary outputs. However, several modifications in Appendix A.1.3 target specific behaviors. For example, SP05 activates when a particular phrase appears, while SP07 affects product recommendations. Neither condition appears in the evaluation prompts. These modifications are detected only if they also change the first-token probabilities of the binary outputs. The paper does not test modifications that leave these probabilities largely unchanged.

Question: How well do the tests detect modifications that affect specific tasks or trigger conditions without noticeably changing the outputs of the three evaluation prompts?

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.