- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aims to detect when an API provider silently changes an open-weight LLM’s configuration, even when the provider hides its sampling parameters. The key observation is that the probability ratio between two chosen first tokens is unaffected by truncation-style samplers and repetition penalties. It depends only on temperature and the bias difference between the two tokens. Querying two prompts with the same token pair therefore gives two estimates of temperature, or of the bias difference, that should agree when the configuration is untampered. The test flags tampering when a 99% bootstrap confidence interval of their gap excludes zero, checked every 6,000 queries up to 30,000. On Qwen2.5-14B-Instruct with 28 tampered variants covering substitution, fine-tuning, system-prompt injection and quantization, the Temperature Test reaches 100% TPR and the Logit Bias Delta Test 92.9% TPR, both at 0% FPR, at an estimated $0.1–2.6 per test.
Critical strengths
-
The invariance result is simple and useful: it reduces about 15 hidden samplers to two unknowns that two prompts can cancel. It needs only output strings and max_tokens=1.
-
The tests are cheap: positive tests stopped after 6,778 queries on average.
Critical weaknesses
-
FPR guarantee assumes exact logit agreement: The test treats any departure from the locally computed logits z_j(t) as tampering. Yet the FPR trials (App. B.1) use the same local stack as both “expected” and “served,” so 0% FPR holds by construction, and no real provider is tested. In Fig. 9, the 6-bit and 5-bit variants end at 0.018 and 0.036, smaller in magnitude than the untampered model’s own −0.056, yet both are flagged. The test therefore reacts to perturbations at the scale of one quantization step. Different kernels, GPUs, batching nondeterminism (which DiFR, cited by the paper, studies), or routing across replicas could cause shifts of that size. A mixture of replicas would also break the single-softmax form behind Cor. 5. Thms 1–3 are standard consistency and delta-method results, and the actual decision rule (bootstrap, five interim checks) has no finite-sample FPR control.
-
Narrow evaluation and no baselines: Only temperature, top-p and logit bias are hidden, with one random draw per regime: T in 0.98–1.10, top-p in 0.88–0.95. In three of the four regimes the bias is equal on “0” and “1”, so Δb = 0 and logit bias, the stated core difficulty, never changes the measured ratio. There is no head-to-head with IRIS, RESF, Model Equality Testing or the rank-based uniformity test, although the motivation rests on IRIS’s 30% FPR. Configurations that never produced both tokens were excluded (A.2.2), but how many were excluded is not reported. Only one base model is tested.
Questions for authors
-
What is the FPR when the expected logits come from a stack with the same nominal configuration but different software or hardware (vLLM vs. Transformers, different GPU, batched serving, a different 8-bit scheme)?
-
How many candidates were disqualified in A.2.2? How do both tests behave at T ≤ 0.5 and with asymmetric biases on the probe tokens?
-
How do IRIS and Model Equality Testing perform under the same hidden parameters and 30,000-query budget?