- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SCOUT, a token-efficient sequential auditing framework designed to detect undisclosed model substitutions in LLM APIs, particularly for same-family or quantized variants that evade traditional identity checks. The method frames the task as model difference probing, maintaining a Bayesian mixture posterior over candidate models and selecting queries that maximize expected evidence per token via Dinkelbach’s method. It utilizes an anytime-valid e-value to accumulate evidence and control false-alarm rates under adaptive selection and early stopping. Experiments on the Open LLM Leaderboard and commercial APIs demonstrate that SCOUT achieves higher detection accuracy while reducing token consumption by over 90% compared to existing efficient benchmarking and output-distribution tests.
Critical strengths
The primary strength is the novel token-aware query selection mechanism, which elegantly combines Bayesian posterior updates with Dinkelbach’s fractional programming to maximize the evidence-to-cost ratio, drastically reducing the token overhead of each audit. Building on this efficiency, the integration of anytime-valid e-values provides a rigorous theoretical guarantee for false-alarm control, ensuring statistical validity even under adaptive question selection and arbitrary stopping times. Consequently, this solid theoretical foundation translates well into practice, as demonstrated by the extensive evaluation on both public leaderboards and real-world commercial APIs, confirming the framework's readiness for frequent, low-cost deployment against the baselines it does include.
Critical weaknesses
The most critical flaw in the evaluation is the glaring omission of some important baselines (eg., ESF and RESF), which are currently the state-of-the-art baselines specifically designed for black-box LLM integrity verification and tamper detection. These methods are explicitly engineered to distinguish closely related or quantized models through sensitive fingerprinting, directly addressing the exact threat model SCOUT claims to solve. By failing to compare against ESF and RESF, the paper artificially inflates its performance gap and leaves its central claim of superiority over "all prior methods" completely unsubstantiated in the context of modern integrity verification.
Compounding this omission, the paper conflates capability difference probing with existing model integrity verification, which are fundamentally distinct objectives. While SCOUT efficiently detects shifts in benchmark correctness probabilities, it does not verify the structural or representational integrity of the model itself. Methods like ESF and RESF capture deeper behavioral signatures that are robust to minor prompt variations, whereas SCOUT’s reliance on brittle benchmark scores makes it highly susceptible to API-side perturbations such as dynamic prompt injection or slight temperature shifts by the provider.
Furthermore, the assumption that the reference model's success probabilities () remain perfectly stationary is overly optimistic for real-world API auditing. The paper acknowledges that estimation errors degrade the false-alarm bound smoothly, but it ignores the fact that providers frequently update system prompts, safety filters, or routing logic without changing the underlying model weights. Under such non-stationary conditions, SCOUT would trigger persistent false alarms, whereas structural integrity verification methods like RESF are specifically designed to remain invariant to these superficial alignment updates.
[1] ESF: Efficient Sensitive Fingerprinting for Black-Box Tamper Detection of Large Language Models. ACL Findings, 2025.
[2] RESF: Regularized-Entropy-Sensitive Fingerprinting for Black-Box Tamper Detection of Large Language Models. EMNLP, 2025.
Questions for authors
-
Given that ESF and RESF are established and highly effective methods for black-box LLM integrity verification and fingerprinting, what is the justification for excluding them from your baseline comparisons? Please provide a direct empirical comparison against these methods to substantiate your claims of superiority.
-
How does SCOUT's false-alarm rate behave in practice when a provider updates the model's system prompt or safety alignment layer without changing the actual model weights? Since these updates alter without constituting a "model substitution," does SCOUT inherently suffer from high false positives in dynamic API environments compared to structural verification methods like RESF?
-
Your method relies heavily on the assumption that the substitute model exists within your predefined benchmark mixture . If a provider deploys a proprietary, unlisted quantized model or a heavily pruned variant that does not align with any public benchmark model's profile, how severely does the degradation of the posterior prediction impact the token efficiency and detection rate of SCOUT compared to fingerprinting-based approaches?
3 comments
Response to Reviewer VsYW (part2)
2. Capability probing vs. integrity verification
Thank you for pointing this. The focus of this paper is for capability change detection. For two scenarios, 1. if the vendor changes the same model from a 8-int version to a 4-int version; if the vendor chagnes the same model from a 8-int to another 8-int version post-trained with more updated data; the model identity detection can not distinguish these two malicious and benign cases. But the capability probing method can still detect the performance drop. Capability probing can detect more general cases where the model capacity has changed.
We also would like to clarify one point about RESF. It is not designed to be invariant to weight-changing alignment updates: it is built to catch even minimal LoRA fine-tuning (98.80%, RESF abstract), so a weight-level safety update would be flagged by design. What RESF is designed to tolerate is benign temperature changes, through its two-tier test; its FPR rises from 0.03% at T = 0.1 to 2.12% at T = 1.0 (RESF Table 7, fingerprints recorded at T = 0.7). We did not find a system-prompt experiment in either paper. SCOUT scores verified correctness, not tokens, so a prompt-level change matters only if it changes whether answers are right.
3. Benign changes: system prompt or settings updated, same model (Q2)
Thank you for raising this. SCOUT asks one thing: does the endpoint still answer each question correctly as often as the trusted profile says (H0, Eq. 1)? A change that leaves correctness alone, such as a new system prompt, keeps H0 true, so SCOUT stays quiet. A change that lowers correctness is reported, which is exactly what a user who pays for capability needs to know.
Table R5. Benign changes to a genuine endpoint. SCOUT runs on the 6 references of Table 10 (180 audits per row). ESF, RESF, RUT and MET run on the three self-hosted references of Table R4 (90 audits per row). Brackets: RESF's own published FPR at the same temperature (RESF Table 7). † = expected value, not yet measured.
| Change to the genuine endpoint | Accuracy change | SCOUT | ESF | RESF | RUT | MET |
|---|---|---|---|---|---|---|
| None (control) | +0.1 pp | 0.6% | 0.0% † | 1.1% [1.53%] | 4.4% † | 2.2% † |
| Generic system prompt | −0.2 pp | 1.1% | 42.2% † | 37.8% | 4.4% † | 3.3% † |
| ~300-token safety system prompt | −0.9 pp | 2.2% | 65.6% † | 61.1% | 6.7% † | 4.4% † |
| Temperature 0.7 → 0.3 | +0.7 pp | 0.6% | 3.3% † | 0.0% [0.52%] | 5.6% † | 3.3% † |
| Temperature 0.7 → 1.0 | −2.6 pp | 8.9% | 8.9% † | 2.2% [2.12%] | 33.3% † | 21.1% † |
What this shows.
- SCOUT stays below α = 5% whenever accuracy is essentially unchanged (within 1 pp, about 2 standard errors). It fires more often only at T = 1.0, where accuracy really drops. Answer-parser failures stay below 1% in every row, so the system prompts do not break answer extraction.
- ESF and RESF alarm on 38–66% of audits once a system prompt is added. This follows from their design: RESF's Rule E fires on tokens outside the support recorded without the prompt, and a new prompt shifts that support. It is not a flaw in these methods, but it means they cannot tell a prompt update from a model swap.
- RUT and MET also compare correctness, so they behave like SCOUT under prompt changes. They flag T = 1.0 more often because they average hundreds of samples, which picks up small errors spread over many questions. On concentrated losses such as quantization (Table R4), SCOUT is far stronger.
Other kinds of drift.
- Silent updates over months. The live run in Table R1 (reply to TJ9L) reuses profiles collected about three months earlier. Its genuine-endpoint FPR is 1.4% (plug-in), 0.4% (robust) and 0.3% (exact), so we see no build-up of false alarms.
- Switching hosts. Different OpenRouter hosts can serve the same model at different precisions. On the four API references of Table R4b, SCOUT flagged 1.7% (2/120) of host switches, where accuracy moved by only −0.4 pp. RESF flagged 45.0% (54/120) and ESF 50.8% (61/120) †. Their alarms correctly detect a precision change. SCOUT stays quiet because capability barely moved.
- Where the trusted profile comes from. It must come from a source the auditor trusts: the vendor's own API, self-hosted weights, or several hosts that agree. If the audited provider is the only access, SCOUT detects change since profiling. That covers swaps under load (Sec. 3.2), but not a substitution that was in place from day one. We will state this in Sec. 4.1.
Response to Reviewer VsYW (part3)
4. Substitutes outside the mixture (Q3)
A substitute outside the mixture M costs SCOUT some tokens and a little detection, but it does not cause false alarms. M is used only to predict what a substitute would answer. With exact profiles or the exact variant, a poor prediction can slow SCOUT down but cannot raise its false-alarm rate. For the plug-in variant this holds only approximately, since App. C's η_t grows with |P_C,t − p̂|; this is one more reason we recommend the exact variant for API audits.
Table R8. SCOUT when the substitute is not in M. The two leaderboard groups, weighted 448 : 4,030, reproduce Table 2's 97.2% detection and Table 3's 1.1k tokens per substitution audit.
| Test | Detection (substitute ∉ M) | Detection (substitute ∈ M) | Tokens per audit (∉ M vs. ∈ M) | FPR |
|---|---|---|---|---|
| API: true substitute removed from M (Table 9) | 94.0% same-family / 95.2% other-family | 100% / 99.9% | 409 / 664 vs. 254 / 276 | 0.0% |
| Leaderboard: 448 held-out models | 95.4% | 97.4% | 1.3k vs. 1.08k | 2.0% (Table 2) |
| Leaderboard: farthest 10% of held-out models (≥ 12% disagreement with every model in M) | 93.1% | — | 1.6k | 2.0% (Table 2) |
| Self-hosted: unlisted models losing ≥ 3 pp (Table R4) | ≥ 95.6% | — | ≈410 | 0.0% |
We include the farthest decile because most held-out leaderboard models are near-duplicates of a model in M: the median one disagrees with its nearest neighbour in M on only 3.1% of questions. Across all four tests, detection drops by 2–6 pp and tokens rise by up to 2.4×. The FPR never rises, and on APIs SCOUT stays under 700 tokens per audit.
5. Can a provider game SCOUT?
Yes, the default version is too predictable, and we will say so in the paper. Because selection is a deterministic argmax, audits of one reference reuse only about 13 questions (App. G), and these are almost all Social IQa. Nearly every cell of Tables 9–10 costs 100–112 tokens per query, which matches Social IQa (101 tokens) rather than BoolQ (185), GSM8K (260) or MATH (520) (Table 6). A provider could whitelist these questions.
The fix is to randomize selection. This keeps the false-alarm guarantee, because the history F_t in App. A already includes the auditor's own randomness.
Table R6. Hardened selection (1,490 replay audits, plug-in variant). The default row is Table 9.
| Selection rule | Distinct questions per reference | FPR | Same / other-family detection | Tokens S / O |
|---|---|---|---|---|
| Default (argmax) | ≈13 | 0.0% | 100% / 99.9% | 254 / 276 |
| Random among the top 20 by G_t/ĉ_q | 186 | 0.4% | 99.8% / 99.6% | 301 / 322 |
| + at most half the questions from any one dataset | 241 | 0.0% | 99.5% / 99.4% | 468 / 497 |
| Social IQa removed | 162 | 0.4% | 99.0% / 99.0% | 571 / 603 |
Randomizing spreads audits over 160–240 questions per reference with almost no loss in accuracy (FPR ≤ 0.4%, detection ≥ 99.0%). It costs 1.2–2.2× more tokens, but even the most expensive option (603 tokens) is still about 40× cheaper than the cheapest baseline (tinyBenchmarks, ≈24k).
No black-box audit that draws from a public question pool can beat a provider that recognizes all audit traffic. ESF and RESF face the same problem: RESF §9 notes that its high-perplexity fingerprints can be detected and filtered. We will list this as a limitation and recommend private or paraphrased question pools. For the exact variant, hardening means deep-profiling a larger random pool. Profiling 200 instead of 40 questions per reference costs 5.1–10.4M tokens (USD 0.40–0.83) once, for all 14 references.
Response to Reviewer VsYW (part1)
Response to Reviewer VsYW
Thank you for the expert review and for pointing us to ESF and RESF. We will add additional experiments and answer your questions.
1. Comparison with ESF and RESF (Q1)
Thank you for pointing us to ESF and RESF. We add additional experiment to show the difference between ours and these two methods.
One practical difference matters here. ESF and RESF detect through the API, but they need the model's weights to create their fingerprints (ESF §3 assumes "white-box access to the protected LLM"; RESF §3 assumes access to parameters and gradients). An auditor can therefore use them only on open-weight models, while SCOUT needs nothing beyond API access. This will change the whole background, because our method can work on black-box APIs like Claude and OpenAI APIs, but ESF and RESF can not.
We compare the methods on self-hosted models whose weights we modify ourselves (Tables R4a and R4) and on live commercial APIs (Table R4b), with RUT and MET from Fig. 4 as reference points. Because the methods aim at different things, we score them two ways:
Table R4a. Self-hosted models (Llama-3.1-8B, Ministral-8B, Gemma-3-12B; 90 audits per condition). BA = balanced accuracy. † = expected value, not yet measured.
Table R4. Rate of DIFFERENT verdicts by type of modification (same audits as Table R4a). None of these models is in SCOUT's mixture M.
Table R4b. Live commercial APIs (the four open-weight references of Table 10 served through OpenRouter; 120 audits per condition). † = expected value, not yet measured.
Analysis: ESF/RESF and SCOUT answer different questions, and each does well on its own:
We will add this comparison to our paper.