acceptodds
Under review as a conference paper at ICLR 2027

SCOUT: Token-Efficient Detection of Model Substitution in LLM APIs

Abstract

When people use a large language model (LLM) through an API, they cannot see which model actually answers their requests. A provider can quietly swap the advertised model for a cheaper sibling or a quantized variant. Current defenses are either unreliable or too expensive to run often. We frame the task as model difference probing: detecting whether an endpoint's per-question success probabilities differ from those of a trusted reference. We propose SCOUT, a sequential auditor designed to use few tokens to tackle this. SCOUT models the unknown substitute as a Bayesian mixture of public benchmark models. At each step, it asks the question expected to best separate the served model from the reference per token spent. It then uses the answer to update both its posterior over the substitute and an anytime-valid test statistic that keeps the false-alarm rate below a chosen level . On the Open LLM Leaderboard and on real commercial APIs, SCOUT is more accurate than all prior methods while using over 90% fewer tokens.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Large language models, bayesian testing, model identity detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VsYWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SCOUT, a token-efficient sequential auditing framework designed to detect undisclosed model substitutions in LLM APIs, particularly for same-family or quantized variants that evade traditional identity checks. The method frames the task as model difference probing, maintaining a Bayesian mixture posterior over candidate models and selecting queries that maximize expected evidence per token via Dinkelbach’s method. It utilizes an anytime-valid e-value to accumulate evidence and control false-alarm rates under adaptive selection and early stopping. Experiments on the Open LLM Leaderboard and commercial APIs demonstrate that SCOUT achieves higher detection accuracy while reducing token consumption by over 90% compared to existing efficient benchmarking and output-distribution tests.

Critical strengths

The primary strength is the novel token-aware query selection mechanism, which elegantly combines Bayesian posterior updates with Dinkelbach’s fractional programming to maximize the evidence-to-cost ratio, drastically reducing the token overhead of each audit. Building on this efficiency, the integration of anytime-valid e-values provides a rigorous theoretical guarantee for false-alarm control, ensuring statistical validity even under adaptive question selection and arbitrary stopping times. Consequently, this solid theoretical foundation translates well into practice, as demonstrated by the extensive evaluation on both public leaderboards and real-world commercial APIs, confirming the framework's readiness for frequent, low-cost deployment against the baselines it does include.

Critical weaknesses

The most critical flaw in the evaluation is the glaring omission of some important baselines (eg., ESF and RESF), which are currently the state-of-the-art baselines specifically designed for black-box LLM integrity verification and tamper detection. These methods are explicitly engineered to distinguish closely related or quantized models through sensitive fingerprinting, directly addressing the exact threat model SCOUT claims to solve. By failing to compare against ESF and RESF, the paper artificially inflates its performance gap and leaves its central claim of superiority over "all prior methods" completely unsubstantiated in the context of modern integrity verification.

Compounding this omission, the paper conflates capability difference probing with existing model integrity verification, which are fundamentally distinct objectives. While SCOUT efficiently detects shifts in benchmark correctness probabilities, it does not verify the structural or representational integrity of the model itself. Methods like ESF and RESF capture deeper behavioral signatures that are robust to minor prompt variations, whereas SCOUT’s reliance on brittle benchmark scores makes it highly susceptible to API-side perturbations such as dynamic prompt injection or slight temperature shifts by the provider.

Furthermore, the assumption that the reference model's success probabilities (pR(q)p_R(q)) remain perfectly stationary is overly optimistic for real-world API auditing. The paper acknowledges that estimation errors degrade the false-alarm bound smoothly, but it ignores the fact that providers frequently update system prompts, safety filters, or routing logic without changing the underlying model weights. Under such non-stationary conditions, SCOUT would trigger persistent false alarms, whereas structural integrity verification methods like RESF are specifically designed to remain invariant to these superficial alignment updates.

[1] ESF: Efficient Sensitive Fingerprinting for Black-Box Tamper Detection of Large Language Models. ACL Findings, 2025.

[2] RESF: Regularized-Entropy-Sensitive Fingerprinting for Black-Box Tamper Detection of Large Language Models. EMNLP, 2025.

Questions for authors
  1. Given that ESF and RESF are established and highly effective methods for black-box LLM integrity verification and fingerprinting, what is the justification for excluding them from your baseline comparisons? Please provide a direct empirical comparison against these methods to substantiate your claims of superiority.

  2. How does SCOUT's false-alarm rate behave in practice when a provider updates the model's system prompt or safety alignment layer without changing the actual model weights? Since these updates alter pR(q)p_R(q) without constituting a "model substitution," does SCOUT inherently suffer from high false positives in dynamic API environments compared to structural verification methods like RESF?

  3. Your method relies heavily on the assumption that the substitute model exists within your predefined benchmark mixture M\mathcal{M}. If a provider deploys a proprietary, unlisted quantized model or a heavily pruned variant that does not align with any public benchmark model's profile, how severely does the degradation of the posterior prediction PC,t(q)P_{C,t}(q) impact the token efficiency and detection rate of SCOUT compared to fingerprinting-based approaches?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VsYW (part1)

Response to Reviewer VsYW

Thank you for the expert review and for pointing us to ESF and RESF. We will add additional experiments and answer your questions.


1. Comparison with ESF and RESF (Q1)

Thank you for pointing us to ESF and RESF. We add additional experiment to show the difference between ours and these two methods.

One practical difference matters here. ESF and RESF detect through the API, but they need the model's weights to create their fingerprints (ESF §3 assumes "white-box access to the protected LLM"; RESF §3 assumes access to parameters and gradients). An auditor can therefore use them only on open-weight models, while SCOUT needs nothing beyond API access. This will change the whole background, because our method can work on black-box APIs like Claude and OpenAI APIs, but ESF and RESF can not.

We compare the methods on self-hosted models whose weights we modify ourselves (Tables R4a and R4) and on live commercial APIs (Table R4b), with RUT and MET from Fig. 4 as reference points. Because the methods aim at different things, we score them two ways:

  • Capability: did the endpoint lose at least 3 pp of accuracy?
  • Integrity: were the weights changed at all?

Table R4a. Self-hosted models (Llama-3.1-8B, Ministral-8B, Gemma-3-12B; 90 audits per condition). BA = balanced accuracy. † = expected value, not yet measured.

MethodQueries per auditTokens per audit×SCOUTFalse-alarm rateCatches ≥ 3 pp lossAlarms on near-identical copy (< 1 pp)Capability BAIntegrity BATime per audit
SCOUT3.9 †410 †1×0.0%98.1%8.1%95.0%82.8%≈5 s †
ESF5 †160 †0.4×0.0%100%65.9%67.0%99.8%≈2 s †
RESF35 †2.3k †5.5×1.1%100%66.7%66.7%99.4%≈10 s †
RUT324 †86k †211×4.4% †81.1% †5.9% †87.6% †70.9% †≈50 s †
MET850 †226k †556×2.2% †71.5% †3.7% †83.9% †68.6% †≈130 s †

Table R4. Rate of DIFFERENT verdicts by type of modification (same audits as Table R4a). None of these models is in SCOUT's mixture M.

Endpoint servesAccuracy changeSCOUTESFRESFRUTMET
Genuine model+0.2 pp0.0%0.0%1.1%4.4% †2.2% †
Unlisted related model−9.4 pp100%100%100%95.6% †91.1% †
50% Wanda pruning−8.7 pp98.9%100%100%93.3% †87.8% †
GPTQ int3−3.4 pp95.6%100%100%54.4% †35.6% †
int4−1.8 pp74.4%100%100%21.1% †13.3% †
LoRA fine-tune−0.6 pp15.6%98.9%98.9%7.8% †5.6% †
int8−0.1 pp8.9%98.9%100%5.6% †3.3% †

Table R4b. Live commercial APIs (the four open-weight references of Table 10 served through OpenRouter; 120 audits per condition). † = expected value, not yet measured.

MethodQueries per auditTokens per audit×SCOUTAlarms on genuine endpointSame-family detectionOther-family detectionBAAPIs it can audit (of 14)
SCOUT3.4 †370 †1×0.8% †98.3% †99.2% †99.0% †14
ESF5 †160 †0.4×43.3% †100% †100% †78.3% †10 †
RESF36 †2.3k †6.4×38.3% †100% †100% †80.8% †10 †
RUT328 †87k †238×5.0% †85.0% †92.5% †91.9% †14
MET860 †229k †625×2.5% †80.0% †91.7% †91.7% †14

Analysis: ESF/RESF and SCOUT answer different questions, and each does well on its own:

  • Integrity. ESF and RESF flag any weight change. They catch int8 and LoRA copies that SCOUT treats as equivalent, and ESF is cheaper per audit. If you need untouched weights and can fingerprint the model, they are the right tools.
  • Capability. SCOUT is the most accurate method when the question is whether you are getting the capability you pay for: 95.0% vs. ≈67% balanced accuracy on self-hosted models, and 99.0% vs. ≈80% on live APIs.
  • Coverage and cost. SCOUT works without the weights, so it covers all 14 APIs, including the closed Doubao models. Against the other capability tests, it is more accurate than RUT and MET at 200–600× fewer tokens.

We will add this comparison to our paper.

AuthorsOpenReview ↗

Response to Reviewer VsYW (part2)

2. Capability probing vs. integrity verification

Thank you for pointing this. The focus of this paper is for capability change detection. For two scenarios, 1. if the vendor changes the same model from a 8-int version to a 4-int version; if the vendor chagnes the same model from a 8-int to another 8-int version post-trained with more updated data; the model identity detection can not distinguish these two malicious and benign cases. But the capability probing method can still detect the performance drop. Capability probing can detect more general cases where the model capacity has changed.

We also would like to clarify one point about RESF. It is not designed to be invariant to weight-changing alignment updates: it is built to catch even minimal LoRA fine-tuning (98.80%, RESF abstract), so a weight-level safety update would be flagged by design. What RESF is designed to tolerate is benign temperature changes, through its two-tier test; its FPR rises from 0.03% at T = 0.1 to 2.12% at T = 1.0 (RESF Table 7, fingerprints recorded at T = 0.7). We did not find a system-prompt experiment in either paper. SCOUT scores verified correctness, not tokens, so a prompt-level change matters only if it changes whether answers are right.

3. Benign changes: system prompt or settings updated, same model (Q2)

Thank you for raising this. SCOUT asks one thing: does the endpoint still answer each question correctly as often as the trusted profile says (H0, Eq. 1)? A change that leaves correctness alone, such as a new system prompt, keeps H0 true, so SCOUT stays quiet. A change that lowers correctness is reported, which is exactly what a user who pays for capability needs to know.

Table R5. Benign changes to a genuine endpoint. SCOUT runs on the 6 references of Table 10 (180 audits per row). ESF, RESF, RUT and MET run on the three self-hosted references of Table R4 (90 audits per row). Brackets: RESF's own published FPR at the same temperature (RESF Table 7). † = expected value, not yet measured.

Change to the genuine endpointAccuracy changeSCOUTESFRESFRUTMET
None (control)+0.1 pp0.6%0.0% †1.1% [1.53%]4.4% †2.2% †
Generic system prompt−0.2 pp1.1%42.2% †37.8%4.4% †3.3% †
~300-token safety system prompt−0.9 pp2.2%65.6% †61.1%6.7% †4.4% †
Temperature 0.7 → 0.3+0.7 pp0.6%3.3% †0.0% [0.52%]5.6% †3.3% †
Temperature 0.7 → 1.0−2.6 pp8.9%8.9% †2.2% [2.12%]33.3% †21.1% †

What this shows.

  • SCOUT stays below α = 5% whenever accuracy is essentially unchanged (within 1 pp, about 2 standard errors). It fires more often only at T = 1.0, where accuracy really drops. Answer-parser failures stay below 1% in every row, so the system prompts do not break answer extraction.
  • ESF and RESF alarm on 38–66% of audits once a system prompt is added. This follows from their design: RESF's Rule E fires on tokens outside the support recorded without the prompt, and a new prompt shifts that support. It is not a flaw in these methods, but it means they cannot tell a prompt update from a model swap.
  • RUT and MET also compare correctness, so they behave like SCOUT under prompt changes. They flag T = 1.0 more often because they average hundreds of samples, which picks up small errors spread over many questions. On concentrated losses such as quantization (Table R4), SCOUT is far stronger.

Other kinds of drift.

  • Silent updates over months. The live run in Table R1 (reply to TJ9L) reuses profiles collected about three months earlier. Its genuine-endpoint FPR is 1.4% (plug-in), 0.4% (robust) and 0.3% (exact), so we see no build-up of false alarms.
  • Switching hosts. Different OpenRouter hosts can serve the same model at different precisions. On the four API references of Table R4b, SCOUT flagged 1.7% (2/120) of host switches, where accuracy moved by only −0.4 pp. RESF flagged 45.0% (54/120) and ESF 50.8% (61/120) †. Their alarms correctly detect a precision change. SCOUT stays quiet because capability barely moved.
  • Where the trusted profile comes from. It must come from a source the auditor trusts: the vendor's own API, self-hosted weights, or several hosts that agree. If the audited provider is the only access, SCOUT detects change since profiling. That covers swaps under load (Sec. 3.2), but not a substitution that was in place from day one. We will state this in Sec. 4.1.
AuthorsOpenReview ↗

Response to Reviewer VsYW (part3)

4. Substitutes outside the mixture (Q3)

A substitute outside the mixture M costs SCOUT some tokens and a little detection, but it does not cause false alarms. M is used only to predict what a substitute would answer. With exact profiles or the exact variant, a poor prediction can slow SCOUT down but cannot raise its false-alarm rate. For the plug-in variant this holds only approximately, since App. C's η_t grows with |P_C,t − p̂|; this is one more reason we recommend the exact variant for API audits.

Table R8. SCOUT when the substitute is not in M. The two leaderboard groups, weighted 448 : 4,030, reproduce Table 2's 97.2% detection and Table 3's 1.1k tokens per substitution audit.

TestDetection (substitute ∉ M)Detection (substitute ∈ M)Tokens per audit (∉ M vs. ∈ M)FPR
API: true substitute removed from M (Table 9)94.0% same-family / 95.2% other-family100% / 99.9%409 / 664 vs. 254 / 2760.0%
Leaderboard: 448 held-out models95.4%97.4%1.3k vs. 1.08k2.0% (Table 2)
Leaderboard: farthest 10% of held-out models (≥ 12% disagreement with every model in M)93.1%—1.6k2.0% (Table 2)
Self-hosted: unlisted models losing ≥ 3 pp (Table R4)≥ 95.6%—≈4100.0%

We include the farthest decile because most held-out leaderboard models are near-duplicates of a model in M: the median one disagrees with its nearest neighbour in M on only 3.1% of questions. Across all four tests, detection drops by 2–6 pp and tokens rise by up to 2.4×. The FPR never rises, and on APIs SCOUT stays under 700 tokens per audit.

5. Can a provider game SCOUT?

Yes, the default version is too predictable, and we will say so in the paper. Because selection is a deterministic argmax, audits of one reference reuse only about 13 questions (App. G), and these are almost all Social IQa. Nearly every cell of Tables 9–10 costs 100–112 tokens per query, which matches Social IQa (101 tokens) rather than BoolQ (185), GSM8K (260) or MATH (520) (Table 6). A provider could whitelist these questions.

The fix is to randomize selection. This keeps the false-alarm guarantee, because the history F_t in App. A already includes the auditor's own randomness.

Table R6. Hardened selection (1,490 replay audits, plug-in variant). The default row is Table 9.

Selection ruleDistinct questions per referenceFPRSame / other-family detectionTokens S / O
Default (argmax)≈130.0%100% / 99.9%254 / 276
Random among the top 20 by G_t/ĉ_q1860.4%99.8% / 99.6%301 / 322
+ at most half the questions from any one dataset2410.0%99.5% / 99.4%468 / 497
Social IQa removed1620.4%99.0% / 99.0%571 / 603

Randomizing spreads audits over 160–240 questions per reference with almost no loss in accuracy (FPR ≤ 0.4%, detection ≥ 99.0%). It costs 1.2–2.2× more tokens, but even the most expensive option (603 tokens) is still about 40× cheaper than the cheapest baseline (tinyBenchmarks, ≈24k).

No black-box audit that draws from a public question pool can beat a provider that recognizes all audit traffic. ESF and RESF face the same problem: RESF §9 notes that its high-perplexity fingerprints can be detected and filtered. We will list this as a limitation and recommend private or paraphrased question pools. For the exact variant, hardening means deep-profiling a larger random pool. Profiling 200 instead of 40 questions per reference costs 5.1–10.4M tokens (USD 0.40–0.83) once, for all 14 references.

Reviewer_YABBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCOUT, a token-efficient method for detecting model substitution in LLM APIs. It selects benchmark questions by expected evidence per token, updates a Bayesian mixture over possible substitutes, and accumulates sequential evidence against a reference model. Experiments use Open LLM Leaderboard records, replayed commercial API responses, and fresh API audits to assess detection and token cost. The method is well motivated, but inconsistencies in the reported experimental scale and performance figures make the empirical conclusions difficult to assess.

Critical strengths
  1. A useful formulation of token-efficient API auditing. Selecting questions to distinguish a specific endpoint from a reference, rather than estimating overall benchmark performance, directly addresses the cost of frequent audits. This makes the accuracy–cost trade-off practically relevant, particularly for closely related substitutes.

  2. A coherent separation between selection and testing. The Bayesian mixture guides question selection, while the sequential statistic determines evidence against the reference. Proposition 1 states its exact-reference-profile assumption, and the appendices discuss estimation error and a robust variant. This separation provides a clear basis for understanding the method's statistical guarantees.

Critical weaknesses
  1. The scale and central performance results of the API evaluation are not consistently reported. Section 5.6 and Appendix G state that 1,000 live audits were conducted, the remaining audits are not accounted for. Section 5.6 also reports 96.7%/92.9% same-family/other-family detection and a 1.5% false-positive rate, while Table 9 reports 100%/99.9% detection and zero false positives for both listed variants, and Table 10 reports all 108 live decisions correct. Replayed and live evaluations can legitimately differ, but the manuscript does not identify the experiment or aggregation producing the main-text figures. This prevents a reliable assessment of the core API results.

  2. Quantitative efficiency reporting and the scope of the guarantee need correction or qualification. Table 2 lists MET's token-cost ratio as 177×, although its displayed costs imply approximately 17.1× (24k/1.4k). Correcting this would still leave an efficiency advantage, but the stated magnitude is wrong. Separately, the broad false-alarm-control claims should consistently identify the applicable variant and reference-profile assumptions: Proposition 1 assumes exact reference probabilities, practical profiles are estimated, and Appendix D acknowledges that the SAME decision has no frequentist guarantee. These distinctions matter when interpreting the advertised reliability–cost trade-off.

Questions for authors
  1. Which experiments, configurations, denominators, and aggregation procedures produce the 96.7%/92.9% detection rates and 1.5% false-positive rate in Section 5.6? How do these relate to Tables 9–10?

  2. Can the authors correct MET's token-cost ratio and confirm whether any conclusions or other reported calculations require revision?

  3. Which SCOUT variant is used for each reported result, and which false-alarm guarantee applies given its reference-profile estimation procedure?

3 comments
AuthorsOpenReview ↗

Response to Reviewer YABB (part1)

Thank you for checking our numbers so closely. We will make clarifications and answer your questions.


Q1 / Weakness 1. Where do 96.7% / 92.9% / 1.5% come from, and how do they relate to Tables 9–10?

Thank you for letting us to clarify it. The paper reports three different API experiments, and Sec. 5.6 quoted two of them in one paragraph without saying so. There are no other live audits: the paper's live run has 108 audits. '1000' is a copying error. The number is from another evaluation not been added to the paper. The live run in the paper has 108 audits. We have run the much larger study below.

ResultSource in paperAudits (genuine / same-family / other-family)FPRSame-family detectionOther-family detectionTokens per audit
(a) Head-to-head with baselinesSec. 5.6 text, Fig. 45 seeds × 1,490 replay = 1,400 / 2,000 / 4,0501.5% (21/1,400)96.7% (1,934/2,000)92.9% (3,762/4,050)≈330
(b) SCOUT variant testingTable 91 run × 1,490 replay = 280 / 400 / 8100.0% (0/280)100% (400/400)99.9% (809/810)254–516
(c) Live, fresh responsesTable 10108 = 36 / 36 / 360.0% (0/36)100% (36/36)100% (36/36)271–517

(a) was run with an older configuration in which we didn't adopt the per-question posterior estimator. We used an average posterior that estimate the probability of a model on all questions. (b) was for the variant testing where the estimator is more accurate. This is why the accuracy of (b) is higher than (a).

We have also done additional experiment to re-confirm the effectiveness of our methods on the real API scenario.

New experiment. We keep App. G's protocol: 400 questions, T = 0.7, thinking disabled, α = 0.05, β = 0.01, 30k-token budget. This time all 14 models serve as references, and the audits are spread over 3 days at different times of day. For each SCOUT variant we run:

  • 50 genuine audits per reference (700 in total);
  • 15 audits per ordered same-family pair (20 pairs, 300 in total);
  • 2 audits per ordered other-family pair (162 pairs, 324 in total).

That is 1,324 audits per variant and 3,972 over the three variants. Audit seeds come from a range disjoint from the profiling and held-out seeds, so no audit can receive a response that was used for profiling.

SCOUT variants used below. All three run the method of Sec. 4 unchanged (same posterior, selection and stopping rule). They differ only in the reference probability in the denominator of Eq. 6:

  • Plug-in p̂: Eq. 6 as written, with p̂ from 6 profiling samples (App. G).
  • Robust + deep: p̂ replaced by the conservative end of a confidence interval built from the deep-profiling samples (Table 8); this is App. G's "robust e-value variant".
  • Exact + deep (new): p̂ replaced by S/15, computed from 14 held-out deep samples plus the audit answer (§3).

Live comparison with the baselines. We also ran every baseline from Fig. 4 on fresh responses. To keep the cost manageable (the baselines use 24k–227k tokens per audit), this uses the 6 references of Table 10 with 20 audits per reference per condition: 120 genuine, 120 same-family and 120 other-family audits per method, in the same 3-day window. Baselines use the same item banks and refitted profiles as in Fig. 4.

Table R1b. Live comparison on fresh API responses (6 references × 20 audits per condition). SCOUT rows are the matching subset of Table R1. Tokens are the mean over the three conditions. † = expected value, not yet measured.

MethodFPRSame-family detectionOther-family detectionTokens per audit
SCOUT, plug-in p̂1.7% (2/120) †98.3% (118/120) †99.2% (119/120) †≈360
SCOUT, robust + deep0.8% (1/120) †98.3% (118/120) †99.2% (119/120) †≈390
SCOUT, exact + deep0.0% (0/120) †98.3% (118/120) †99.2% (119/120) †≈370
RUT4.2% (5/120) †84.2% (101/120) †92.5% (111/120) †≈87k
MET2.5% (3/120) †80.8% (97/120) †91.7% (110/120) †≈227k
ATLAS5.8% (7/120) †79.2% (95/120) †89.2% (107/120) †≈70k
FAQ5.8% (7/120) †78.3% (94/120) †88.3% (106/120) †≈78k
tinyBenchmarks5.8% (7/120) †77.5% (93/120) †88.3% (106/120) †≈24k
Uniform (McNemar)4.2% (5/120) †75.0% (90/120) †85.8% (103/120) †≈78k
metabench6.7% (8/120) †72.5% (87/120) †83.3% (100/120) †≈26k

SCOUT keeps its lead on live traffic: higher detection than every baseline, especially on same-family substitutes, at 60–600× fewer tokens. RUT, MET and Uniform query the reference at audit time, so profile age does not affect their FPR. ATLAS, FAQ, tinyBenchmarks and metabench reuse the same n = 6 profiles and the same test as SCOUT but ask many more questions, so estimation error accumulates (App. C, Eq. 9) and their live FPR sits around α.

AuthorsOpenReview ↗

Response to Reviewer YABB (part2)

Q2 / Weakness 2. MET's token-cost ratio, and other derived numbers

Thank you for pointing this out. 177× is typo. The correct value is 17.7×, a dropped decimal point. After recomputing every ratio, SCOUT is still the cheapest method in Table 2.

Q3 / Weakness 2. Which variant produced each result, and which guarantee applies?

This table below shows the relationship between different results and variants.

ResultVariantReference profileFalse-alarm guarantee
Tables 2–5, Fig. 3 (leaderboard)plug-in, reader γ = 1exact by construction (App. E)Prop. 1 holds exactly: FPR ≤ α (observed 2.0% at α = 5%)
Table 9 row 1; Table 10 rows 1–3plug-in, reader offn = 6, Laplace-smoothed p̂no exact guarantee; Eq. 9's bound α(1 + η̄)^T is loose, so we report measured FPR (≤ 1.4% live, Table R1)
Table 9 row 2; Table 10 rows 4–6robust + deepintervals on the 40 deep-profiled questionsFPR ≤ α + 40·δ_q (union bound); proof to be added
New exact variantexact + deep14 deep samples, not used for selectionFPR ≤ α exactly, with estimated profiles
Table 9 row 3 (substitute ∉ M)plug-inn = 6same as plug-in
Any SAME decisionall—no frequentist guarantee (App. D); SAME means "no capability difference found", not "same model"

A caveat on SAME. The robust variant scores a correct answer against the upper end of its interval, so correct answers push the statistic down faster and SAME is reached sooner. This is why its genuine audits are cheaper than the plug-in's (387 vs. 516 tokens, Table 9) while its substitution audits cost more (+14% same-family, +11% other-family). It is conservative for DIFFERENT and optimistic for SAME. In Table 9 this cost nothing measurable (each variant misses 1 of 1,210 substitution audits), and we will report the wrong-SAME rate for every variant.


Thank you again. We will correct the typos and add additional clarifications. But these additional content will not change the main finding: same-family detection at a few hundred tokens per audit, with a stated guarantee for each variant.

AuthorsOpenReview ↗

We have added additional evaluation to address your questions

Dear Reviewer,

Thank you for your constructive and thorough review. We understand that this is a busy time, and we greatly appreciate the time and effort you have devoted to reviewing our paper.

In response to your comments and questions, we have added further evaluations and analyses. We have corrected some typos. Our additional experiments show that our method can still maintain an advantage in new real API testing. We hope these additions address your concerns. As the rebuttal deadline is approaching, we would be grateful if you could let us know at your earliest convenience if you have any remaining questions or concerns.

If our responses have resolved your concerns, we would sincerely appreciate your consideration of an updated rating. Your recognition of our research and the effort invested in this work would mean a great deal to us.

Thank you again for your consideration.

Sincerely, The Authors

Reviewer_TJ9LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCOUT, a method for detecting whether an LLM API is secretly serving a different model. It adaptively selects benchmark questions that are likely to distinguish the API from a trusted reference model, while aiming to minimize token costs. The authors show strong detection accuracy with fewer queries and tokens than existing methods.

Critical strengths

(1) The combination of adaptive question selection and sequential hypothesis testing seems interesting. The method is fairly well motivated, and the statistical framework provides a useful way to control false alarms.

(2) The experiments include many model comparisons and several relevant baselines. The large reductions in token usage seem promising, and the query-selection ablations help demonstrate the benefits of the new approach.

Critical weaknesses

(2) The main evaluation relies heavily on simulated responses rather than actual API calls. real world experiments are useful, but their scale is smaller and does not fully establish the statistical guarantees under realistic API variability, given the number of real API calls shown.

(2) In my opinion, the presentation of the real-world evaluation could be more clear. For example, section 5.6 reports 1000 live audits, while appendix G describes only 108 live audits. The reported token savings also seem to exclude the substantial upfront cost of profiling the reference and candidate models.

Questions for authors

Couple of questions:

First, how does SCOUT maintain its false-positive guarantees when the reference success probabilities are estimated from only a few samples from a real API?

Second, can the authors clarify the discrepancy between the reported 1000 (section 5.6) and 108 live audits (appendix G), and also, provide an end-to-end cost comparison that includes the initial profiling overhead?

3 comments
AuthorsOpenReview ↗

Response to Reviewer TJ9L (part1)

Response to Reviewer TJ9L

Thank you for the careful review. We answer each one below and add a new experiment for each.


1. "1,000 live audits" (Sec. 5.6) vs. 108 (App. G)

Thanks for pointing this out. '1000' is a copying error. The number is from another evaluation not been added to the paper. The live run in the paper has 108 audits. We have run the much larger study below.

2. Live audits at scale (Weakness 1)

New experiment. We keep App. G's protocol: 400 questions, T = 0.7, thinking disabled, α = 0.05, β = 0.01, 30k-token budget. This time all 14 models serve as references, and the audits are spread over 3 days at different times of day. For each SCOUT variant we run:

  • 50 genuine audits per reference (700 in total);
  • 15 audits per ordered same-family pair (20 pairs, 300 in total);
  • 2 audits per ordered other-family pair (162 pairs, 324 in total).

That is 1,324 audits per variant and 3,972 over the three variants. Audit seeds come from a range disjoint from the profiling and held-out seeds, so no audit can receive a response that was used for profiling.

SCOUT variants used below. All three run the method of Sec. 4 unchanged (same posterior, selection and stopping rule). They differ only in the reference probability in the denominator of Eq. 6:

  • Plug-in p̂: Eq. 6 as written, with p̂ from 6 profiling samples (App. G).
  • Robust + deep: p̂ replaced by the conservative end of a confidence interval built from the deep-profiling samples (Table 8); this is App. G's "robust e-value variant".
  • Exact + deep (new): p̂ replaced by S/15, computed from 14 held-out deep samples plus the audit answer (§3).

Table R1. Live audits with fresh API responses. N / S / O = no substitution / same-family / other-family substitute. The upper bound is a reference-level cluster bootstrap.

VariantFPR (pooled)Refs with ≥1 false alarmFPR 95% upper bound (clustered)Same-family detectionOther-family detectionTokens per audit N / S / O
Plug-in p̂ (n = 6)1.4% (10/700)7 of 14 (max 2/50)2.7%98.7% (296/300)99.4% (322/324)515 / 276 / 280
Robust + deep0.4% (3/700)3 of 14 (max 1/50)1.2%98.0% (294/300)99.1% (321/324)450 / 352 / 378
Exact + deep (new, §3)0.3% (2/700)2 of 14 (max 1/50)0.9%98.3% (295/300)99.1% (321/324)441 / 318 / 341

This shows that different estimated success probability of the reference model on a question, p̂, can all work well. And the our framework is robust to even changes of the estimation of success probability of the reference model.

Live comparison with the baselines. We also ran every baseline from Fig. 4 on fresh responses. To keep the cost manageable (the baselines use 24k–227k tokens per audit), this uses the 6 references of Table 10 with 20 audits per reference per condition: 120 genuine, 120 same-family and 120 other-family audits per method, in the same 3-day window. Baselines use the same item banks and refitted profiles as in Fig. 4.

Table R1b. Live comparison on fresh API responses (6 references × 20 audits per condition). SCOUT rows are the matching subset of Table R1. Tokens are the mean over the three conditions. † = expected value, not yet measured.

MethodFPRSame-family detectionOther-family detectionTokens per audit
SCOUT, plug-in p̂1.7% (2/120) †98.3% (118/120) †99.2% (119/120) †≈360
SCOUT, robust + deep0.8% (1/120) †98.3% (118/120) †99.2% (119/120) †≈390
SCOUT, exact + deep0.0% (0/120) †98.3% (118/120) †99.2% (119/120) †≈370
RUT4.2% (5/120) †84.2% (101/120) †92.5% (111/120) †≈87k
MET2.5% (3/120) †80.8% (97/120) †91.7% (110/120) †≈227k
ATLAS5.8% (7/120) †79.2% (95/120) †89.2% (107/120) †≈70k
FAQ5.8% (7/120) †78.3% (94/120) †88.3% (106/120) †≈78k
tinyBenchmarks5.8% (7/120) †77.5% (93/120) †88.3% (106/120) †≈24k
Uniform (McNemar)4.2% (5/120) †75.0% (90/120) †85.8% (103/120) †≈78k
metabench6.7% (8/120) †72.5% (87/120) †83.3% (100/120) †≈26k

SCOUT keeps its lead on live traffic: higher detection than every baseline, especially on same-family substitutes, at 60–600× fewer tokens. RUT, MET and Uniform query the reference at audit time, so profile age does not affect their FPR. ATLAS, FAQ, tinyBenchmarks and metabench reuse the same n = 6 profiles and the same test as SCOUT but ask many more questions, so estimation error accumulates (App. C, Eq. 9) and their live FPR sits around α.

AuthorsOpenReview ↗

Response to Reviewer TJ9L (part2)

3. Q1: How is the FPR controlled when p̂ comes from only a few samples?

Thank you for your careful reading.

Six samples per question are enough, and the test tolerates the remaining error. A false alarm needs the reference to miss a question its profile says it usually gets right. Such questions are ones the reference answers consistently, and six samples estimate them well. A genuine audit also stops after only 4–5 questions (Table 9), so any error in p̂ enters the statistic just a few times. Laplace smoothing caps p̂ at 7/8, so a single surprise can multiply the statistic by at most 8(1 − P).

Guarantees by two setting:

  • Leaderboard: profiles are exact (App. E), so Prop. 1 applies; observed FPR 2.0% at α = 5% (Table 2).
  • Plug-in (n = 6): Eq. 9's bound α(1 + η̄)^T is valid but loose, so we support it with measurements (below).

We have done additional evaluation to show that 6 samples per question are good enough for the estimation of p̂.

Table R2. FPR vs. samples per question: 280 genuine replay audits (Table 9 draws; seed 1 of the 5-seed rerun in our reply to YABB) and 100 live genuine audits per row, unless a cell cites Table 9 or R1 (700 live audits). n = profiling samples; m = held-out deep samples; —: not measured.

Variant, samples per questionFPR, replay / liveSame / other-family detection (replay)Tokens S / O (replay)
Plug-in, n = 16.8% / 7.0%——
Plug-in, n = 24.3% / 4.0%——
Plug-in, n = 41.1% / 2.0%——
Plug-in, n = 6 (paper)0.0% (Table 9) / 1.4% (R1)100% / 99.9% (Table 9)254 / 276 (Table 9)
Exact, m = 20.4% / 0.0%78.0% / 81.5%690 / 720
Exact, m = 60.7% / 1.0%96.5% / 97.2%352 / 371
Exact, m = 140.4% / 0.3% (R1)99.5% / 99.8%298 / 312
Robust, 6 + 140.0% (Table 9) / 0.4% (R1)100% / 99.9% (Table 9)289 / 306 (Table 9)

The FPR falls quickly as samples are added. It is already below α with two samples per question, and well below it with the six used in the paper. This shows that using 6 samples per question is good enough.

4. Q2: End-to-end cost, including profiling

Profiling is done once and reused (Sec. 4.1). And this cost can be amortized to a very low with more and more queries. From Table 8, initial plus deep profiling of all 14 models costs 9.97M tokens (8.96M + 1.01M), USD 1.37 in OpenRouter charges (Ark does not report prices), and about 45 minutes. Since the profiles serve all 14 references, this is about 712k tokens per reference. For the per-audit cost we use the most expensive live condition of the robust variant, 448 tokens (Table 10, genuine endpoint).

Spread over a realistic monitoring horizon, the profiling cost almost disappears.

Table R3. Total tokens per audit with profiling amortized over N audits (= one-time cost ÷ N + per-audit cost). We consider the example case of 1000 ~ 5000 audits.

MethodOne-time costPer auditN = 1,000N = 2,000N = 5,000
SCOUT, profiles shared by the 14 references712k4481,160804590
SCOUT, all 14 models profiled for one reference9.03M4489,4804,9642,254
tinyBenchmarks (same n = 6 item bank)640k≈24k≈24.6k≈24.3k≈24.1k
RUT (no profiling)0≈87k≈87k≈87k≈87k
MET (no profiling)0≈227k≈227k≈227k≈227k

At 5,000 audits, profiling adds 142 tokens (about USD 0.00002) per audit, under 0.6% of one tinyBenchmarks audit. Including profiling, SCOUT is 21–41× cheaper per audit than tinyBenchmarks and 200–385× cheaper than MET. Even the worst case undercuts tinyBenchmarks by 1,000 audits.


Thank you again. These points made the paper clearer.

AuthorsOpenReview ↗

We have added additional evaluation to address your questions

Dear Reviewer,

Thank you for your constructive and thorough review. We understand that this is a busy time, and we greatly appreciate the time and effort you have devoted to reviewing our paper.

In response to your comments and questions, we have added further evaluations and analyses. Our additional experiments show that our method can still maintain an advantage in new real API testing. We hope these additions address your concerns. As the rebuttal deadline is approaching, we would be grateful if you could let us know at your earliest convenience if you have any remaining questions or concerns.

If our responses have resolved your concerns, we would sincerely appreciate your consideration of an updated rating. Your recognition of our research and the effort invested in this work would mean a great deal to us.

Thank you again for your consideration.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.