SCOUT: Token-Efficient Detection of Model Substitution in LLM APIs

When people use a large language model (LLM) through an API, they cannot see which model actually answers their requests. A provider can quietly swap the advertised model for a cheaper sibling or a quantized variant. Current defenses are either unreliable or too expensive to run often. We frame the task as model difference probing: detecting whether an endpoint's per-question success probabilities differ from those of a trusted reference. We propose SCOUT, a sequential auditor designed to use few tokens to tackle this. SCOUT models the unknown substitute as a Bayesian mixture of public benchmark models. At each step, it asks the question expected to best separate the served model from the reference per token spent. It then uses the answer to update both its posterior over the substitute and an anytime-valid test statistic that keeps the false-alarm rate below a chosen level $\alpha$. On the Open LLM Leaderboard and on real commercial APIs, SCOUT is more accurate than all prior methods while using over 90% fewer tokens.