- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors consider an attacker aiming to reverse engineer a model via query-level access to it from a theoretical perspective. They connect stateful and stateless defenders to prior learning theory concepts (namely PAC and its adjacent concepts). In addition to a variety of formal results, they present a positive and negative example which provide further intuition.
In total, the paper serves to develop the foundation of adversarial learning theory.
Critical strengths
The paper leans on existing learning theory and makes ties to already well-established theoretical frameworks.
The authors provide a thorough and clear treatment of various quantifier orderings.
The result of section 4 ("anti-distillation defenses must exploit the fact that the attacker is computationally bounded") is profound. It reminds me of the work out there (that doesn't often get used in adversarial learning) where you consider a polynomial-time bounded attacker. I could see the theoretical foundations laid by this work leading to a rich set of works further diving into how one can exploit an attacker's computational limitations.
Critical weaknesses
Unless I missed something, all results are for a finite model class. The term |H| shows up in the bounds, but for practical models (the authors use the example of LLMs under 1TB in size), that's very big (2^2^40). Even sqrt(log(|H|)) is still extremely large, making me question the vacuousness of the bounds.
Separately: Oh how I would benefit from a notation table.
Questions for authors
-
Can you plug in some numbers? Namely, for the example you provide (LLMs less than 1TB in size), for particular loss functions (e.g., TV as you suggest) and reasonable numbers of queries, what do the bounds actually state?
-
I'm struggling with Theorem 4.1: Let H be the set of all linear classifiers using 2^40 bits of precision. Being \gamma-insecure (with the gamma value you state) means there exists an h such that the risk is greater than 2(alpha + 2^40/sqrt(n))? (What) am I doing wrong?
-
Do any of your results apply to infinite hypothesis classes (say, with finite VC dimension)?
2 comments
I thank the authors for their response and I maintain my positive assessment of the work.
Thank you for your positive review and constructive feedback. We answer your questions and respond to the weaknesses below.
1. Vacuousness of the ∣H∣ Bound & Plugging in Numbers (Weakness 1 & Questions 1–2)
Plugging in numbers. Consider a 1TB model, i.e. ∣H∣=2243. Under TV loss, Theorem 4.1 gives Risk≤2α+2n1ln∣H∣. This bound is non-trivial once n≳2.4×1013, and the additive error drops to 0.1 at n≈2.4×1015.
Counting each generated token as one query, OpenRouter alone produced roughly 5×1013 output tokens in October 2026 (7.9×1014 total, about 6% of which are output). So the threshold n≈2.4×1013 is a few weeks of one router's output. So even this worst-case bound, which ignores all structure in the model, becomes meaningful at deployment scale.
In general, the log∣H∣ dependence cannot be improved: identifying the model requires log2∣H∣ bits, and each query reveals only O(1) bits. Real models have far more structure than an arbitrary B-bit function -- we are not making any assumptions about the model architecture, precision, etc. -- so in practice distillation needs far fewer queries. For classes of finite VC dimension, Remark C.4 already replaces ln∣H∣ with dVC. (See our response to the second question below.)
The point of this generic result is that there is no fundamental barrier to distillation. But the attack itself is not practical, since it runs in time polynomial in ∣H∣; this is why any real defense must rely on computational hardness.
2. Infinite Hypothesis Classes with Finite VC Dimension (Questions 2 & 3)
Yes—Theorem 4.1 extends to infinite hypothesis classes with finite VC dimension. Specifically, for any infinite class of deterministic binary classifiers H⊆0,1X with finite VC dimension dVC≥1, honest-query ERM achieves universal γ-distillation insecurity with an explicit bound γ=2α+n2dVCln(en/dVC) for n≥dVC or an asymptotic bound γ=2α+O(dVC/n) even against stateful servers (added in Section 4 and Remark C.4).
If H is the class of linear classifiers in low dimension d (over Rd), then even if each number uses 1000 bits of precision (or infinite real precision), its VC dimension is dVC=d+1, so this bound gives a non-trivial guarantee independent of bit precision. For example, with d=100 (dVC=101), α=0.05, and n=106 queries, the explicit bound above gives n2dVCln(en/dVC)≈0.045, yielding Risk<0.15.
3. Remark on the Direction of γ (Question 2)
The theorem says all α-accurate servers are γ-insecure i.e. there exists a computationally unbounded attacker that can attack any accurate server and find a h such that the risk is less than the stated bound. (Note that the log∣H∣ term is also inside the square root so it would be γ=2(α+220/n).)
4. Notation Table (Weakness 2)
We have added a notation table (Table 3.1) in Section 3.