- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SASR, a defense against on-policy distillation (OPD). At student-generated contexts, it shrinks the teacher–student log-probability gap at high-disagreement positions, while preserving the teacher's top-1 prediction and bounding the KL distortion. Across three OPD variants on math reasoning benchmarks with Qwen models, SASR reduces distilled student accuracy more than the CMI baseline.
Critical strengths
- The paper is easy to read and follow.
- OPD is quickly becoming a standard post-training technique, and this paper is among the first to study defenses tailored to its token-level, student-conditioned supervision.
- It casts protection as constrained maximin optimization, and its decoupling of the teacher's decision from the learning signal (Prop. 1) motivates a simple, closed-form, training-free method.
Critical weaknesses
- Positioning and motivation
1.1 Top-1 preservation is the wrong utility measure: First, real LLM decoding is rarely greedy. It samples with temperature, top-p or top-k, so keeping the top-1 token does not keep the output the same. Second, SASR only changes scores during distillation, so users never see the modified distribution anyway. Preserving top-1 therefore protects nothing that users actually experience.
1.2 Unrealistic threat model. For closed-source models, major APIs expose log-probabilities only for tokens the model generates itself, and the defender cannot access the attacker’s student probabilities. For open-weight models, the attacker can run the teacher locally, so an interface-level defense does not apply, and distillation is generally permitted anyway. Neither case matches the paper’s setup.
- Dense OPD supervision is neither necessary nor uniformly reliable:
Teacher feedback on student prefixes is noisy (Zhu et al., 2026). Supervising only early tokens can beat full-rollout OPD (Zhou et al., 2026). Concurrent work finds that 1–2 tokens per trajectory can match full-token training (Extremely Sparse Supervision, 2026).
So an attacker could train on a small, clean subset of tokens and bypass SASR, which perturbs only 20% of positions. The recovery attacks evaluated in the paper do not cover this. It is also unclear whether high- positions carry useful signal or noise that was already there. An analysis by token position, plus an evaluation against sparse or early-token attackers, is needed.
references: Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi, and Ge Liu. The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes. arXiv preprint arXiv:2605.11182, 2026. Ziheng Zhou, Jiaqi Li, Huacong Tang, Ying Nian Wu, and Demetri Terzopoulos. Less is More: Early Stopping Rollout for On-Policy Distillation. arXiv preprint arXiv:2605.27028, 2026. Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, and Karim Bouyarmane. Extremely Sparse Supervision Incentivizes Reasoning Ability. arXiv preprint arXiv:2609.04565, 2026.
Questions for authors
- Experiments cover only math reasoning, only Qwen models, and only 0.5B–4B scales. How about generalization to other domains, model families, and larger teachers?
- How sensitive are the results to , , and ? Does allowing sign flips () strengthen protection?
- Where along the response do the selected high- positions concentrate? Is the degradation driven by early or late tokens?
2 comments
Follow-up comment
Thank you for the response and the additional experiments, which address part of my questions. Two main concerns remain.
-
Sparse or early-token attackers. The ESR100 results support this concern: SASR lowers accuracy by only 0.48 pp at step 10, with a confidence interval that includes zero, so the attacker keeps about 84% of the distillation gain. Sparser attackers are still untested.
-
Top-1 utility. No evaluation of the teacher’s generation quality under sampling is provided. Since SASR only changes the scoring interface, which users never see, it remains unclear what the top-1 constraint protects.
I therefore maintain my rating of 2.
Response to the Reviewer 3JPs(Question 2 and Question 3)
We thank the reviewer for the comments. We address each point below.
weaknees 1:
1.1 Teacher Fidelity: Top-1 preservation constrains local predictions at the OPD scoring interface. We emphasize that SASR combines Top-1 decision preservation with KL constraints to maintain teacher local prediction , thereby aiming to minimize its impact on teacher generation behavior. Under stochastic decoding, these constraints also help preserve sampling behavior, although they cannot guarantee unchanged behavior across all decoding settings.
1.2 threat model: In this paper, we use open-weight models as a controlled testbed. For provider-controlled scoring interfaces,we evaluated Proxy-SASR using Qwen2.5-0.5B as the proxy and Qwen2.5-1.5B as the target student. As shown in Table below, after 40 training steps, accuracy under Proxy-SASR was 4.83 percentage points lower on MATH-500 and 5.63 pp lower on AMC 2023 than under Clean OPD. These results provide preliminary evidence that Proxy-SASR can limit distillation gains when the defender cannot access token probabilities from the target student.
weakness 2: Our position audit confirms that SASR also modifies early tokens. When distillation uses only the first 100 tokens (ESR100, Table 3), SASR yields slightly lower student accuracy, with a paired confidence interval that includes zero.
Q2: Hyperparameter Sensitivity and Sign Flipping
We provide an exploratory sensitivity analysis of SASR's hyperparameters and examine the effects of varying these parameters individually, along with the resulting sign-flipping rates, as summarized in Table below.
1. Parameter Sensitivity:
Student accuracy rose by 1.37 pp as the selection rate increased from 20% to 30%, indicating weaker protection. Increasing intervention strength from 1.05 to 1.25 improved protection (−5.47 pp). The 2.15 pp drop after equalizing the asymmetric weights (0.5:1 to 1:1) suggests that the balance between the two branches matters.
2. Effect of Sign Flipping:
A higher sign-flipping rate does not necessarily lead to stronger protection. Varying λ lowered student accuracy by 5.47 pp with little change in flip rate (12.93% → 12.87%). Flip rates of 42.89% and 65.16% corresponded to accuracies of 53.52% and 58.01%, respectively, across configurations that varied in intervention strength, token selection, and direction-aware reshaping.
For the default configuration used in our paper, we further audit over 8.03 million valid tokens from the 40-step training run. Teacher decisions were preserved, with mean KL of 0.0201 nats overall and 0.1004 nats at selected positions. At steps 1, 10, 20, and 40, mean KL was 0.0228, 0.0215, 0.0178, and 0.0200 nats, respectively.
Q3: Distribution of Selected Token Positions and Early-Token Supervision We appreciate the reviewer for the question regarding the influential supervision positions selected by SASR and their impact. We conduct additional token-position distribution audits and early-token supervision distillation (ESR100) experiments.
1. Influential Supervision Positions. SASR selects sampled tokens by the absolute teacher–student log-probability gap Δt and reshapes supervision asymmetrically by sign. Across 64 frozen trajectories, selection peaked at 10%–25% of response length (22.98% vs. 19.22% in the final quarter). Of the first 100 tokens, 14.69% were selected and 14.39% modified. Negative gaps accounted for 76.91% of early selections.
2. Early-Token Supervision (ESR100). Following Zhou et al. (2026), we distilled only the first 100 generated tokens. At Step 10, Avg@8 was 33.59% with Clean OPD and 33.11% with SASR (Table 3), suggesting modest suppression of early-token distillation gains (0.48 pp). Overall, SASR's interventions are not concentrated at late response positions but are distributed across generation positions based on teacher–student discrepancies. The early-token supervision experiment also exhibits a consistent directional trend toward reduced distillation gains.
Table 3