acceptodds
Under review as a conference paper at ICLR 2027

PAINT: Adaptive Privileged-Context Self-Distillation for Reasoning

Abstract

Effective reasoning supervision must act on the states a model visits while offering richer guidance than a final-answer reward. We frame on-policy self-distillation as an information-allocation problem: training-only context re-scores student-generated prefixes, and its influence is set by how much information is exposed and where the resulting token targets are calibrated. We propose PAINT (Privileged-context Adaptive INterpolated Training), which makes these two choices at different resolutions. For mathematical reasoning, rollout-reference overlap determines how much of a verified solution the privileged scorer sees; a teacher/student entropy ratio then selects a sparse set of prefixes for small energy-space interpolation. Conditional mutual information explains the value of context exposure, and geometric interpolation controls the local target. Across three competition-level math benchmarks and three Qwen3 scales, PAINT improves over full-solution on-policy self-distillation. On Qwen3-8B, it raises macro Avg@12 by 2.1 points over that baseline and 2.9 points over GRPO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on policy distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5F9pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds on on-policy self-distillation (OPSD; Zhao et al., 2026). A frozen copy of the initial model, given the reference solution, re-scores the prefixes of the student's own rollout, and the student matches this distribution with a clipped full-vocabulary forward KL. PAINT adds two components:

  • Overlap-adaptive suffix masking. Anchors are extracted from the reference solution (the boxed answer, formula-like spans, numbers), and α\alpha is the fraction of them that appear in the rollout. A character-level suffix of the reference is then hidden, with ratio ρ(α)=0.30+0.10α\rho(\alpha) = 0.30 + 0.10\alpha.
  • Sparse energy interpolation. At the top 3% of positions per sample by teacher/student entropy ratio HT/HSH_T/H_S, the teacher target is replaced by p~T∝pT0.97 pS0.03\tilde p_T \propto p_T^{0.97}\, p_S^{0.03}.

The theory consists of Prop 3.1 (a conditional mutual information view), Thm 3.2 (monotonicity of the mask ratio), Thm 3.3 (suffix placement), and Prop 3.4 (the geometric mixture as a KL barycenter).

Experiments train Qwen3-1.7B/4B/8B with LoRA for 100 steps on OpenThoughts math, using one rollout of up to 1024 tokens per problem, and report Avg@12 on AIME24, AIME25, and HMMT25. PAINT improves macro Avg@12 over OPSD by 0.8–2.1 points and over GRPO by 1.7–6.8 points. All ablations are on Qwen3-1.7B, AIME24 only.The paper builds on on-policy self-distillation (OPSD; Zhao et al., 2026). A frozen copy of the initial model, given the reference solution, re-scores the prefixes of the student's own rollout, and the student matches this distribution with a clipped full-vocabulary forward KL. PAINT adds two components:

  • Overlap-adaptive suffix masking. Anchors are extracted from the reference solution (the boxed answer, formula-like spans, numbers), and α\alpha is the fraction of them that appear in the rollout. A character-level suffix of the reference is then hidden, with ratio ρ(α)=0.30+0.10α\rho(\alpha) = 0.30 + 0.10\alpha.
  • Sparse energy interpolation. At the top 3% of positions per sample by teacher/student entropy ratio HT/HSH_T/H_S, the teacher target is replaced by p~T∝pT0.97 pS0.03\tilde p_T \propto p_T^{0.97}\, p_S^{0.03}.

The theory consists of Prop 3.1 (a conditional mutual information view), Thm 3.2 (monotonicity of the mask ratio), Thm 3.3 (suffix placement), and Prop 3.4 (the geometric mixture as a KL barycenter).

Experiments train Qwen3-1.7B/4B/8B with LoRA for 100 steps on OpenThoughts math, using one rollout of up to 1024 tokens per problem, and report Avg@12 on AIME24, AIME25, and HMMT25. PAINT improves macro Avg@12 over OPSD by 0.8–2.1 points and over GRPO by 1.7–6.8 points. All ablations are on Qwen3-1.7B, AIME24 only.The paper builds on on-policy self-distillation (OPSD; Zhao et al., 2026). A frozen copy of the initial model, given the reference solution, re-scores the prefixes of the student's own rollout, and the student matches this distribution with a clipped full-vocabulary forward KL. PAINT adds two components:

  • Overlap-adaptive suffix masking. Anchors are extracted from the reference solution (the boxed answer, formula-like spans, numbers), and α\alpha is the fraction of them that appear in the rollout. A character-level suffix of the reference is then hidden, with ratio ρ(α)=0.30+0.10α\rho(\alpha) = 0.30 + 0.10\alpha.
  • Sparse energy interpolation. At the top 3% of positions per sample by teacher/student entropy ratio HT/HSH_T/H_S, the teacher target is replaced by p~T∝pT0.97 pS0.03\tilde p_T \propto p_T^{0.97}\, p_S^{0.03}.

The theory consists of Prop 3.1 (a conditional mutual information view), Thm 3.2 (monotonicity of the mask ratio), Thm 3.3 (suffix placement), and Prop 3.4 (the geometric mixture as a KL barycenter).

Experiments train Qwen3-1.7B/4B/8B with LoRA for 100 steps on OpenThoughts math, using one rollout of up to 1024 tokens per problem, and report Avg@12 on AIME24, AIME25, and HMMT25. PAINT improves macro Avg@12 over OPSD by 0.8–2.1 points and over GRPO by 1.7–6.8 points. All ablations are on Qwen3-1.7B, AIME24 only.The paper builds on on-policy self-distillation (OPSD; Zhao et al., 2026). A frozen copy of the initial model, given the reference solution, re-scores the prefixes of the student's own rollout, and the student matches this distribution with a clipped full-vocabulary forward KL. PAINT adds two components:

  • Overlap-adaptive suffix masking. Anchors are extracted from the reference solution (the boxed answer, formula-like spans, numbers), and α\alpha is the fraction of them that appear in the rollout. A character-level suffix of the reference is then hidden, with ratio ρ(α)=0.30+0.10α\rho(\alpha) = 0.30 + 0.10\alpha.
  • Sparse energy interpolation. At the top 3% of positions per sample by teacher/student entropy ratio HT/HSH_T/H_S, the teacher target is replaced by p~T∝pT0.97 pS0.03\tilde p_T \propto p_T^{0.97}\, p_S^{0.03}.

The theory consists of Prop 3.1 (a conditional mutual information view), Thm 3.2 (monotonicity of the mask ratio), Thm 3.3 (suffix placement), and Prop 3.4 (the geometric mixture as a KL barycenter).

Experiments train Qwen3-1.7B/4B/8B with LoRA for 100 steps on OpenThoughts math, using one rollout of up to 1024 tokens per problem, and report Avg@12 on AIME24, AIME25, and HMMT25. PAINT improves macro Avg@12 over OPSD by 0.8–2.1 points and over GRPO by 1.7–6.8 points. All ablations are on Qwen3-1.7B, AIME24 only.

Critical strengths

1. A natural question and a nearly free method. PAINT needs no extra generation, only a masked teacher prompt and a target interpolation at a few positions. If the results hold, two findings are practically useful: moderate suffix masking beats the full solution while heavier masking hurts (Table 2), and suffix masking beats middle and prefix masking (Table 3).

2. Consistent direction. PAINT is above OPSD in all nine benchmark–scale cells of Table 1. If the comparison is controlled, this is the strongest evidence in the paper.

Critical weaknesses

1. The comparison may not be controlled, and the gains are within noise.

Baselines. All 48 baseline numbers in Table 1 (No update, SFT, GRPO, OPSD × 3 scales, including the macro average) are identical to Table 2 of the OPSD paper (arXiv:2601.18734v3). The paper cites Zhao et al. (2026) for the method but never states that these numbers are taken from it. The text presents them as the authors' own runs, and OPSD as run in the same pipeline as PAINT: see the Table 1 caption (same training split, rollout budget, and fixed privileged teacher), §4.1 ("We evaluate four updates from the same base checkpoints"), and App. B (the vanilla baseline keeps the same training budget and teacher anchoring). In the OPSD paper, the OPSD numbers are the best of checkpoints evaluated every 20 steps up to step 100, and the GRPO numbers are the peak within 500 steps (v3, Table 2 caption). These are checkpoints selected on the test benchmarks. The ablation reference value of 57.2 (Table 2, first row; §4.3) is the same number.

Selection. For PAINT, App. B says only that checkpoints were evaluated every 20 steps up to step 100. It does not say which checkpoint is reported. With no validation split mentioned, the reported checkpoint was likely chosen on the test benchmarks. The final hyperparameters are also the best of the 15 PAINT variants in the ablations, all on 1.7B AIME24. On 1.7B, the gain on this tuning benchmark is +2.5, versus +0.3 and +0.5 on the other two.

Noise. With 30 problems × 12 samples, sampling noise alone gives a standard error of pi(1−pi)‾/360\sqrt{\overline{p_i(1-p_i)}/360} per benchmark (pip_i: per-problem accuracy). That is roughly 1–2 points, and at most about 2.6. The standard error of a difference between two models is about 2\sqrt{2} times larger. The OPSD paper itself shows fluctuations of this size: within one run, its 1.7B AIME25 score was 43.9 at step 50 and 41.1 at step 100 (v3, Table 3, forward-KL row). Seven of the nine gains in Table 1 are 1.4 points or less, and the ablation gains are 1.4 (masking) and 1.1 (interpolation). No seeds, confidence intervals, or training curves are reported.

Why this matters. The central claim that PAINT outperforms OPSD is not established by the current evidence. The 9/9 consistency is suggestive, but what it means depends on how the baselines were obtained and how checkpoints were selected.

2. The mechanism is not isolated, and the theory does not support the design.

Adaptivity vs. average mask ratio. The adaptive range is ρ∈[0.30,0.40]\rho \in [0.30, 0.40], but Table 2 has no fixed ratio in that range (0.30, 0.35, 0.40). The fixed ratios are only 0, 0.25, 0.50, and 0.75, and there is no reversed schedule. The +0.8 over fixed 0.25 may simply reflect a better average ratio. The distribution of α\alpha is not shown either. If rollouts are cut at 1024 tokens before reaching the later anchors (including the boxed answer), α\alpha may measure how far the rollout progressed rather than how closely it followed the reference.

Confounded with hiding the answer. A 30–40% suffix mask usually hides the final answer at the end of the solution, while prefix and middle masks leave it visible. The paper's own Fig. 2 example hides everything after (a−6)(b−6)=36(a-6)(b-6)=36, so the answer is not shown, yet the prompt asks the teacher to "derive the same final answer." The OPSD paper attributes SFT's degradation to the reference solutions being concise (v3, Table 2 caption and §4.2), which suggests y⋆y^\star is short and ends with the answer. Tables 2 and 3 are therefore consistent with two explanations. One is Thm 3.3's (keep the opening setup visible). The other is simpler (do not show the teacher the answer).

Theory–method gap.

  • Prop 3.1 is the definition of conditional mutual information plus data processing.
  • Prop 3.4 is the standard geometric-mixture / KL-barycenter identity. Moving the target toward the student lowers the loss at that position by construction, so it is not evidence that training improves. It also concerns the unclipped KL, whereas the loss actually used (Eqs. 3 and 12) is clipped.
  • Thm 3.2 is a result for a posited surrogate objective, not one derived from the training objective. Nothing links the character-level mask ratio to the mixture weight, or α\alpha to ϵ\epsilon, so it cannot justify the direction of the schedule.
  • Thm 3.3 follows directly from Assumption D.8, which is not tested. By Lemma D.7, Γ(t)=K−E[I(An;Ct∣x,y^<n)]\Gamma(t) = K - \mathbb{E}[I(A_n; C_t \mid x, \hat y_{<n})], so the assumption is equivalent to the conclusion.
  • The interpolation shifts the target 3% of the way toward the student at 3% of positions, a very small change. Its only support is a 0.8–1.1 point difference on one benchmark.

Why this matters. "Information allocation" is presented as the explanation for the gains, but it is not distinguished from simpler explanations (the average mask ratio, hiding the answer).

Questions for authors

Clarification requests (answers could change my assessment)

Q1. Baselines. Did you run the No update / SFT / GRPO / OPSD baselines yourselves, or take the numbers from the OPSD paper? If they were taken, please say so. At minimum, please re-run OPSD with the same code, seeds, and checkpoint rule as PAINT.

Q2. Selection and variance.

  • Was the PAINT checkpoint chosen per benchmark or per model?
  • Were the hyperparameters chosen on 1.7B AIME24?
  • Please report all five checkpoints (steps 20–100) with at least three seeds (or, at minimum, bootstrap CIs), and select hyperparameters on held-out data.

Q3. What the teacher sees.

  • Is y⋆y^\star a short solution or a long reasoning trace?
  • For each mask type, how often is the final answer visible to the teacher?
  • A mask that hides only the answer would directly test the answer-hiding explanation.

Q4. The overlap signal.

  • Please report:
    • the distribution of α\alpha;
    • the fraction of examples with no anchors;
    • whether training rollouts use thinking mode;
    • the fraction of rollouts hitting the 1024-token cap;
    • the fraction containing a boxed answer;
    • the correlation between α\alpha and correctness.
  • Please also add fixed ρ\rho = 0.30, 0.35, 0.40 and a reversed schedule to Table 2.

Q5. Selected positions and clipping.

  • How is RnR_n handled when HSH_S is close to 0?
  • Are the selected positions mostly formatting/LaTeX tokens or reasoning decisions?
  • Eq. (3) clips each vocabulary term at τ=0.06\tau = 0.06, and a term can only be clipped when pT(v)>pS(v)p_T(v) > p_S(v). A clipped term has no direct gradient. Through the softmax normalization, the loss gradient on that logit is, up to a positive constant, pS(v)∑u∉CpT(u)>0p_S(v)\sum_{u \notin \mathcal{C}} p_T(u) > 0 (C\mathcal{C}: the clipped set). So these tokens, which the teacher prefers, are only pushed down. Since interpolation changes which terms are clipped, please report the clipping frequency at selected vs. unselected positions.

Suggestions

  • Relate the work to hint / partial-solution methods:
    • QuestA (Li et al., ICLR 2026) adds partial solutions to the prompt during RL.
    • UFT (Liu, Farina & Ozdaglar, NeurIPS 2025) uses solution prefixes as hints.
    • Guide (Nath et al., 2025) adds guidance only when all unguided attempts fail.
  • Explain why the cited self-distillation methods (Hübotter et al., 2026; Shenfeld et al., 2026) are not compared.
  • Test the motivating claim that the full solution concentrates the target on a single trace, e.g., with pass@k, output entropy, and response length.
  • Describe the GRPO setup, and compare compute including the teacher prefill.
Reviewer_toSrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper extends OPSD through adaptive reference-solution masking and sparse teacher-distribution interpolation. Reference exposure is determined by rollout–reference overlap, while teacher/student entropy ratios select positions for interpolation toward the student. Experiments cover three Qwen3 model sizes and three mathematical reasoning benchmarks, showing consistent improvements over the matched OPSD baseline.

Critical strengths
  1. The method is simple, integrates naturally into an existing self-distillation pipeline, and introduces no additional inference-time operations.
  2. The comparison with OPSD is reasonably controlled, with improvements across all nine model–benchmark combinations.
  3. The appendix includes ablations on masking ratios, masking positions, and interpolation parameters, supporting the practical usefulness of the overall recipe.
Critical weaknesses
  1. The entropy-ratio criterion has an insufficiently examined failure mode. A high ratio may arise when the student is confidently wrong, rather than when teacher supervision is unreliable. Interpolating toward the student could then weaken useful supervision. The paper does not analyze prediction quality at selected positions or the effects of interpolation on such cases.
  2. The theoretical justification for adaptive masking remains indirect. Theorem 3.2 concerns an idealized mixture, without establishing how its governing quantity relates to the implemented anchor-overlap score. Theorem 3.3 assumes that later masking preserves more informative context. These results provide motivation, but limited justification for the actual masking rule.
  3. Figure 1 does not clearly communicate the method. Key data-flow connections are missing or ambiguous. Some arrows have unclear meanings. Small labels and typography inconsistent with the manuscript further reduce readability.
Questions for authors
  1. How often are selected positions cases where the student is confidently wrong but the teacher predicts correctly? What effect does interpolation have on these positions?
  2. Can the authors clarify the connection between the masking analysis and the implemented overlap-based schedule?
  3. Could Figure 1 be redrawn with explicit connections and legible, consistent typography? If generative AI was used to construct it, please clarify this in the AI-use statement.
Reviewer_rbcVOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper conceptualizes privileged on-policy self-distillation (OPSD) for large language models as an information-allocation proble. In standard OPSD, a student model generates a reasoning trajectory, and a privileged teacher (the same model initialized with frozen parameters) re-scores that exact trajectory with full access to a verified reference solution. However, exposing the full solution uniformly ignores how the teacher's information advantage varies across reasoning prefixes, risking over-committing to a single reference trace. To address this, the authors introduce PAINT (Privileged-context Adaptive INterpolated Training), which dynamically regulates information allocation at two distinct resolutions:

  1. Rollout-Level (Overlap-Adaptive Partial Context): PAINT evaluates a continuous recall-style overlap score (α\alpha) between the student's rollout and a set of reference anchors (like formula-like spans or boxed answers). If the overlap is low, the teacher receives more solution context; if it is high, a contiguous suffix mask hides downstream details to prevent rote trace-imitation.
  2. Token-Level (Sparse Energy Interpolation): After re-scoring, PAINT ranks token positions using a teacher/student entropy ratio (RnR_n) . For a sparse budget (top ftopf_{\text{top}} fraction) where the ratio is highest—meaning the solution context resolved the least relative uncertainty—the teacher's targets are geometrically interpolated slightly toward the student's logits to smooth out distribution mismatches. PAINT is evaluated using Qwen3 (1.7B, 4B, and 8B) backbones on competitive math benchmarks (AIME 2024, AIME 2025, and HMMT 2025) . It consistently outperforms full-solution OPSD and matches or beats GRPO while requiring only a single short rollout per problem instead of multiple long rollouts.
Critical strengths
  • Dual-Resolution Information Control: The conceptual shift from static "full-context" exposure to two-resolution adaptive filtering (rollout-level masking + token-level logit interpolation) represents an elegant paradigm shift for self-distillation architectures.
  • Information-Theoretic Foundation: Grounding the method in conditional mutual information (I(A;Z∣c)I(A; Z | c)) and proving mathematically that geometric calibration contracts the teacher-student forward-KL divergence provides strong theoretical legitimacy.
  • Highly Compute-Efficient Alternative to RL: PAINT achieves competitive accuracy with GRPO while only needing a single student rollout per problem of up to 1,024 tokens, compared to GRPO's typical group layout of eight long trajectories up to 16k tokens each.
  • Preservation of Structural Setups: The mathematical proofs and ablations convincingly support suffix masking over prefix or middle masking, demonstrating that preserving early setup text provides the teacher a coherent framework while hiding downstream shortcuts.
Critical weaknesses
  • Anchor Extraction Dependency: The rollout-level adaptation relies on extracting discrete, normalized reference anchors (such as specific formula formats and numeric spans) . For non-mathematical tasks or soft-reasoning domains (e.g., law, code comment synthesis, philosophy), defining and normalizing these anchors becomes vastly more ambiguous.
  • Hyperparameter Sensitivity across Scales: The optimal configuration relies on tightly budgeted constants (kbase=0.03k_{\text{base}} = 0.03, ftop=0.03f_{\text{top}} = 0.03) . The paper shows that interpolating over a larger token fraction hurts performance, indicating that the method is sensitive to tuning and might require recalibration when applied to alternative model families.
  • Diminishing Returns on Intermediate Scales: While PAINT shows a strong +2.1 point macro-average jump on Qwen3-8B, the performance margin narrows significantly to a +0.8 point gain on the intermediate Qwen3-4B model, suggesting variable stability during scaling.
Questions for authors
  1. Extreme Overlap Boundaries: What happens to the teacher's guidance when a student achieves an overlap score of α=1.0\alpha = 1.0 (perfect structural recall) ? In this scenario, the mask ratio reaches its maximum boundary ρcorrect\rho_{\text{correct}}, but does the heavily masked context still provide any useful corrective signal?
  2. Drift of the Entropy Ratio (RnR_{n}): Because the teacher model is held completely frozen (ϕ=θ0\phi = \theta_0) while the student updates via LoRA, do the teacher/student entropy ratios systematically inflate or skew as training progresses and the student diverges from its initialization?
  3. Extension to Multi-Turn and Code: Since the geometric interpolation acts purely on log odds cancellation and predictive distributions independent of text format, have the authors experimented with adapting PAINT to multi-turn programming benchmarks where intermediate execution feedback can serve as the privileged context?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.