acceptodds
Under review as a conference paper at ICLR 2027

Self-Policy Distillation via Capability-Selective Subspace Projection

Abstract

Self-distillation bootstraps large language models (LLMs) by training on their own generations. However, existing methods either rely on external signals to curate self-generated outputs (e.g., correctness filtering, execution feedback, and reward search), which are costly and unavailable for the best-performing frontier models, or skip curation entirely and train on all raw outputs, an approach that is often domain-specific and hard to generalize. Both also share a deeper weakness that self-generated outputs entangle task-relevant capability with others, such as stylistic patterns, formatting artifacts, and model-specific errors, diluting the signal for the specific capability one aims to improve. In this paper, we propose **Self-Policy Distillation (SPD)**, which achieves generalizable, capability selective with only minimal external signal (<5% labeled data in each domain). Specifically, SPD uses extracts a low-rank capability subspace from the model's own gradients on correctness-defining tokens, projects key-value (KV) activations into this subspace during self-generation, and fine-tunes on the resulting raw outputs with standard next-token prediction loss. Through extensive experiments across code generation, mathematical reasoning, and multiple-choice QA, we show that SPD achieves up to 13% improvement over state-of-the-art self-distillation methods and up to 16.3% improvement over pre-trained baselines. Notably, SPD demonstrates superior generalizability, achieving 15.7% better performance under out-of-domain generalization settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-distillation, Representation learning, Subspace projection, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t7mdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Policy Distillation (SPD), which extracts a KV gradient subspace by leveraging a loss function linked to the correctness of a small set of labeled samples. It then intervenes in the model’s generation process via projection onto this subspace and utilizes the resulting generated data for self-distillation. The paper addresses a practically significant question: how to construct more valuable self-training data while reducing reliance on large-scale validation sets and external teacher models.

Critical strengths
  1. SPD uses a small labeled calibration set to construct a reusable intervention that guides subsequent generation. This offers a promising way to extend the benefit of limited correctness supervision without requiring an external teacher or correctness filtering of every generated response.

  2. The experiments demonstrate overall improvements across code, mathematics, and question answering. Comparisons with direct fine-tuning and self-distillation baselines, together with subspace ablations, provide meaningful support for the approach. The observed cross-domain transfer also suggests potential beyond narrowly matching the calibration task.

Critical weaknesses
  1. Although the proposed combination is meaningful, comparisons with simpler activation-based steering methods under matched supervision, generation, and training budgets would better establish the distinctive value of the KV gradient subspace.

  2. The results support the usefulness of correctness-sensitive subspaces, but do not conclusively establish that these subspaces isolate capability from style or other correlated factors. The ablations strengthen the empirical case for the design; the broader claim of capability separation should nevertheless be presented more cautiously.

Questions for authors

Please refer to weaknesses.

Reviewer_vvwrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Self-Policy Distillation (SPD), a self-distillation framework that steers self-generation via low-rank representation hooks. Specifically, the method computes gradients of a correctness-aligned cross-entropy loss restricted to key answer spans on a small calibration set (<5% labeled samples), applies SVD to key and value activation gradients to extract a low-rank subspace, and projects KV activations onto this subspace during generation. The resulting outputs form an uncurated training corpus for LoRA-based next-token prediction fine-tuning. Experiments across five dense open-source backbones up to 14B parameters demonstrate gains over base models and two uncurated self-distillation baselines (PSR and SSD) across code generation, arithmetic reasoning, and question answering.

Critical strengths
  1. Targeted formulation of representation control for self-distillation: Identifying capability-aligned gradient directions specifically from correctness tokens to construct activation projection hooks is well-motivated and mathematically clear.
  2. Comprehensive exploratory controls: The inclusion of rank-matched random subspace controls (Bottom, Orthogonal, and Shuffle variants) and direct fine-tuning on the calibration set (Direct SFT) provides useful validation that the projection direction matters compared to arbitrary projection.
  3. Low inference overhead: Restricting the projection to a low-rank operation on key and value states at selected layers adds negligible computational cost to decoding (+0.57% runtime overhead), preserving training throughput.
Critical weaknesses
  1. Severe baseline performance collapse and non-standard evaluation: The reported base model numbers diverge significantly from published benchmarks and official model reports. For instance, Qwen2.5-14B-Instruct achieves 57.0% on GSM8K in Table 1 compared to its official score of ~94.8%, and Llama-3.1-8B-Instruct achieves 39.0% on GSM8K and 28.4% on MBPP compared to established community baselines exceeding 80% and 70% respectively. This discrepancy indicates the absence of standard few-shot chain-of-thought prompting, proper chat templates, or standard extraction parsers. Demonstrating performance gains on a severely degraded base model fails to establish whether the proposed method provides utility on properly evaluated models. Furthermore, evaluating on only 100 test examples for GSM8K and SVAMP yields standard errors of 3% to 5%, which exceeds the reported marginal improvements of 1% to 3% across larger models, undermining statistical significance.
  2. Disconnect between generation quality and fine-tuning gains: The empirical premise of the paper is that projection hooks produce superior capability-aligned self-supervision. However, Table 3 demonstrates that SPD-generated outputs underperform base model generations on MBPP (16.3% vs. 17.0%) and CodeAlpaca NLL (0.705 vs. 0.683). The subsequent downstream gain on MBPP after fine-tuning (27.0% in Table 4) contradicts the claim that downstream improvements stem from higher-quality generated supervision. Figure 3 indicates that hooked generation primarily removes verbose text, comments, and extra assertions. The current evaluation lacks controls to separate genuine reasoning capability extraction from surface-level output length compression and format standardization.
  3. Scaling behavior contradicts the core motivation: The introduction motivates the framework by targeting frontier regimes where verifiers, labels, and reward models become inaccessible. However, empirical evaluation is restricted to small dense models (≤14B), and the observed improvements decay sharply with scale: average cross-domain gains drop from +9.5% on 0.5B to +3.6% on 14B and +3.3% on Llama-3.1-8B. The abstract highlights the peak absolute gain of 16.3 percentage points observed exclusively on the smallest 0.5B model on SVAMP, obscuring the diminishing returns at larger scales.
  4. Factual contradictions and methodological ambiguities: Line 358 claims that SPD consistently outperforms the base model, PSR, and SSD across all backbones, yet Table 1 shows that PSR outperforms SPD on MBPP for Qwen2.5-0.5B-Instruct (28.7% vs. 27.0%), and Table 13 shows OPSD outperforms SPD on SVAMP (34.7% vs. 32.3%). Additionally, the paper contains naming discrepancies across sections, listing "Qwen3.5-4B" in Table 7 and Table 16 while citing "Qwen3-4B-Instruct" in Table 1, and evaluates only an arbitrary 6-subtask subset of BBH without providing inclusion criteria.
Questions for authors
  1. Please provide the exact generation prompts, system templates, and answer-parsing routines used for GSM8K and MBPP evaluation, and report results on the complete standard test sets (1,319 problems for GSM8K and 500 for MBPP) under standard evaluation harnesses.
  2. What accounts for the discrepancy in Table 3 where SPD generates lower-accuracy solutions than the base model on MBPP, yet fine-tuning on these same samples yields the highest downstream accuracy?
  3. What happens to downstream performance if the base model is fine-tuned on outputs regularized purely by length truncation or standard brevity system prompts, without KV subspace projection?
6 comments

I thank the authors for their response and the additional results on Llama-3.1-8B-Instruct and BBH. While these additions are helpful, several key questions regarding experimental details and mechanism attribution remain unresolved:

  1. Missing Details in the Updated Evaluation (W1 & Q1): The new 5-shot CoT results on Llama-3.1-8B-Instruct are encouraging, as they bring the baseline in line with standard expectations. However, the response still omits the exact prompt templates, chat template settings, and answer-parsing rules requested in Q1. It is also unclear whether the entire SPD pipeline (calibration, hooked generation, and LoRA SFT) was re-run under this setting, or only the final evaluation, and whether similar gains hold on the larger backbones (e.g., Qwen2.5-14B-Instruct) under standard evaluation.

  2. Validity and Details of the Length Truncation Control (W2, Q2 & Q3): First, the rebuttal does not specify which model backbone or truncation criteria (token- vs. character-level) were used for the 50%/70% truncation table. More importantly, naive length truncation is not an appropriate control for Q3: in math and coding tasks, cutting off the last 30%–50% of a sequence frequently removes the final numeric answer or return statement, resulting in incomplete training targets. Our concern—also illustrated by the James Runs Sprints example in Appendix E, where the base model computes the correct math answer but hallucinates follow-up questions due to plain-text prompting—is whether the gains primarily come from suppressing post-answer continuation noise and standardizing output format. A fair baseline would involve semantic-preserving brevity controls (e.g., brevity system prompts or proper stopping criteria) rather than hard truncation.

  3. Lack of Empirical Validation for Scaling Hypotheses (W3): The two hypotheses proposed to explain the diminishing returns on larger models (fixed data size and a smaller proportion of hooked layers in deeper models) are reasonable, but remain unverified without supporting experiments (e.g., scaling the number of hooked layers on the 7B/14B models).

Overall, I appreciate the authors' efforts during the discussion period, but given the remaining ambiguities in experimental setup and mechanism attribution, I maintain my current score.

  1. Missing Details in the Updated Evaluation (W1 & Q1)

We thank the reviewer for raising this further question. The new 5-shot CoT results on Llama-3.1-8B-Instruct already incorporate the following exact prompt templates, chat-template settings, and answer-parsing rules throughout the entire SPD pipeline. Additionally, we conduct another experiment in which only the final evaluation stage uses the standard evaluation protocol, while the preceding SPD pipeline remains unchanged using Llama-3.1-8B-Instruct.

MethodGSM8K ↑SVAMP ↑
Base model83.8%71.5%
Final-stage Rerun85.9%73.5%
Full SPD Rerun87.9%74.2%

Additionally, we conduct a further experiment using Qwen2.5-14B-Instruct. The exact prompt templates, chat-template settings, and answer-parsing rules used in this experiment are also provided below. The performance gains of SPD are consistently observed on Qwen2.5-14B-Instruct as well.

MethodGSM8K ↑SVAMP ↑
Base model92.0%91.5%
SPD94.3%96.4%

Exact prompt templates. For the additional 5-shot CoT experiments with Llama-3.1-8B-Instruct, we use five fixed demonstrations for each task and prepend the same five demonstrations to every evaluation example. For GSM8K and SVAMP, each demonstration follows:

Question: {demonstration_question}
Answer: {demonstration_reasoning_and_answer}

and the target example is formatted as:

Question: {target_question}
Answer:

For MBPP, we use:

Instruction: {problem}
Your code should pass these tests:
{test_cases}
Code:

For CodeAlpaca, we use:

Instruction: {instruction}
Code:

Chat-template settings.( Llama-3.1-Instruct) We use the official Llama-3.1-Instruct chat template provided by Hugging Face Transformers through tokenizer.apply_chat_template(..., add_generation_prompt=True). Messages are represented using the standard system, user, and assistant roles. The same chat-template configuration is used for the base model and all compared methods.

Chat-template settings (Qwen2.5-Instruct). We use the official Qwen2.5-Instruct chat template provided through Hugging Face Transformers via tokenizer.apply_chat_template(..., add_generation_prompt=True). Following the official Qwen2.5 usage example, the messages are constructed as:

system: You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

user: {the complete 5-shot task prompt shown above}

The model then generates the response from the assistant turn. We use the same chat-template configuration for the base model and all compared methods.

During SPD calibration, system-prompt and demonstration tokens are excluded from gradient extraction, and gradients are collected only from the designated answer region of the current calibration example.

Answer-parsing rules. For GSM8K, we follow the standard evaluation harness and extract the final numerical answer associated with the #### marker. Commas, currency symbols, content preceding the #### marker, and a trailing period are ignored before exact-match comparison. Consistent with the standard evaluation harness, generation is deterministic, with do_sample=False and temperature=0.0.

For SVAMP, we instruct the model to conclude with The final answer is: {answer} and extract the numerical value from this final-answer field. The extracted value is normalized and compared with the gold answer using exact match.

For MBPP, following the standard execution-based evaluation protocol, we extract the generated Python code and execute it against the provided test cases. A sample is counted as correct only when the generated program successfully passes the corresponding tests, and we report Pass@1.

For CodeAlpaca, which does not provide executable test cases or a standard generation-based answer parser, we follow our evaluation implementation and report the mean token-level negative log-likelihood of the reference code completion conditioned on the instruction prompt.

  1. Validity and Details of the Length Truncation Control (W2, Q2 & Q3)
MethodGSM8K ↑SVAMP ↑MMLU ↑BBH ↑
Brevity system prompts20.0%23.0%46.5%33.3%
SPD23.0%32.3%49.7%38.1%

We thank the reviewer for raising this point. For the previous truncation table, we use Qwen2.5-0.5B-Instruct and perform token-level truncation. We additionally conduct a comparison between SPD and the brevity system prompt baseline using Qwen2.5-0.5B-Instruct. The corresponding prompts used for the two methods are listed below.

Brevity system prompt

System:
Provide a concise response while preserving all task-relevant reasoning and information necessary to solve the problem correctly.

User:
Question: {question}
Answer:

SPD

User:
Question: {question}
Answer:

We apply the same standard calibration, hooked generation, and LoRA SFT procedures to both methods. Additionally, for the Brevity system prompt baseline, to ensure a fair comparison during calibration, we only extract gradients corresponding to the question and answer tokens, while excluding gradients corresponding to the system prompt tokens. The results are shown above. SPD outperforms the Brevity system prompt baseline, indicating that its gains cannot be explained simply by suppressing post-answer continuation noise or standardizing the output format.

  1. Lack of Empirical Validation for Scaling Hypotheses (W3)
MethodGSM8K ↑SVAMP ↑
Base model57.0%77.0%
2 layers(24,48)60.0%84.0%
4 layers (23,24,47,48)66.7%88.5%
6 layers (22,23,24,46,47,48)67.3%90.7%

We thank the reviewer for further raising this point. Extending our discussion on the proportion of hooked layers, we conduct an additional experiment to investigate the relationship between the number of hooked layers and model accuracy using Qwen2.5-14B-Instruct. The results are shown above. As the number of hooked layers increases from 2 to 4, the accuracy improves substantially, whereas increasing the number from 4 to 6 leads to only a modest additional improvement. This suggests that the scaling behavior is influenced by multiple factors rather than simply increasing monotonically with the number of hooked layers.

Qwen2.5-14B-Instruct contains a total of 48 Transformer layers. During training, we use only a simple prompt format:

User:
Question: {question}
Answer:

Reviewer_AXcBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Self-Policy Distillation (SPD). It uses a small labeled calibration set to compute gradients on correctness-defining tokens, extracts low-rank subspaces of attention key/value activations via SVD, and projects those activations during self-generation. The hooks are then removed, and the original model is LoRA-fine-tuned on its generated completions. The contribution is to use limited labels to steer generation without an external teacher or per-output filtering. Experiments span code, math, QA, and multiple backbones. For example, on Qwen2.5-0.5B, GSM8K rises from 11.0% to 23.0% and SVAMP from 16.0% to 32.3%; the paper also includes loss and subspace ablations.

Critical strengths

Strength 1: Novel integration of activation steering with self-distillation, creating a capability-selective paradigm. Existing self-distillation faces a fundamental dilemma: accept noisy self-generated data (e.g., SSD) or rely on costly external signals (verifiers, reward models, reference answers as in OPSD). SPD resolves this via internal representation-level steering — requiring only <5% labeled data, with no external teacher, reward model.

Strength 2: Correctness-span formalism provides interpretability-to-performance alignment. SPD's correctness-span design provides a crisp answer: only tokens directly determining task success guide subspace extraction (assertion spans for code, final numeric answer for math, option letter for QA). This reflects a principled hypothesis about task capability encoding in KV space rather than mere engineering. Ablations confirm this criticality: full-sequence loss underperforms correctness-aligned loss, random subspaces fail.

Critical weaknesses

Weakness 1: Benchmark selection is outdated and entirely omits open-ended tasks, undermining the paper's empirical claims of generality. The evaluation suite — HumanEval/MBPP, GSM8K/SVAMP, MMLU/BBH — was already approaching saturation by 2025. At the 2026 time point, a method paper claiming cross-domain generality cannot credibly omit LiveCodeBench or SWE-bench for code, AIME or MATH for math, and GPQA or HLE for QA. The 1–2 point gains reported on MMLU (where 14B models already reach 72%+) are insufficient to establish meaningful improvement. Furthermore, the paper provides zero validation on open-ended subjective tasks (creative writing, summarization, dialogue), despite positioning SPD as a general capability-improvement framework.

Weakness 2: Correctness-defining spans require task-specific manual specification, fundamentally constraining the method's applicability and contradicting its "minimal external signal" narrative. What SPD requires is strong annotation: a human must not only label whether an answer is correct, but also demarcate precisely which token span determines correctness. This annotation burden is qualitatively heavier than simple correctness labels and is entirely unavailable for open-ended tasks.

Questions for authors
  1. In the OOD experiments (e.g., training on GSM8K calibration set → generating on SVAMP → evaluating), how was the calibration set built?

  2. The paper asserts spans are "correctness-defining" — for example, assertion-relevant tokens in code generation. How were these spans delimited in practice (automated heuristics, manual annotation, or rule-based extraction)? what quality control was applied?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.