- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Self-Policy Distillation (SPD), which extracts a KV gradient subspace by leveraging a loss function linked to the correctness of a small set of labeled samples. It then intervenes in the model’s generation process via projection onto this subspace and utilizes the resulting generated data for self-distillation. The paper addresses a practically significant question: how to construct more valuable self-training data while reducing reliance on large-scale validation sets and external teacher models.
Critical strengths
-
SPD uses a small labeled calibration set to construct a reusable intervention that guides subsequent generation. This offers a promising way to extend the benefit of limited correctness supervision without requiring an external teacher or correctness filtering of every generated response.
-
The experiments demonstrate overall improvements across code, mathematics, and question answering. Comparisons with direct fine-tuning and self-distillation baselines, together with subspace ablations, provide meaningful support for the approach. The observed cross-domain transfer also suggests potential beyond narrowly matching the calibration task.
Critical weaknesses
-
Although the proposed combination is meaningful, comparisons with simpler activation-based steering methods under matched supervision, generation, and training budgets would better establish the distinctive value of the KV gradient subspace.
-
The results support the usefulness of correctness-sensitive subspaces, but do not conclusively establish that these subspaces isolate capability from style or other correlated factors. The ablations strengthen the empirical case for the design; the broader claim of capability separation should nevertheless be presented more cautiously.
Questions for authors
Please refer to weaknesses.
I thank the authors for their response and the additional results on Llama-3.1-8B-Instruct and BBH. While these additions are helpful, several key questions regarding experimental details and mechanism attribution remain unresolved:
Missing Details in the Updated Evaluation (W1 & Q1): The new 5-shot CoT results on Llama-3.1-8B-Instruct are encouraging, as they bring the baseline in line with standard expectations. However, the response still omits the exact prompt templates, chat template settings, and answer-parsing rules requested in Q1. It is also unclear whether the entire SPD pipeline (calibration, hooked generation, and LoRA SFT) was re-run under this setting, or only the final evaluation, and whether similar gains hold on the larger backbones (e.g., Qwen2.5-14B-Instruct) under standard evaluation.
Validity and Details of the Length Truncation Control (W2, Q2 & Q3): First, the rebuttal does not specify which model backbone or truncation criteria (token- vs. character-level) were used for the 50%/70% truncation table. More importantly, naive length truncation is not an appropriate control for Q3: in math and coding tasks, cutting off the last 30%–50% of a sequence frequently removes the final numeric answer or
returnstatement, resulting in incomplete training targets. Our concern—also illustrated by the James Runs Sprints example in Appendix E, where the base model computes the correct math answer but hallucinates follow-up questions due to plain-text prompting—is whether the gains primarily come from suppressing post-answer continuation noise and standardizing output format. A fair baseline would involve semantic-preserving brevity controls (e.g., brevity system prompts or proper stopping criteria) rather than hard truncation.Lack of Empirical Validation for Scaling Hypotheses (W3): The two hypotheses proposed to explain the diminishing returns on larger models (fixed data size and a smaller proportion of hooked layers in deeper models) are reasonable, but remain unverified without supporting experiments (e.g., scaling the number of hooked layers on the 7B/14B models).
Overall, I appreciate the authors' efforts during the discussion period, but given the remaining ambiguities in experimental setup and mechanism attribution, I maintain my current score.