- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies implicit chain-of-thought distillation through supervision targets, positions, and alignment spaces. It proposes ALUS, which matches teacher and student local state updates at all valid positions in a fixed low-dimensional subspace constructed from training-answer token output weights. The paper analyzes error aggregation and fixed linear readouts, and reports 41.17% accuracy on GPT-2/GSM8K, an 8.31-point gain over no alignment, with additional evaluations on Pythia-1B, ProsQA, and numerical transfer tasks.
Critical strengths
- The comparisons keep the student architecture fixed, preserve per-example aggregation weights, and calibrate initial alignment-gradient scales. Rank-matched random, teacher-PCA, and vocabulary-source controls help isolate the effect of supervision-space selection.
- ALUS improves accuracy across three seeds, with additional evidence from another backbone and a nonnumeric task. Loss ablations support the contribution of alignment beyond reference-text training, and the method adds no inference-time module.
Critical weaknesses
- The subspace is constructed from training-answer tokens, but their coverage of test answers and performance at different coverage levels are not reported. How to choose the subspace for other tasks, how that choice affects generalization, and how a rank-matched full-vocabulary basis performs remain untested.
- The K=3/6 comparison changes both student computation and teacher segmentation, without analyzing performance by difficulty. It remains unclear how K should scale with reasoning complexity or whether increasing K helps harder problems.
- With K=6, Local does not outperform State/Cumulative; on ProsQA, the answer subspace outperforms the random subspace in only one of three seeds. The conditions under which these choices help remain unexplained.
Questions for authors
Please address the concerns raised in W1–W3 above.
2 comments
Response to Reviewer H4rV (2/2)
Response to W2
Thank you for pointing this out. We capped the number of teacher spans at three and supervised only nonempty spans. Under this limit, doubling student updates from three to six changed mean GSM8K accuracy from 41.17±0.20% to 41.14±0.74%. The 4–5-reference-step questions show why the distinction matters. Relative to the same three-update student under the three-span limit, six updates gained 1.98 points under the six-span limit and changed accuracy by −1.20 points under the three-span limit. Extra student updates alone therefore do not explain the original subgroup gain.
Three-span limit. During the rebuttal period, we trained the six-update student to take two updates per span and align their combined change to one teacher target, keeping one supervision point per nonempty span. Across three paired seeds, the mean change over all 1,319 questions was −0.03 percentage points. Table D3 gives the overall and length-stratified accuracies.
Reference-solution length. At fixed segmentation, the changes for questions with 1–3, 4–5 and at least six reference steps were +1.10, −1.20 and −1.56 points. On the 150 questions with the longest reference solutions, accuracy fell from 11.33% to 9.78%. Longer reference solutions thus gave no observed advantage to the larger student budget in this control.
Conclusion. Under the three-span limit on GSM8K, K=3 is the more economical choice. For a new task, we would compare candidate K values on representative validation questions at fixed segmentation before assigning extra updates to longer problems.
Accuracy by reference-solution length (Table D3). Each configuration is labeled as student updates / maximum teacher spans. Brackets give the change from 3/3 in percentage points, computed before rounding.
| Reference steps | Questions | 3/3 (%) | 6/6 (%) [Δ pp] | 6/3 (%) [Δ pp] |
|---|---|---|---|---|
| 1–3 | 697 | 58.92 | 58.35 [−0.57] | 60.02 [+1.10] |
| 4–5 | 472 | 24.44 | 26.41 [+1.98] | 23.23 [−1.20] |
| ≥6 | 150 | 11.33 | 11.11 [−0.22] | 9.78 [−1.56] |
Response to W3
Thank you for your question. The controlled comparisons point to two different conditions. The benefit of a local update target depends on the step configuration; the benefit of answer directions on GSM8K depends strongly on whether supervision reaches intermediate steps.
Local targets. On three-update GSM8K, Local exceeds State and Cumulative by 2.20 and 2.00 points. Shifting the same teacher updates to different steps costs 7.38 points, which shows the value of assigning each change to its corresponding span. With six updates and finer segmentation, the three targets are within 0.43 points. In our rebuttal diagnostic (seed 11, 200 updates), the Local alignment and answer gradients approach orthogonality at K=3 (cosine −0.127 to +0.001), but become more opposed at K=6 (−0.042 to −0.095). We then tested whether removing the conflicting gradient component helped at K=6. After 50 further updates, answer cross-entropy at step 250 was 2.5306 with the conflict removed and 2.5297 with the original gradient. Removing the conflict gave no short-run improvement on this trajectory.
Answer directions. With Local targets and rank 32 fixed on GSM8K, switching from random to answer directions gains 5.89 points at K=3 and 7.81 at K=6 when all valid updates are supervised. With supervision only at the last valid update, the differences are −0.99 and −0.10. On ProsQA, where each question supplies two candidates, the answer basis averages 89.53% versus 88.40% for random and is ahead in one of three seeds.
Conclusion. The clearest evidence for Local comes from ordered, three-step supervision. For answer directions, the numerical comparisons at both K values point to a different condition: the selected directions are useful when they guide intermediate computations throughout the solution. On ProsQA the incremental contribution of direction selection is smaller.
Response to Reviewer H4rV (1/2)
Response to W1
Thank you for your question. With space-prefixed GPT-2 tokenization, our GSM8K coverage analysis found all 1,462 test-answer token occurrences covered across all 1,319 questions at full retention. In our rebuttal experiments, retaining even half the vocabulary beats the non-answer control in every seed. The rank-32 basis built from all GPT-2 tokens scores 35.38% on GSM8K, compared with 41.17% for the training-answer basis.
Coverage and accuracy. During the rebuttal period we trained students with nested subsets of the training-answer vocabulary at rank 32. The 50% and 75% subsets covered 47.47% and 71.27% of test-answer token occurrences, respectively. Complete-answer coverage rose from 44.96% to 69.37%, while mean accuracy stayed near 40%. Even the 50% group exceeded the non-answer control in all three seeds. For the 50% subset, mean accuracy across three seeds was 42.66% on the 593 questions whose answers were fully covered and 37.92% on the other 726. These descriptive groups may differ in answer length and difficulty.
Construction on another task. For a new task, we collect token types from its training answers, center their pretrained output-weight rows, and freeze the top 32 right singular vectors before student training. We applied this rule to symbolic reasoning on ProsQA. Its training vocabulary covers all 38 test answer concepts across 500 questions. ALUS achieves 89.53% accuracy, an 8.07 percentage-point gain over the student trained without alignment across three paired seeds.
Full-vocabulary basis. When a task-specific answer vocabulary is difficult to specify, the same construction can use all 50,257 pretrained GPT-2 tokens. In the GSM8K experiment conducted during the rebuttal, this rank-32 basis yields 460, 486 and 454 correct answers out of 1,319 for seeds 11, 12 and 13 (35.38±1.29%). The submitted rank-32 answer basis scores 41.17±0.20% and leads in all three seeds, by 5.79 percentage points on average. On ProsQA, in a separate rebuttal experiment, this full-vocabulary basis achieves 88.33±0.76% at rank 32, compared with 89.53±1.33% for the answer basis. The full-vocabulary run used a different loss aggregation and validation frequency from the submitted answer-basis run. Under their respective training protocols, the answer basis led in all three seeds, by 1.20 percentage points on average.
Transfer without retraining. The GSM8K-trained model also transfers to new numeric question distributions without further training. ALUS improves over the no-alignment student by 7.87 points on SVAMP, 6.88 on ASDiv, and 22.03 on MultiArith (revised Table D9).
Conclusion. On GSM8K, even 50% answer-type retention yields higher mean accuracy than either the non-answer control or the rank-matched full-vocabulary basis. For a new task, the ProsQA results support using training-answer tokens when available and show that a rank-matched full-vocabulary basis is a viable alternative. The numeric transfer tests further show gains on new question distributions without retraining.
GSM8K test accuracy by training-answer token retention (mean ± SD over three seeds).
ProsQA vocabulary comparison (Table D8).