- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to internalize a halt vector in reasoning models and compares two training objectives: full-vector reconstruction and on-axis reconstruction. The authors conduct three comparative studies on DeepSeek-R1-Distill-Qwen-7B, seeking to match the achieved reductions in reasoning length before evaluating differences in answer accuracy and format compliance. At the one target where matched shortening is established, the two methods perform similarly under Math-Verify but differ under a grader that requires a boxed final answer. The paper therefore concludes that the available evidence does not establish full-vector reconstruction as necessary for internalizing halting.
Critical strengths
- The evaluation is carefully designed, treating matched reductions in reasoning length as a condition for comparison and using different graders to assess answer quality.
- The experimental evaluation is fairly extensive, covering multiple comparative studies, large held-out test sets, and supplementary analyses. The paper also reports uncertainty intervals and unresolved results.
Critical weaknesses
- The paper primarily compares existing types of training objectives but does not establish a clear conclusion to guide the choice between them. Although it identifies differences in answer-format compliance, it neither explains the underlying mechanism nor clarifies the respective advantages and applicable conditions of the two objectives, limiting the practical value of the findings.
- Math-Verify can extract answers from the entire completion, including the reasoning process, whereas the other grader requires a specific final-answer format. The two graders therefore differ in more than their formatting requirements. Although the authors conduct an additional output audit, their assessment of answer correctness relies mainly on post-hoc model-based judgments without human-annotated ground truth. Consequently, the effects of answer correctness and format compliance are not sufficiently disentangled.
- The core comparison is largely restricted to a single model, mathematical reasoning tasks, and a fixed answer-format requirement. Whether the findings extend to other model architectures, reasoning tasks, or broader settings of behavior internalization remains to be validated.
4.The overall quality of the figures needs substantial improvement.
Questions for authors
See Weakness
4 comments
Thank you for the detailed response and the additional experiments. They have addressed most of my concerns and significantly strengthened the paper.
I still have one question regarding the role of the LM loss. The results show that the format-compliance gap between full-vector and on-axis reconstruction largely disappears when this loss is removed. Moreover, on-axis training without the loss provides a viable alternative to full-vector training with it at matched shortening. Given these findings, what role is the LM loss intended to play in behavior internalization, and what practical considerations would justify retaining it rather than omitting it?
Thank you for the follow-up.
Intended role. The language-model (text) loss trains the adapted model on its own teacher completions alongside the activation target, as is usual for activation losses in internalization (Abstract, Section 2). Under on-axis training it is the only loss constraining the off-axis components (Section 2, Eq. 1).
What our results show. At equal strength, keeping it was associated with lower compliance and less shortening for both objectives, and with slightly higher full-vector Math-Verify accuracy (0.59 pp) at that smaller shortening (Appendix D, Table 22). This does not establish a benefit of the text loss at matched shortening.
Keeping or omitting it. A practical reason to keep it is continuity with the recipe evaluated in Study 3, the 20-seed retest and Llama-8B (Figure 1). Omitting it with on-axis training is reasonable for boxed-answer mathematics like the retest's MATH panel, where that recipe was equivalent within 2.6 pp to full-vector training with the loss in both Math-Verify accuracy and compliance (Section 5, Table 2). That evidence covers one model and one panel, and the comparison changes objective and loss together, so elsewhere we would validate it first (Sections 8 and 10).
Thank you for the detailed response. Your explanations have addressed my remaining concerns, and I am willing to increase my score to 3.
I also encourage the authors to further improve the quality and readability of the figures and tables.
Response to Reviewer AR1L
Thank you for the review. On your points:
1. Guidance on choosing an objective. Short answer: for boxed-answer math trained with the language-model loss, keep the off-axis penalty (full-vector training) unless an alternative has been validated in the intended setting. Dropping the off-axis penalty cost 4.49 points of boxed-format compliance on Qwen-7B, and on Llama-8B on-axis training was lower on both accuracy and compliance. On-axis training without the language-model loss was a supported alternative on the retest's MATH panel, in a comparison against full-vector training with that loss that changes objective and loss together (Sections 5, 8 and 10). When both objectives were trained without the language-model loss, their compliance difference was 0.37 pp [−0.16, 0.90] at equal shortening on local curves, with Math-Verify accuracy also equivalent (Section 5); this supports the loss-dependent recommendation, not a general advantage for on-axis training. Section 8 also places budget forcing, distillation and a concise prompt next to the two objectives.
2. Grader differences and human ground truth. Grading only the stated final answer, with the same extraction and Math-Verify checker for both objectives, gives −0.12 [−0.62, 0.38] for Study 3's matched pair and 0.56 [0.15, 0.98] for the 20-seed retest (full-vector minus on-axis), both within the 2.6-point margin without extracting answers from the reasoning (Section 3; Supplement P.4). The retest also measures format compliance directly (Section 5). Blinded, post hoc human labels by non-author readers give −0.37 [−1.76, +1.31] on 2,000 matched-pair outputs and +0.27 [−2.14, +1.88] on 1,600 retest outputs (Section 6, Table 3). Both are within the margin under the registered intervals but not under conservative bounds, so they support the correctness reading without settling it.
3. Mechanism. Intervening on activations at one checkpoint and layer, the change along the halt direction carries the shortening and the off-axis change carries the compliance drop (compliance scored with a looser window rule). A fixed-strength penalty test raised whole-output compliance by 9.8 pp per unit weight [7.6, 12.0] (Section 5). This locates the effect but does not explain why the objectives differ: it does not show that drift causes the drop, and the penalty test did not hold shortening equal.
4. One model, math, one format. We added Llama-8B with its own steering direction, a final-answer-line format, fresh DeepMath problems, and MMLU-Pro, OlympiadBench and IFEval without retraining (Sections 5 and 7). On Llama-8B, on-axis training was lower on both measures (Math-Verify accuracy by 3.12 points, compliance by 2.13), so equivalence is not shown and broader generality is still open. That test used its own steering direction, built after halt-direction training failed to shorten its thinking, so it is not a repeat of the same intervention (Section 7).
5. Figures. Figure 1 is new (accuracy and compliance against shortening for both objectives and both models), and Study 3's figure is redrawn as Figure 2 with a self-contained caption.