acceptodds
Under review as a conference paper at ICLR 2027

Internalizing the Halt Vector: Full-Vector and On-Axis Reconstruction

Abstract

Can a reasoning model learn earlier stopping by matching only the projection of a steered activation, rather than reconstructing the full vector? We compare full-vector and on-axis reconstruction on DeepSeek-R1-Distill-Qwen-7B, separating reasoning length, mathematical answer correctness and compliance with the requested answer format. An outcome-informed follow-up to an earlier held-out comparison, with development-only selection, evaluated three shortening targets on 2,000 held-out problems. One target established matched shortening within one percentage point: full-vector and on-axis reconstruction reduced thinking by 23.56% and 23.39%, respectively. Their accuracy was equivalent within the prespecified 2.6-point margin under Math-Verify, while a boxed-answer grader favored full-vector reconstruction by 2.80 points. The on-axis model produced more closed, unboxed answers despite an explicit boxing instruction. A post-hoc, model-assessed audit supports a predominantly formatting-based explanation of this discrepancy, subject to assessor uncertainty and parsing sensitivity. Matching remained unresolved at the other two targets. These results do not establish that full-vector reconstruction is necessary for internalizing halting, nor that on-axis training is free of instruction-following costs. They show why comparisons of internalization objectives should distinguish achieved shortening, answer correctness and format compliance. The evidence is conditional on selected checkpoints from one model and does not establish training-seed robustness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning models, efficient reasoning, activation steering, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AR1LOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to internalize a halt vector in reasoning models and compares two training objectives: full-vector reconstruction and on-axis reconstruction. The authors conduct three comparative studies on DeepSeek-R1-Distill-Qwen-7B, seeking to match the achieved reductions in reasoning length before evaluating differences in answer accuracy and format compliance. At the one target where matched shortening is established, the two methods perform similarly under Math-Verify but differ under a grader that requires a boxed final answer. The paper therefore concludes that the available evidence does not establish full-vector reconstruction as necessary for internalizing halting.

Critical strengths
  1. The evaluation is carefully designed, treating matched reductions in reasoning length as a condition for comparison and using different graders to assess answer quality. 
  2. The experimental evaluation is fairly extensive, covering multiple comparative studies, large held-out test sets, and supplementary analyses. The paper also reports uncertainty intervals and unresolved results.
Critical weaknesses
  1. The paper primarily compares existing types of training objectives but does not establish a clear conclusion to guide the choice between them. Although it identifies differences in answer-format compliance, it neither explains the underlying mechanism nor clarifies the respective advantages and applicable conditions of the two objectives, limiting the practical value of the findings. 
  2. Math-Verify can extract answers from the entire completion, including the reasoning process, whereas the other grader requires a specific final-answer format. The two graders therefore differ in more than their formatting requirements. Although the authors conduct an additional output audit, their assessment of answer correctness relies mainly on post-hoc model-based judgments without human-annotated ground truth. Consequently, the effects of answer correctness and format compliance are not sufficiently disentangled. 
  3. The core comparison is largely restricted to a single model, mathematical reasoning tasks, and a fixed answer-format requirement. Whether the findings extend to other model architectures, reasoning tasks, or broader settings of behavior internalization remains to be validated.

4.The overall quality of the figures needs substantial improvement.

Questions for authors

See Weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer AR1L

Thank you for the review. On your points:

1. Guidance on choosing an objective. Short answer: for boxed-answer math trained with the language-model loss, keep the off-axis penalty (full-vector training) unless an alternative has been validated in the intended setting. Dropping the off-axis penalty cost 4.49 points of boxed-format compliance on Qwen-7B, and on Llama-8B on-axis training was lower on both accuracy and compliance. On-axis training without the language-model loss was a supported alternative on the retest's MATH panel, in a comparison against full-vector training with that loss that changes objective and loss together (Sections 5, 8 and 10). When both objectives were trained without the language-model loss, their compliance difference was 0.37 pp [−0.16, 0.90] at equal shortening on local curves, with Math-Verify accuracy also equivalent (Section 5); this supports the loss-dependent recommendation, not a general advantage for on-axis training. Section 8 also places budget forcing, distillation and a concise prompt next to the two objectives.

2. Grader differences and human ground truth. Grading only the stated final answer, with the same extraction and Math-Verify checker for both objectives, gives −0.12 [−0.62, 0.38] for Study 3's matched pair and 0.56 [0.15, 0.98] for the 20-seed retest (full-vector minus on-axis), both within the 2.6-point margin without extracting answers from the reasoning (Section 3; Supplement P.4). The retest also measures format compliance directly (Section 5). Blinded, post hoc human labels by non-author readers give −0.37 [−1.76, +1.31] on 2,000 matched-pair outputs and +0.27 [−2.14, +1.88] on 1,600 retest outputs (Section 6, Table 3). Both are within the margin under the registered intervals but not under conservative bounds, so they support the correctness reading without settling it.

3. Mechanism. Intervening on activations at one checkpoint and layer, the change along the halt direction carries the shortening and the off-axis change carries the compliance drop (compliance scored with a looser window rule). A fixed-strength penalty test raised whole-output compliance by 9.8 pp per unit weight [7.6, 12.0] (Section 5). This locates the effect but does not explain why the objectives differ: it does not show that drift causes the drop, and the penalty test did not hold shortening equal.

4. One model, math, one format. We added Llama-8B with its own steering direction, a final-answer-line format, fresh DeepMath problems, and MMLU-Pro, OlympiadBench and IFEval without retraining (Sections 5 and 7). On Llama-8B, on-axis training was lower on both measures (Math-Verify accuracy by 3.12 points, compliance by 2.13), so equivalence is not shown and broader generality is still open. That test used its own steering direction, built after halt-direction training failed to shorten its thinking, so it is not a repeat of the same intervention (Section 7).

5. Figures. Figure 1 is new (accuracy and compliance against shortening for both objectives and both models), and Study 3's figure is redrawn as Figure 2 with a self-contained caption.

Thank you for the detailed response and the additional experiments. They have addressed most of my concerns and significantly strengthened the paper. 

I still have one question regarding the role of the LM loss. The results show that the format-compliance gap between full-vector and on-axis reconstruction largely disappears when this loss is removed. Moreover, on-axis training without the loss provides a viable alternative to full-vector training with it at matched shortening. Given these findings, what role is the LM loss intended to play in behavior internalization, and what practical considerations would justify retaining it rather than omitting it?

Thank you for the follow-up.

Intended role. The language-model (text) loss trains the adapted model on its own teacher completions alongside the activation target, as is usual for activation losses in internalization (Abstract, Section 2). Under on-axis training it is the only loss constraining the off-axis components (Section 2, Eq. 1).

What our results show. At equal strength, keeping it was associated with lower compliance and less shortening for both objectives, and with slightly higher full-vector Math-Verify accuracy (0.59 pp) at that smaller shortening (Appendix D, Table 22). This does not establish a benefit of the text loss at matched shortening.

Keeping or omitting it. A practical reason to keep it is continuity with the recipe evaluated in Study 3, the 20-seed retest and Llama-8B (Figure 1). Omitting it with on-axis training is reasonable for boxed-answer mathematics like the retest's MATH panel, where that recipe was equivalent within 2.6 pp to full-vector training with the loss in both Math-Verify accuracy and compliance (Section 5, Table 2). That evidence covers one model and one panel, and the comparison changes objective and loss together, so elsewhere we would validate it first (Sections 8 and 10).

Thank you for the detailed response. Your explanations have addressed my remaining concerns, and I am willing to increase my score to 3.

I also encourage the authors to further improve the quality and readability of the figures and tables.

Reviewer_oRGdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits whether internalizing a halt-vector intervention requires full-vector reconstruction or whether an on-axis reconstruction objective is sufficient. It compares the two objectives through three studies, with the final study selecting configurations intended to achieve comparable reductions in reasoning length. One of three target pairs satisfies the prespecified held-out matching criterion. At that point, the objectives are equivalent within the specified accuracy margin under Math-Verify, whereas the legacy grader favors full-vector reconstruction. Additional analyses investigate answer-format differences and limited training refits.

The paper provides a carefully qualified reassessment of an existing claim, but its main finding remains narrow and substantially dependent on the evaluation procedure.

Critical strengths

The comparison addresses an important confound. Comparing objectives at approximately matched reasoning lengths is more informative than comparing configurations with different amounts of shortening. This is a useful improvement in experimental design.

The paper reports uncertainty and limitations transparently. It distinguishes established equivalence from unresolved comparisons and discloses grader disagreements and inconclusive refits. These practices make the reported evidence easier to assess.

Critical weaknesses

The contribution rests on a narrow result whose reproducibility across training runs remains unestablished. Only one of the three target pairs satisfies the held-out matching criterion, and the principal result concerns one backbone, intervention layer, and task setting. The additional refits do not establish matching because their uncertainty is too large; they should therefore be treated as inconclusive rather than as evidence of either robustness or failure. Although this result can qualify a universal necessity claim, the paper does not establish how reliably the alternative objective works or when its behavior should be expected. With no new objective and no validated mechanism, I find the demonstrated contribution insufficiently substantial for acceptance.

The main interpretation depends on an unresolved measurement issue. At the matched point, Math-Verify reports an accuracy difference of approximately −0.04 percentage points, while the legacy grader favors full-vector reconstruction by approximately 2.80 points. The graders differ in both answer extraction and equivalence checking, so this disagreement cannot be attributed to answer formatting alone without further controls. The post-hoc model-assessed audit is informative, but it does not provide independent human validation of final-answer correctness. Consequently, the evidence does not yet cleanly distinguish preserved task performance from changes in answer presentation or grader behavior. This uncertainty directly affects the interpretation of the central result.

Questions for authors

Can the matched comparison be reproduced across independently trained runs with enough evaluation precision to assess the prespecified matching criterion? In particular, does the conclusion persist when the selection procedure itself is repeated?

Can both objectives be evaluated using the same final-answer extraction procedure and the same mathematical equivalence checker, with a blinded human audit of remaining disagreements? This would help separate formatting effects from differences in answer correctness.

Can the trained adapters, halt direction, and per-output predictions and scores be released to permit independent rescoring of the central comparison?

1 comment
AuthorsOpenReview ↗

Response to Reviewer oRGd

Thank you for the detailed review and the concrete questions. In brief: (Q1) partly. Twenty seeds per objective give a seed-averaged matched result for fixed recipes with shared traces and direction, not an independent replication; of three selection repeats, one reproduced the conclusion as planned, one only through a decode-acceptance amendment made after one output's decode had been inspected, and one did not reproduce matching. (Q2) Yes, with the limits below. (Q3) Released (link below).

Contribution. The claim is narrower than in the submission and, we think, clearer. On Qwen-7B the choice of objective is mainly a format-compliance choice, and the accompanying language-model loss controls that cost: when both objectives are trained without it, compliance and Math-Verify accuracy are equivalent at equal shortening on local curves (Section 5). For evaluating such methods more broadly, a box-requiring grader can report a format cost as lost accuracy, so correctness and format compliance should be measured separately (Section 1, contribution (i)). We add no new objective, and the refits remain inconclusive, as the paper says; the 20-seed retest establishes a fixed-recipe, seed-averaged result, repeated selection remains less secure, and Llama-8B, with its own direction, did not show the equivalence (Section 7).

Independent runs and a repeated selection. Across 20 new training seeds per objective, seed-averaged shortening is matched and Math-Verify accuracy differs by 0.57 points [0.16, 0.98], within the 2.6-point margin (Section 4). This tests retraining variability with shared teacher traces and halt direction (Table 1), so it is not an independent replication. The selection repeats used the same panels and are descriptive counts (Section 3). On rebuilt inputs (new problems, teacher traces and halt directions), a refit found Math-Verify accuracy equivalent and full-vector compliance higher, but at unmatched shortening (Section 7).

Same extraction and checker, and a human audit. A post hoc regrade applies the same answer extraction and the Math-Verify checker to both objectives: −0.12 [−0.62, 0.38] at the matched pair (Section 3). Blinded non-author audits labelled 2,000 matched-pair outputs (one reader) and 1,600 retest outputs (two readers), giving −0.37 [−1.76, +1.31] and +0.27 [−2.14, +1.88]: within the margin under intervals registered before each audit's own labels were collected, though the conservative bounds in Table 3 exceed it, so the audits support but do not settle correctness (Section 6). On the remaining disagreements themselves (closed, unboxed outputs credited by Math-Verify alone), one author's blind labels also found an answer given and correct in 40 of 40 on-axis and 39 of 40 full-vector outputs (Supplement O.4; one annotator, a selected stratum).

Release. A second anonymous repository (https://anonymous.4open.science/r/halt-internalization-release) holds the matched pair's adapters and halt direction, all 96,000 held-out outputs of that pair and the base model with scores, and a rescoring script; its README links 551 further trained adapters. The supplementary archive adds per-output rows for Study 3, the retest, and the per-level and Llama-8B studies, with code that recomputes their abstract and Table 2 values (except the retest's human check, whose inputs are in the repository). The remaining artifacts follow with the final version.

Only one matched target. A re-evaluation of the 12.5% pair on new draws of the same problems establishes matching there, with Math-Verify −0.38 [−0.63, −0.12], favoring on-axis (Section 4, Appendix A). It is outcome-informed, and the original 12.5% and 17.5% cells stay unresolved.

When the alternative works, and mechanism. With the language-model loss, on-axis compliance is 4.49 points [3.76, 5.22] lower; with both objectives trained without that loss, their compliance is equivalent at equal shortening, read off local curves (Section 5). A local intervention locates the compliance drop, scored with a looser window rule, in the off-axis change at one checkpoint and layer, without showing that drift causes it. Section 8 turns this into a conditional guide.

Reviewer_mksBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies two approaches to train a model to stop thinking early: full-vector and on-axis. It conducts three studies. Study 1 gave both methods an equal number of tries. Study 2 tried to find versions of each method that shortened by the same amount, so accuracy could be compared fairly. The on-axis method gets the math right just as often, but follows formatting instructions less reliably.

Critical strengths
  1. The paper's evaluation is rigorous and transparent. It made choices based on practice data only. The paper fixed the rules before testing, used two graders, and reported every null result.

  2. The paper separates correctness from formatting, e.g., on-axis training weakens adherence to the requested answer format but not the mathematical accuracy. This is useful for people training models to steer activations and challenges the necessity of full-vector reconstruction.

Critical weaknesses
  1. The paper didn't answer the main question. Matched shortening was shown at one target, on one seed and one model, and even there the finding depends on the grader. Off-axis drift is what differentiates the two objectives and the probable cause of the formatting decline, but it was never measured on the selected checkpoints.

  2. The paper's test has not enough statistical power. The matching test is underpowered: at 2,000 problems, a match can be confirmed only if the cuts differ by under about 0.3 points. Seed-to-seed variation is as large as the effects being compared.

  3. The writing is too dense for the reader to follow

Questions for authors

Please see the weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer mksB

Thank you for the review. You were right that the submitted version left the main question open. Direct answer: on Qwen-7B, dropping the off-axis penalty barely changes Math-Verify accuracy (equivalent within the 2.6-point margin across 20 seeds, with a small, nonzero 0.57-point full-vector edge), but with the language-model loss it costs 4.49 points of boxed-format compliance. On Llama-8B, which used a different steering direction, on-axis training was lower on both, and the paper now recommends keeping the penalty when training with the language-model loss, unless an alternative is validated in the intended setting. On each point:

One target, one seed, one model. The 20-seed retest at matched shortening finds Math-Verify accuracy equivalent within the margin, with a small full-vector edge of 0.57 [0.16, 0.98] (Section 4). A second model and fresh DeepMath problems are in Section 7. Two caveats: the retest and Study 3 share teacher traces and the halt direction (Table 1), so they are not independent replications, and on Llama-8B on-axis training was lower on both measures.

Grader dependence. Grading only the stated final answer, with the same extraction for both objectives, gives −0.12 [−0.62, 0.38]; almost all of the original grader gap comes from answer extraction (Section 3, Appendix C). Blinded, post hoc human checks keep the matched pair and the retest within the margin under their registered intervals, though not under more conservative bounds (Section 6).

Off-axis drift. It is now measured on the selected checkpoints: larger under on-axis training at Study 3's matched pair (full-vector minus on-axis: −7.80 [−7.96, −7.65], raw activation units) and across eight seeds, although its association with compliance within a recipe is unresolved (Appendix E). Intervening at one checkpoint and layer (compliance scored with a looser window rule), the halt direction carries the shortening and the off-axis change the compliance drop; a fixed-strength penalty test raised whole-output compliance by 9.8 pp per unit weight [7.6, 12.0] (Section 5). Neither shows that drift causes the drop, and the penalty test did not hold shortening equal.

Statistical power. Agreed for Study 3: with 2,000 problems a match can be certified only if the observed difference is below about 0.2 to 0.3 points, and two of its targets stay unresolved (Section 3). You were also right about seed variation: across the earlier seeded studies, the estimated standard deviation of the shortening difference between one adapter per objective ranged from 0.98 to 2.58 points, so a match between single checkpoints is fragile (Appendix B; Supplement P.5, Table S43). The retest therefore certifies a match between seed-averaged cuts (20.71% against 20.65%), not between every trained pair, and its accuracy intervals include seed variation (Section 4).

Density. Each results section now opens with its question and answer, Table 1 defines the terms, and Table 2 holds the central results. We agree the paper is still dense. For a short check of your concerns: Table 2 and Section 4 for seed-averaged matching, Section 3 for shared-extraction grading, Appendix E for measured drift, and Section 6 and Table 3 for the human audits and their limits.

Studies 1 to 3 and their registered results are unchanged. The added studies were designed after the submitted results; Appendix Tables 36 and 39 give their status and amendments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.