acceptodds
Under review as a conference paper at ICLR 2027

HAIR: Fine-Strand Hidden-State Anchored Implicit Reasoning with Single-Prefill Inference

Abstract

Chain-of-thought (CoT) reasoning is accurate but slow, because every intermediate step is decoded as text. Latent reasoning moves these steps into hidden states, but without dense step-level supervision it stays below CoT beyond 1B parameters, it never commits an intermediate result, and it is compared with CoT baselines that differ in trace format, objective and decoding at once. We present HAIR, a streaming reasoning framework that commits each intermediate result as a one-line textual checkpoint, such as G1=G0*2=18, and places a block of b latent positions before each checkpoint in one append-only KV cache. Goal-boundary distillation (GBD) anchors the hidden state at each checkpoint to a shared-weight explicit path that reads the teacher's rationale for that step. Because the latent blocks and GBD can be removed while the checkpoint stream, data and decoding stay the same, HAIR also tests whether latent computation adds anything beyond the checkpoints. On GSM8K with Qwen3-1.7B, HAIR retains 97.5–99.1% of CoT accuracy at 42–59% of its latency from 18k training records, while CODI and Coconut trained on the same records score 30 or more points lower. The checkpoints carry this accuracy: latent blocks and GBD change it by less than the seed-to-seed spread, although GBD lowers its loss by three orders of magnitude. The savings carry over to SVAMP, GSM-Hard and MATH.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent reasoning, implicit chain-of-thought, semantic checkpoints, hidden-state distillation, single-prefill inference, inference efficiency, causal language models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_L6ZAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HAIR interleaves latent blocks with compact textual checkpoints and applies goal-boundary distillation from a shared-weight explicit reasoning path. Experiments show substantial inference savings over full CoT. Paired ablations indicate that the checkpoint-only variant preserves most accuracy, while latent blocks and distillation provide no consistent improvement.

Critical strengths
  1. The paired ablations and multi-seed experiments provide useful evidence on the effects of latent blocks and distillation.
  2. The paper is clearly written, with well-organized figures and tables that make the method and results easy to follow.
Critical weaknesses
  1. The checkpoints constitute compact explicit CoT, while latent blocks and GBD provide no consistent benefit. Without comparisons against standard equation-only or compressed-CoT training, the substantive contribution remains unclear.
  2. The low CODI (18.95%) and Coconut (28.28%) results rely on single-seed runs without reported tuning or convergence evidence. Inconsistent training descriptions between Section 4.1 and Appendix G further weaken confidence in the claimed 30-point advantage.
Questions for authors
  1. Could the authors compare HAIR with standard equation-only or compressed-CoT training, including checkpoint-only SFT without the auxiliary rationale objective?
Reviewer_Quw4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

HAIR alternates short latent blocks with textual checkpoints that record intermediate results. Goal-boundary distillation aligns latent-path hidden states with those of an explicit reasoning path during training, while inference uses a single prefill and a shared, growing KV cache. Experiments use Qwen3-1.7B on GSM8K, SVAMP, GSM-Hard, and MATH. On GSM8K, the default configuration achieves 58.43% accuracy versus 59.93% for explicit CoT, with 49.5% lower measured latency. The most informative finding is that latent blocks and distillation provide no consistent accuracy improvement: the checkpoint-only variant achieves 59.14% accuracy at approximately 42% of CoT’s latency.

Critical strengths

Informative ablations, including a useful negative result. Comparisons preserve the checkpoint stream, training records, backbone, and decoding while varying latent blocks and distillation. Multiple training seeds and paired tests support the conclusion that neither addition consistently improves accuracy in the evaluated setting. The paper reports this result clearly despite its implications for the proposed architecture.

Efficiency gains are measured directly. Same-GPU latency measurements use warm-up, three counterbalanced rounds, and a common inference setup. The checkpoint-only variant preserves most of CoT’s accuracy while substantially reducing decoding and latency, providing a concrete practical result.

Critical weaknesses

The large advantage over external latent-reasoning baselines is less securely established than the internal ablations. CODI and Coconut are each evaluated with one training seed and one reported configuration. Appendix G documents their adaptations and training schedules, but provides no convergence or hyperparameter-sensitivity analysis. The reported gap therefore establishes an advantage over these particular implementations and settings; its robustness remains unclear. This matters because the conclusion prominently highlights an advantage of at least 30 percentage points.

Questions for authors

How were the CODI and Coconut training configurations selected, and what evidence indicates that their performance is not primarily limited by optimization or adaptation to Qwen3-1.7B? Would additional seeds or a small validation-based tuning study preserve the reported gap?

What accuracy does the checkpoint-only variant achieve without the auxiliary explicit-rationale objective LauxL_{\mathrm{aux}}? The paper correctly notes that its comparison with answer-only changes both checkpoints and rationale supervision; this control would clarify which training components are necessary for the practical benefit.

Reviewer_iuoiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper seems to introduce a latent reasoning method, where the model reasons in something that the authors refer to as "checkpoints". They then use distillation to learn the latent reasoning with b steps. Results in Table 1 suggest that no reasoning (b=0) works on par or even better than the proposed reasoning method.

Critical strengths

Could not identify

Critical weaknesses

The paper is very hard to read. It uses paper-specific terms and never defines them and a very dense language. I could not determine what the actual method was or what was supposed to be new. This makes it impossible to assess the paper’s novelty and/or relevance.

Questions for authors

None

1 comment

Dear Reviewer iuoi,

Could you read the responses from the authors and post your opinions after rebuttal? The authors think they may resolve your concerns and your comments should provide more details. You can post your idea about this paper. Any discussion is welcome.

Thanks,
AC

Reviewer_rQafOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HAIR finetunes Qwen3-1.7B-Base to solve math problems by writing one compact textual checkpoint per step instead of the usual CoT in natural language, with a block of continuous latent positions before each checkpoint in a KV cache. They introduce a novel training objective GBD, building on CODI applied at textual boundaries. They test on GSM8k and MATH with multiple seeds and seem to show a significant improvement with regards to latency compared to explicit CoT while maintaining accuracy. They claim to be the first to show latent CoT improvements for models over a billion parameters.

Critical strengths
  1. Their main results are very strong showing HAIR practically matching explicit CoT (in table 1) on GSM8k with significantly fewer deocded tokens and half the latency of explicit CoT, surpassing CODI and Coconut

  2. They test quite rigorously on multiple seeds over the augmented GSM-8k datasets and the ablations are also very detailed.

Critical weaknesses
  1. The results in Table 1 are promising but the comparison is skewed since only 1 seed has been tested for CODI and Coconut as compared to 3-6 seeds for the HAIR variations. Also Answer-only which is direct no CoT response is reported as having an accuracy of 20.32% which means both CODI and Coconut fall below it, likely since they were distilled from full natural-language rationales while HAIR is trained on its own compact checkpoint traces. In table 2, CODI and Coconut numbers from their own papers both improve significantly once they are put in the appropriate regime, so HAIR does not have a fair comparison here.

  2. In the ablations, the best number (98.7% of CoT at 42% latency) belongs to HAIR b=0, which contains neither component mentioned in contribtions 1 and 2 (latent blocks and hidden state anchoring), so neither contribution seems to have a significant impact on the final results.

Questions for authors
  1. Can CODI and Coconut be trained on the checkpoint traces with matched epochs and seeds and then added as a fair baseline?
  2. Could an equation only cot baseline be added so the contribution of the checkpoint format can be isolated?
Reviewer_bp7GOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces HAIR, which interleaves latent computation with compact textual checkpoints. It achieves comparable accuracy with CoT with lower inference latency, while ablations find no consistent accuracy benefit from the latent blocks or distillation.

Critical strengths
  1. Compact textual checkpoints preserve near-CoT accuracy while substantially reducing inference latency and making intermediate results inspectable.
  2. Ablations isolate the contributions of latent blocks and GBD, providing evidence that neither consistently improves accuracy beyond the checkpoint-based control.
Critical weaknesses
  1. The checkpoints already encode operations and intermediate results, effectively forming a compact explicit CoT that may leave limited room for latent computation to contribute. Moreover, the �0��0� control retains auxiliary rationale supervision, so its performance cannot be attributed to checkpoints alone. A checkpoint-only SFT baseline without �1��1�, together with ablations of checkpoint content, would clarify what sustains accuracy and when additional latent computation becomes useful.
  2. The evaluation uses 1.7B backbone and is restricted to mathematical reasoning, limiting benefits of latent computation generalize to other model scales and domains.
Questions for authors

See weakness

Comments on the paper

2 comments
Reviewer_Quw4OpenReview ↗

Title: Stronger evidence, remaining novelty and generalization concerns

Thank you for the detailed rebuttal and the substantial additional experiments. The CODI and Coconut experiments across three seeds, learning-rate sensitivity analysis, and convergence results address my main methodological concerns, though Coconut's best-validation checkpoint substantially narrows the gap relative to its final latent stage. Auxiliary rationale supervision has little effect (59.19 ± 1.53% without it versus 59.14 ± 1.12% with it). My novelty and generalization reservations remain: gains mainly reflect compact explicit equations, latent blocks and GBD add no consistent benefit, and the numeric equation-only baseline performs similarly to the proposed checkpoint format. I view the contribution primarily as efficient, structured CoT compression and an analysis of latent reasoning's limitations. I maintain my recommendation of 3 (Weak Accept).

Thank you for the follow-up and for maintaining your positive recommendation! We agree with your reading of the contribution. Below we briefly clarify three places where our intended meaning differs slightly from the summary.

1. Coconut's best-validation checkpoint We agree that this checkpoint narrows the gap. Even so, it remains about 12 points below checkpoint-only on average (47.41% against 59.19%). As noted in our first response, it is the first latent stage, where only the first step is latent and the later steps are still written as text. Once all three steps are latent, validation accuracy falls from 83.4% to 55.2% (Table 22), so Coconut's accuracy follows how much of the chain is still written, which is the same pattern we find inside HAIR.

2. G1=9*2=18 is a form of our checkpoint The numeric equation-only row (G1=9*2=18, Table 8) is one of two forms of the same HAIR checkpoint, and we will label it more clearly in the paper. G1=G0*2=18 and G1=9*2=18 use the same teacher goals and the same training, and they differ only in whether an earlier result is written as G0 or as 9. The ablation below changes what each step writes, one element at a time:

Each step writesExampleAccuracy (%)Decoded tokensLatency (ms)
Nothing (answer only)–20.32––
Value onlyG1=1837.88 ± 1.0324.11,008
Value only, with latent blocksG1=1838.44 ± 2.3623.81,250
Checkpoint, reference formG1=G0*2=1859.94 ± 1.5142.41,642
Checkpoint, numeric formG1=9*2=1860.70 ± 0.9543.71,791
TokenSkipcompressed CoT59.62 ± 2.5898.73,764
Explicit CoTrationale and expression61.28 ± 1.76108.83,860

GSM8K test, Table 8.

The table shows three things:

  • The two checkpoint forms score alike, so the result does not depend on the notation.
  • The expression is necessary. Writing only the value costs 22.06 points (Table 5), and latent blocks do not recover this loss.
  • The rationale is not necessary. Checkpoints stay within 1.34 points of explicit CoT with 61% fewer decoded tokens. At the accuracy of TokenSkip, they decode 57% fewer tokens with 56% lower latency.

3. Generalization We agree that this is the main open point. On Qwen3-4B, checkpoint-only keeps 93.1% of CoT accuracy at 44% of its latency, which leaves a 5.43-point gap (Table 2). The gap comes mostly from long problems: it is 2.6 points on questions with one or two goals and 13.1 points on questions with five or more (Table 23). Such long problems make up fewer than 3% of our training records (Appendix B).

Beyond arithmetic, on MATH-500 a checkpoint is a short statement rather than an equation, and HAIR_aux stays within 0.6 points of CoT with 45.5% fewer positions (seed 11, Table 4). Our evidence remains mainly in arithmetic and mathematics, because the checkpoint design relies on steps whose results can be checked. We state this scope in the Limitations paragraph.

Thank you again for your careful review and support, and please let us know if you have any further questions:)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.