acceptodds
Under review as a conference paper at ICLR 2027

Latent Reasoning Works: Recovering Chain-of-Thought Behaviour with Latent Thoughts

Abstract

Chain of thought (CoT) improves LLM reasoning, but requires a long, autoregressively generated trace, which is subsequently discarded: its value lies in its effect on the distribution over final answers; latent reasoning seeks that effect without the cost. Given the limited success of existing approaches, which do not achieve near-CoT performance at increased efficiency, we ask a more foundational question: do short latent contexts that provide near-CoT performance even exist? We demonstrate that they do: just a single latent token on HumanEval+ recovers 98% of CoT performance and 256 latent tokens even exceeds it on MATH-Hard; by contrast, the median CoT lengths on these datasets are over 1000 and 3000, respectively. We use knowledge distillation, distilling the effect of many CoT rollouts on answers into the latent context in a frozen LLM. Validation loss tracks training loss, indicating that the latents carry some concept of reasoning, not just memorisation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent reasoning, knowledge distillation, chain-of-thought, prompt tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 6 reviews, above 51% of the others.
Reviewer_sU7ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is an existence study, which aims to answer whether latent thoughts can replicate CoT performance/behavior. It optimizes question-specific latent tokens by knowledge distillation from plenty CoT rollouts. Experiments with Qwen3-1.7B/8B show that just one latent token can restore 98% CoT accuracy on HumanEval+, and even exceed it on MATH-Hard with 256 latent tokens. Held-out rollout diagnostics and GSM-Symbolic augmentation experiments further show that latent thoughts can provide performance gains on unseen augmentations.

Critical strengths
  1. Recovering near-CoT accuracy with short learned latent tokens provides useful evidence about the capacity of latent reasoning, might be a meaningful reference for future latent generation research.
  2. Evaluation goes beyond accuracy. Held-out teacher rollouts, output-overlap diagnostics, and unseen augmentations strengthen the evidence that the results extend beyond memorising individual training responses.
Critical weaknesses
  1. Overstatement on failure mode diagnostics using similarity

Section 4.2 claimed that by comparing cross-similarity with self-similarity, we can diagnose failure modes, but it doesn't strictly hold.

Consider distributions over the same two answers: T=(0.8,0.2),S=(0.6,0.4)T=(0.8,0.2), S=(0.6,0.4). Their self-similarities are 0.68 and 0.52, and cross-similarity is 0.56, sitting between them, but the student added no new answers. This is contradictory to the statement in line 367.

Similarly, with T=(0.6,0.3,0.1),S=(0.3,0.6,0.1)T=(0.6,0.3,0.1), S=(0.3,0.6,0.1), both self-similarities are 0.46, while cross-similarity is 0.37, below both, but the student didn't replace teacher answers with new ones. This is contradictory to the row 2 in Table 4.2.

  1. Accuracy gain on MATH-Hard weakened by some evidence

In line 315 and 475, the paper highlights that on MATH-Hard, there is a strong upward trend as kk increases, and CoT accuracy is recovered with 64 latent tokens, and even exceeded with 256. But in Table D.3, with k=256k = 256, the student drops 1.2pts for 1.7B and only gets 0.3pts increase for 8B. This weakens the claimed superiority and should feature prominently.

I am willing to raise my score once any of the above are properly addressed.

Questions for authors

See weaknesses.

2 comments

Response to Reviewer sU7R

We thank the reviewer for their time and effort.

For each key point/question raised, we first describe what we're trying to get across in the relevant section, then what we'll change.

We hope that these changes satisfy your concerns. But, if they do not, please do raise this.

Overstatement of Failure-mode Diagnostics (W1)

This is a fair concern. Our intent in Section/Table 4.2 is to highlight 'typical' errors, and to demonstrate what effect that error would have. By not observing that effect, we rule out the error.

But, you are completely correct: a different error may produce the same effect. We (inadvertently) phrased an "if" as an "if and only if".

We'll rectify this - eg, at line 367, as you point out, but also 351.

Removed Rollouts (W2)

Two reviewers mentioned removed rollouts. We respond to this point in a shared comment. In short, we'll soften the phrasing to ensure that there is no superiority or over-claim.

Thanks for the clarification. This helps address my concerns. I will keep my score.

Reviewer_vqB2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study whether short latent contexts can recover CoT performance in frozen Qwen3 models. They optimize question-specific latent vectors using KL-based knowledge distillation to match the teacher's answer distribution. The experiments show near-CoT accuracy on HumanEval+ and MATH-Hard, with much shorter latent contexts than the original reasoning traces. GSM-Symbolic evaluates transfer to unseen variations of the same question template. The paper establishes the existence of useful short latents; efficient generation of these latents remains future work.

Critical strengths
  1. The paper recovers substantial CoT accuracy gains without changing the model weights. This provides a controlled demonstration that frozen models can use very short continuous contexts to recover much of the benefit of long reasoning traces.

  2. The augmentation study shows that the same trained latent sequence can improve accuracy on unseen numerical and entity variations of a question template. This demonstrates transfer beyond the exact training instances.

Critical weaknesses
  1. Limited novelty and scope. The main experiments optimize a separate latent sequence for each question using teacher rollouts. Section 5 extends this to unseen variations of the same template, but does not demonstrate reuse across different templates. I understand that this is an existence study. Still, I am not convinced that matching CoT answer distributions with question-specific latents is a sufficiently substantial advance over existing soft-prompt and context-distillation methods. This limits the significance of the contribution for me.

  2. The conclusions go beyond what the tests show. A small training/validation gap supports limited overfitting to the sampled answers, but does not establish recovery of the teacher's full answer distribution. The similarity measures also do not establish the coverage claimed in Section 4.2. Section 5 shows within-template transfer, but does not determine whether the latents encode a reasoning procedure or compressed worked examples. The authors acknowledge this alternative. The claim of exceeding CoT performance also needs qualification: Table D.3 shows that, after excluding teacher timeouts, the MATH-Hard student underperforms the 1.7B teacher and exceeds the 8B teacher by only 0.3 percentage points.

Questions for authors
  1. Generalization: Is there evidence that the same trained latent sequence generalizes to different question templates without being retrained? Section 5 demonstrates only transfer to unseen variations of the same template. This limited reuse is my main concern about the contribution.

  2. Evidence: What evidence shows that the student recovers the teacher's answer distribution, beyond a small training/validation gap and the reported similarity statistics? Please clarify what these tests establish and what remains uncertain.

  3. Output length: Does the number of generated answer tokens change between the CoT teacher and latent student? Please report the comparison for different latent lengths, particularly on MATH-Hard, where the answer includes a worked solution.

5 comments

Response to Reviewer vqB2

We thank the reviewer for their carefully written review. Precise statements like they have given are appreciated by authors, as they allow a very directed response.

We respond to each of the raised points below, including what changes we propose making where appropriate. We sincerely hope that these address the raised concerns. If they do not, then please do raise this - we are very open to a dialogue.

Reasoning/Generalisation vs Memorisation (W1/Q1)

The latents do not transfer between questions - nor should we necessarily expect them to, since they only see the one question. In practice, the latent reasoning would be generated per prompt.

We see this as analogous to sampling a CoT zCoTz_\text{CoT} to question qq, then running free-generation from prompt (q′,zCoT)(q', z_\text{CoT}), for q′≠qq' \ne q, and evaluating against q′q''s verifier; unsurprisingly, this performs poorly.

Question–Latent Swap

This isn't just a conjecture. Since the reviews came in, we ran an experiment to test this, on MATH-Hard:

  • Concatenate a question and a different question's latent
  • Evaluate against both questions' verifiers

In short, the latent doesn't encode the answer in a form that the model can simply read off:

  • The model tries to answer the original question, and 'mismatched' latent slightly harms accuracy
  • When the question is removed, performance collapses

Other reviewers raised similar questions. We provide more details in a shared comment.

That said, experiments on GSM-Symbolic, referenced below, do demonstrate that some transferability is possible.

GSM-Symbolic

Whilst latents do not transfer between questions, §5 demonstrates that some transferability is possible:

  • The latents are evaluated on unseen prompts, of varying diversity
  • Even a single prompt (a=1a = 1) closes approximately half the CoT-induced gap (Figures 5.1 and D.7)
  • Simply encoding the answers seen during training doesn't naturally explain this performance transfer

Other reviewers raised similar points. We provide more discussion in a shared comment, including outlining further experiments that we'll run.

Existing Methods

Several authors raised related points around prompt tuning and the like. So, we address these in a unified manner in a shared comment.

In short, soft-prompt and context-distillation methods have a different objective:

  • Theirs is to represent a single piece of text or objective
  • Ours is to represent the effect of CoT on the answer distribution

More details are in the shared comment. The second section, "Other Existing Methods and Bit Counts", is most relevant to your point, but the others are not irrelevant.

We'll certainly explain this more clearly when revising the paper.

Evidence Behind Conclusions (W2/Q2)

Recovery of Distribution

The loss directly trains zz to reproduce the teacher distribution, but, you are right: it doesn't recover the full answer distribution. This is one reason why sample 2048 answers, to cover as much of the mass as possible. Rather, what we can judge is how closely the characteristics of importance match.

Each test addresses a specific question:

  • Accuracy:
    • This is the primary metric of interest
    • It checks that reduction in teacher-forced loss translates to increase in free-generation accuracy
  • Held-out loss:
    • This checks that the latent hasn't overfit to the seen data
    • This is a genuine concern with so many parameters
  • Similarities:
    • These can catch failures that accuracy can't
    • Eg, the student could collapse to a few of the teacher's answers and match the accuracy

You're right, though, and we'll reword §4.2 accordingly:

  • Neither held-out loss nor similarities establishes coverage
  • Both are dominated by the teacher's frequent answers

After you raised this, we've checked coverage directly, in both directions, at k=64k = 64.

For model MM ∈ {CoT, latent, no-CoT}:

  • Novel: chance that a new rollout from model MM gives an answer not in the training bank
  • Missed: chance that a new rollout from CoT gives an answer not in a collection of 1024 rollouts from model MM

To avoid notational or formatting differences, we say that answers match if they're the same program (HumanEval+) or final answer (MATH-Hard). Percentages are all "CoT / latent / no-CoT".

DatasetSizeNovelMissed
HumanEval+1.7B9.8% / 11.4% / 53.0%11.8% / 14.6% / 80.3%
HumanEval+8B5.7% / 6.7% / 53.9%7.3% / 9.8% / 82.8%
MATH-Hard1.7B0.8% / 4.7% / 31.3%1.0% / 3.6% / 8.8%
MATH-Hard8B0.2% / 1.9% / 25.1%0.2% / 1.9% / 6.2%

The student is only slightly further from the teacher than the teacher's own samples, whereas the no-CoT backbone misses over 80% of the teacher's answers on HumanEval+, and gives novel answers over 25% of the time.

Reasoning or Answer-Encoding

The GSM-Symbolic results discussed above - particularly, the preliminary results on Base→P1 - provide, in our opinion, strong evidence that the latents aren't merely encoding seen answers.

This, of course, can't be ruled out definitively without some kind of decoder. But, it is difficult to see how this performance could transfer from Base to P1 without some concept of the underlying argument.

Exceeding CoT

Two reviewers mentioned removed rollouts. We respond to this point in a shared comment. In short, we'll soften the phrasing to ensure that there is no superiority or over-claim.

Output Length (Q3)

We've looked into this since your comment, and here are the preliminary results.

DatasetSizeCoT teacherLatent studentNo-CoT backbone
HumanEval+1.7B66.367.7–70.9 (+2–7%)76.9 (+16%)
HumanEval+8B55.956.3–56.8 (+1–2%)57.3 (+3%)
MATH-Hard1.7B700724–771 (+3–10%)1031 (+47%)
MATH-Hard8B709724–745 (+2–5%)1062 (+50%)
  • The latent student has pretty similar-length answers to the CoT teacher, typically get closer as kk increases
  • The no-CoT backbone has markedly longer answer; they also vary more question to question

This is a good, positive observation, that we should have included! We'll provide full details in the revised version

Thank you for the detailed response. The coverage table, the output-length comparison, and the softened claim about exceeding CoT address my concerns on distribution recovery and output length. However, my main concern remains. A latent does not transfer to other questions, which makes the main contribution trivial in my view. Producing each latent also requires the CoT rollouts plus training, so it costs more than simply running CoT. I keep my current score.

Reviewer_eTc6OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether continuous contexts exist that reproduce the effect of a chain of thought (CoT) on a frozen LLM’s answer distribution. For each question, the authors optimize latent vectors by knowledge distillation, so that the frozen LLM can generate the answer conditioned on the question and the continuous thoughts. Across Qwen3-1.7B and Qwen3-8B on HumanEval+ and MATH-Hard, k=1 recovers most of the CoT gap on code, and k=64–256 matches CoT on math.

Critical strengths

The question studied in this paper is critical for latent reasoning LLMs. The evaluation goes beyond accuracy: it also compares the teacher’s and the student’s answer distributions with self- and cross-similarity measures.

Critical weaknesses
  • The paper uses teacher-generated prefixes to optimize the continuous thoughts, so the key conclusion is almost inevitable in this case.
  • In §4, the held-out data consists of new rollouts of the same question. This merely shows that z has learned the distribution of answers for that question. It cannot distinguish between reasoning and memorization of the answer to that question.
  • LRT (Rethinking LLM Reasoning: From Explicit Trajectories to Latent Representations, ICLR 2026) mentions a similar claim and provides a method for deriving Z from Q. The authors should consider discussing the differences between their work and LRT.
Questions for authors

N/A

2 comments

Response to Reviewer eTc6

We thank the reviewer for their comments - particularly directing us towards LRT (Rethinking LLM Reasoning: From Explicit Trajectories to Latent Representations, ICLR 2026), a paper of which we were not aware and is certainly highly related.

We address the three raised points, including proposing changes to help alleviate the reviewer's concerns. Our reading is that the LRT reference is the most important, so we start with that.

LRT Citation (W3)

First, an admission: we were not aware of this paper. We'll certainly include it in our updated version!

On the paper itself, there are definitely similarities, but also key differences:

  • LRT:
    • Dataset: triples {(qi,ri,yi)}i\lbrace (q_i, r_i, y_i) \rbrace_i from OpenR1-Math-220k; reasoning traces rir_i explicitly dropped
    • Goal: recover exact answer yiy_i given question qiq_i at inference time
    • Loss: SFT/NLL loss over sequences; matches tokens only, describing KL as "computationally prohibitive"
  • Us:
    • Dataset: questions {qi}i\lbrace q_i \rbrace_i; self-generated rollouts {{(ri,j,yi,j)}j=12048}i\lbrace \lbrace (r_{i,j}, y_{i,j}) \rbrace_{j=1}^{2048} \rbrace_i and corresponding logits
    • Goal: recover CoT distribution over answers offline
    • Loss: KL between CoT and latent answer-distributions

Their algorithm really distils a specific CoT into a single latent; ours distils the effect of randomly sampling a CoT. Ideally, we'd use a fresh CoT rollout every optimisation step. But, CoT rollouts are expensive, so we amortise the cost over steps (and hyperparameters) by presampling many (2048) upfront.

Their performance is great on GSM8K, but weak, even harmful relative to no-CoT, on AMC/MATH-500:

  • Accuracies relative to no-CoT baseline at k=256k = 256
    BenchmarkQwen3-1.7BQwen3-4BQwen3-8BMean
    AMC0.0%−2.1%+8.7%+2.2%
    MATH-500−7.8%+2.7%+1.5%−1.2%
    GSM8K+15.3%+17.9%+23.2%+18.8%
  • CoT-induced accuracy gap recovery at k=256k = 256 using Qwen3-8B
    BenchmarkNo-CoTLRTCoTGap Recovery
    AMC46.3%50.3%55.8%42.1%
    MATH-50077.1%78.2%90.1%8.8%
    GSM8K73.9%91.0%91.0%100%

Their CoT budget is pretty constrained too, at 4k tokens; Table 5 (§D.1) suggests that the accuracy would increase if the budget were increased.

Contrastingly, we recover the full CoT performance on MATH-Hard and well over 90% of it on HumanEval+, with substantially fewer latent tokens - even k=1k = 1 for HumanEval+.

To be clear, this is no slight on their paper: they're attacking the substantially harder question of test-time generation. The fact that they do well even on one benchmark is impressive. But, the other benchmarks demonstrate that it is far from a solved problem.

In our opinion, though, this only makes the case for separating our existence from generation even stronger.

  • They establish existence on GSM8K, but no gain on MATH-500, in their framework
  • Is that a fundamental restriction of MATH-500, or is it just really hard?

Regardless, it's certainly a highly relevant paper, and we'll cite and discuss it prominently in the revised paper.

Inevitability (W1)

Several reviewers raised related points, so we have addressed them in a unified manner in a shared comment.

In short, we agree that the result is not surprising, but don't feel that it's inevitable. For one, no latent-reasoning method has had success - meaning both performant and short. The lack of success is what motivated us to separate existence from (the harder question of) generation.

Prompt tuning and the like tends to address 'offline' evaluations, such as (teacher-forced) reconstruction of a piece of text; it's not used as an inference-time prompt in the way that latent reasoning would be.

Regardless, multiple reviewers raise this point, so we'll make it clearer in the revised version of the paper.

Reasoning/Generalisation vs Memorisation (W2)

Indeed, our set-up is the usual distribution learning, in which the objective is for zz to learn the distribution of answers to a given question.

Whilst there is no agreed-upon definition of "reasoning", we try to address "generalisation vs memorisation" in §4.2.

  • If zz merely memorises the answers that is sees in training:
    • the held-out loss will be substantially higher than the training loss
    • the student's self-similarity will be higher than the teacher's
  • If zz is able to avoid both of these:
    • it is (going towards) achieving the goal of learning the distribution
    • in some sense, it's encoded the underlying 'principles'

The stronger evidence for lack of memorisation, in our opinion, comes from §5:

  • The latents are evaluated on unseen prompts, of varying diversity
  • Even a single prompt (a=1a = 1) closes approximately half the CoT-induced gap (Figures 5.1 and D.7)

Simply encoding the answers seen during training doesn't fit well this performance transfer, in our opinion. We provide more discussion in a shared comment, including outlining further experiments that we'll run.

We acknowledge that your comment starts "In §4", yet our primary evidence comes from §5. But, the a=1a = 1 case in §5 is exactly the same training set-up as §4; it is only the evaluation that differs.

Regardless, we are happy to soften the phrasing, making it clear that there is no definitive analysis saying, "The latent is doing X." Further, the §4–§5 link is not made clear at all - the two sections currently read very independently. We'll address this in the revised version.

Reviewer_NxXoOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this work the authors probe whether or not chain-of-thought can, through oracle distillation, be squeezed into a small number of latent tokens while recovering most of the chain-of-thought performance. To do this, the authors inject a small number of latent tokens and distill chains-of-thought into a few learnable embedding to minimize KL divergence on the output tokens under a frozen language model backbone. The authors find that this oracle distillation retains performance on HumanEval+ and MATH hard. Excitingly, the model was able to fully answer questions that otherwise would've ran out of context in the parent model, which results sometimes in better-tha-parent performance.

Critical strengths
  • The work represents a promising direction for reducing the cost of chain-of-thought tokens while recovering or improving performance
  • Specifically, teaches the work the reader that full autoregressive chain-of-thought maybe too expensive than needed, and provides a pathway to study compressed structures of chain-of-thought
Critical weaknesses
  • Evaluations are relatively limited in terms of the types of reasoning tasks considered; I find it difficult to believe that certain types of tasks that explicitly require complex sequential reasoning behavior (things that would fall outside of perfect TC0TC0 class) can be solved through simple compression. It would be very exciting to be proven wrong in this regard, however, and evidence suggests it maybe possible through the randomised nature of LLM sampling.
  • While section 5 studies generalization across problem templates, length generalization is not studied which may exactly be the tasks that require longer CoTs. Particularly, I'd love to see cases where learned chains of thoughts can extrapolate to problems that require more steps (or even less!) than seen in training.
  • The evaluation for section 4.3 is a bit misleading. Authors uses "n% of recovered performance", but the denominator and numerator isn't quite the same setting: for the student model the latent tokens are directly optimized through KL distillation and provided during inference, while for the teacher model the CoT is not provided and the model has to autoregressively generate the whole thing. This does not detract from the main contribution of the work, however, but confounds the interpretation of the performance.
Questions for authors
  1. Has authors tried any tasks that require length generalization or even tasks that require sequential solving such as S4S4 permutations? This would elide the majority of my concerns regarding the evaluation setting.
  2. Adding on to above, how do the authors expect performance to vary across datasets with different properties, such as those that require sequential reasoning, more or less verbalization of chain-of-thought, and various levels of state-tracking, etc.?
  3. Have the authors investigated the relationship between the CoT being distilled and the quality of distillation? As in, over longer chain-of-thought contexts, must the number of distilled tokens change accordingly or do the CoT tokens act as a form of steering that simply primes the distribution of the model as authors highlighted.
1 comment

Response to Reviewer NxXo

We thank the reviewer for their review.

We hope that our responses address their concerns. If any concerns remain, please do raise these. We are very open to having a dialogue.

Different Tasks and Dataset (W1/Q1/Q2)

We focused on maths and code because they are 'traditional' reasoning tasks. That said, we agree that other datasets, of different style, are interesting.

To that point, during the development of this project, we ran some preliminary experiments on the distillation procedure on ZebraLogic. These are logic puzzles, a typical constraint-satisfaction problem.

There is a massive difference between the no-CoT and CoT accuracy: close to 0% and 100%, respectively. This makes sense, in our opinion: the questions aren't inherently difficult, but they definitely need working through methodically. The latent student achieves 90%+ in most cases, even at small values of kk.

It's not exactly your proposal, but is this the sort of thing that you were thinking of?

Whilst we do find this very interesting, it is quite a different style. There is a limit to what we could include in the paper - both to keep the narrative clear and the details not overwhelming, but also compute resources. So, we focused on the most well-known benchmarks.

Length Generalisation (W2)

We actually ran some preliminary experiments to this effect: eg, train on Base and evaluate on P1 - or vice versa. Other reviewers raised similar points, so we address these in a shared comment.

Given that multiple reviewers discussed this, we are in the process of rerunning these experiments. We'll include them in the revised version.

CoT Recovery (W3)

We aren't absolutely sure what you mean by this. If our response below doesn't address your point, please do raise this and we'll do our best to correct it.

By "recovers nn%", we mean of the performance induced by CoT: (student−no-CoT)/(CoT−no-CoT)≥n%(\text{student} - \text{no-CoT}) / (\text{CoT} - \text{no-CoT}) \ge n\%. However, we realised that, in at least one case, we inadvertently used the plain student/CoT\text{student} / \text{CoT}. For example, (No-CoT, student, CoT) = (80%, 90%, 100%) should be (90 - 80) / (100 - 80) = 50% recovery, not 90/100 = 90%.

Naturally, we'll fix this and search for any similar mistakes when revising the paper.

Perhaps this was the cause of confusion?

Relationship Between CoT and Distillation Quality (Q3)

We did look into this. Like you, we thought that perhaps questions with long reasoning trace would require a longer latent 'trace'. We were never able to establish anything particularly concrete, though. Below, we hypothesise why:

  • Confounding:
    • We also expect that questions that are harder would both have longer CoTs and require more latent tokens
    • Separating these two effects, particularly with the inherent randomness of LLMs, seems difficult
  • What's actually being learnt:
    • Only the (change in) answer distribution needs to be learnt, not the CoT's text
    • Likely, long CoTs consist of many incorrect attempts, but the final (correct) attempt dominates effect on the answer
    • Is this roughly what your final point is suggesting?
Reviewer_xXQBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on a fundamental question in latent reasoning: Do short latent contexts that replicate CoT performance exist? It separates the question of existence from that of generation. The authors train a per-question latent context via knowledge distillation, so that the student can answer directly after the latent and match the teacher’s CoT and answer distribution. On Qwen3-1.7B/8B, a single latent token recovers almost the full CoT performance on HumanEval+, and 256 latent tokens exceed the teacher on MATH-Hard. Besides, The authors use GSM-Symbolic to argue that the latents carry some form of reasoning rather than just memorizing answers.

Critical strengths
  1. Evaluation is comprehensive: Two datasets × two model sizes × k sweep, ≥1024 rollouts per question, and error bars throughout. The similarity/cross-similarity/plagiarism diagnostics and Table 4.2 is informative.
  2. The loss derivation is clear and makes sense, as shown in Appendix B.1.
Critical weaknesses
  1. Per-question optimization with $ \sim 10^{-5}$ parameters against a distribution that can be sampled thousands of times could be considered as per-instance prompt tuning, which is proven to be highly expressive. The conclusion of "existence" is trivial and expected.
  2. A per-question latent may simply encode the answer (or solution) content of that question. The held-out loss, similarity and n-gram results only check whether the student copies rollouts from the training bank. They do not rule out a latent that stores the answer itself. The authors should add experiments such as removing questions from context and only keeping latent, and latents swapped across questions.
  3. For HumanEval+, the authors admit that answer diversity is low and the training bank effectively covers the teacher’s support. In this case, held-out loss tracking training loss cannot rule out memorization. On MATH-Hard, self-similarities are around 10−4–10−610^{-4}–10^{-6} (Figure D.5), so the similarity statistics carry little signal. In other words, each diagnostic metric is only valid on one dataset.
  4. The motivation of this paper is to separate generation failure from non-existence, but there is no evaluation and comparison with the existing generative methods (e.g., Soft Thinking, SoftCoT) against these latents under the same metrics.
Questions for authors
  1. Given the known expressivity of per-instance prompt tuning, what about the existence result is unexpected? What would non-existence look like?
  2. How do you get the conclusion of "recovers 98% of CoT performance" in the abstract, for which setting?
  3. Why do you use different prompts for 1.7B and 8B models on HumanEval+?
  4. What would the student’s accuracy be with the question removed from the context (z only)?
  5. Can §5 be extended to Qwen3-8B and more questions?
  6. There is no discrete baseline and the claim in Section 6 “standard, discrete CoTs cannot” is not tested. How does a few-shot discrete baseline compare?
4 comments

Response to Reviewer xXQB

We thank the reviewer for their extensive review, with clearly itemised key points/questions.

We address these below, including proposing changes to rectify the concerns where appropriate. We apologise in advance for the length, but we have tried to keep them focussed on your particulars and give a comprehensive response.

We hope that these changes satisfy your concerns. But, if they do not, please do raise this.

Inevitability (W1/Q1)

Several reviewers raised related points, so we have addressed them in a unified manner in a shared comment.

In short, we agree that the result is not surprising, but don't feel that it's inevitable. For one, no latent-reasoning method has had success - meaning both performant and short. The lack of success is what motivated us to separate existence from (the harder question of) generation.

Prompt tuning, for example, address 'offline' evaluations, such as (teacher-forced) reconstruction of a piece of text; it's not used as an inference-time prompt in the way that latent reasoning would be.

Regardless, multiple reviewers raise this point, so we'll make it clearer in the revised version of the paper.

Reasoning vs Memorisation (W2/W3/Q4)

Your point is valid that the latent could store something like,

"The answer is {answer}. Write this in your own words/notation."

This would likely even give a pretty good validation loss.

Whilst we do not rule out some form of answer compression, we provide some evidence to the contrary.

Question–Latent Swap

We ran the experiments as you suggested, on MATH-Hard:

  • Concatenate a question and a different question's latent
  • Evaluate against both questions' verifiers

In short, the latent doesn't encode the answer in a form that the model can simply read off:

  • The model tries to answer the original question, and 'mismatched' latent slightly harms accuracy
  • When the question is removed, performance collapses

Other reviewers raised similar questions. We provide more details in a shared comment.

We'll certainly include this new study in the revised version of the paper - thank you!

GSM-Symbolic in §5

In our opinion, the strongest evidence against memorisation comes from §5:

  • The latents are evaluated on unseen prompts, of varying diversity
  • Even a single prompt (a=1a = 1) closes approximately half the CoT-induced gap (Figures 5.1 and D.7)
  • Simply encoding the answers seen during training doesn't naturally explain this performance transfer

Other reviewers raised similar points. We provide more discussion in a shared comment, including outlining further experiments that we'll run.

Framing, Low Diversity and Similarity Metrics

Another reviewer raised similar points around the framing and (low-)diversity. We provide a unified answer in a shared comment.

In short, many of the HumanEval+ question are not low-diversity, and the latents' performance is hardly affected by the question's diversity.

Your statement that similarities carry little signal for MATH-Hard is fair, though; we'll add a clear statement to this effect. We did consider not including it, but that felt like it was hiding the result.

Stronger evidence of "generalisation", in our opinion, comes from GSM-Symbolic in §5, referenced above.

Comparison with Existing Generative Methods (W4)

Comparisons with existing methods was certainly something on our mind. The primary metrics are the free-generation and teacher-forced loss - the latter of which wouldn't be fair to them, since they aren't trained to optimise it.

For accuracy, we did not want to spend substantial time implementing all these different methods just to run on the particular dataset in question. However, that doesn't prohibit us from being more precise:

  • The main point of interest is how much of the CoT-induced accuracy gap is recovered by the latent-reasoning method
  • We can simply quote other, related datasets for which the methods already provide results
  • Alongside this, we can report whether the 'reasoning' is short (eg, 64 tokens) or similar to discrete CoT

CoT-recovery Value (Q2)

If we denote by AkA_k the accuracy of the student with kk latents and A0/∞A_{0/\infty} the no-CoT/CoT accuracy, then Ak/A∞≥98.5%A_k / A_\infty \ge 98.5\% for k=64k = 64 across both datasets.

In hindsight, we should really be reporting how much of the CoT-induced gain is recovered, not the absolute gain - ie, the right-hand pane in Figure 4.3. This satisfies (Ak−A0)/(A∞−A0)≥93.4%(A_k - A_0) / (A_\infty - A_0) \ge 93.4\% for k=64k = 64 across both datasets.

We'll change the wording to emphasise "gap recovery" and reduce the percentage - possibly giving different expressions for HumanEval+ at k=1k = 1 and MATH-Hard at k≥64k \ge 64.

Prompt Disparity on HumanEval+ (Q3)

This was simply a mistake on our end. I'll briefly explain the difference, to justify why we felt that it was ok to leave.

  • We ran (HumanEval+, 1.7B) and (MATH-Hard, 8B) first, whilst exporing; after we'd completed those, we did (HE+, 8B) and (MH, 1.7B)
  • Unfortunately, we found that Qwen3-8B formatted the HumanEval+ code differently:
    • 1.7B wrote the entire function whilst 8B wrote only the code, unindented and often wrapped in Markdown ticks
    • We should have specified the instructions more clearly, and were just lucky that Qwen3-1.7B worked

The full 'hero run' was very compute heavy, and we made the call not to rerun it - particularly as this was close to the deadlines. What is important is not the exact numbers, but the evidence that the CoT-induced gain can be recovered. We made the judgement call that a subtly different prompt doesn't evaluate that.

GSM-Symbolic Extension (Q5)

We have started this going with more questions, still on Qwen3-1.7B, alongside the additional evaluations referenced above (in the GSM-Symbolic section).

If we have time, we'll certainly run the 8B variant. But, these things are pretty compute intensive, and we only have limited resources and time.

Few-shot Discrete-CoT Baseline (Q6)

All that the "standard, discrete CoTs cannot" was supposed to convey is that the CoT will address the question that it's asked, by default. Eg, "Solve x2+3x+2=0x^2 + 3x + 2 = 0." will (likely) go via x2−3x+2=(x−2)(x−1)x^2 - 3x + 2 = (x - 2) (x - 1) and deduce that the solutions are x∈{1,2}x \in \lbrace 1, 2 \rbrace; it won't also provide information for how to solve x2+5x+6=0x^2 + 5x + 6 = 0.

But, in our desire for conciseness, we didn't phrase this clearly. We wrote, "Can discrete CoT be adjusted (prompted) to carry such information as well? If so, does it (further) bloat the length substantially?" But, that nuance only comes after; we'll adjust the wording to soften the claim upfront.

As referenced above (in the GSM-Symbolic section), we'll also investigate a few-shot baseline, as you suggest: evaluate [q1,r1,q2,r2,...,qa,ra,q][q_1, r_1, q_2, r_2, ..., q_a, r_a, q] with CoT.

Thanks for the authors’ response, which answers several of my questions. Some concerns remain unresolved:

W1/Q1: The argument for non-trivial existence still rests on existing methods failing at generation, which this paper does not address. No clear criterion for non-existence is given. W2/Q4: The swap and z-only experiments do not rule out answer information that the latent encodes conditional on the question. Q6: The few-shot discrete baseline as the key control for §5 is still missing. W4: There is still no comparison with generative methods under the same setup.

I will maintain my current score.

Thanks for your response. Let us clarify a few point briefly.

W1/Q1/W4. Let us focus, for example, on prompt tuning. Previous works study narrow questions - eg, represent a single piece of text with a latent context. This is substantially easier than distribution learning. We have a concrete experiment, suggested by Rog3, to evidence this: the training bank is reduced from n=2048n = 2048 teacher samples to n∈{1,16,256}n \in \lbrace 1, 16, 256 \rbrace.

  • For small n∈{1,16}n \in \lbrace 1, 16 \rbrace, the latent easily (and quickly) fits to the training data, but the validation loss is high - indicating overfitting
  • For large n∈{256,2048}n \in \lbrace 256, 2048 \rbrace, the training loss drops more slowly and plateaus higher, but the validation loss largely tracks it - enabling generalisation

More details, including plots, are given in our response to Rog3's review.

Again, they exist, and we find them, and we are not arguing that it was especially hard. But, "trivial" is far stronger than "expected", and requires justification beyond success in easier areas.

W2/Q4. This same experiment provides evidence against simple encoding of the training data. It surely encodes some concept of the solution - indeed, that's what CoT does, so this is, in essence, the goal.

Q6. We only just finished running this experiment this afternoon. Colours match the augmentation count:

  • The latent is trained on aa augmentations
  • The CoT gets the same aa question–trace pairs as few-shot context

Both are evaluated on held-out augmentations.

Click to view plot of few-shot discrete CoT baseline

The results are perhaps as expected: increasing aa improves both cases. The few-shot baseline is extremely computationally expensive, and was limited to a≤4a \le 4, otherwise, we ran out of context; indeed, at a median of around 3k tokens, a=16a = 16 is pushing 50k tokens. In contrast, our performance saturates around k=16k = 16, roughly 750× below the a=4a = 4 few-shot CoT length.

PS. Apologies, we posted this yesterday, but accidentally on Rog3's thread. It has been removed from there and copied here

Reviewer_Rog3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Latent COT replaces discrete reasoning tokens with much fewer continuous vectors. The authors note that current latent COT methods don’t reach discrete COT performance. They study whether a small number of continuous vectors can even exist that match discrete reasoning performance. For a given prompt, they first generate a set of discrete reasoning chains + answers from a frozen model (Qwen3-1.7B/8B). They then give the same prompt + a sequence of continuous vectors + the same answer to the same frozen model, and minimize the KL divergence between the next-token distributions of the two (teacher-forced on the sampled answers) wrt the continuous vectors. The vectors are optimized separately for each prompt. The conclusion is that the loss can successfully be minimized and as such a small number of continuous vectors exists that matches discrete reasoning performance.

Critical strengths
  • Latent COT is a promising direction for improving reasoning efficiency.
  • The motivation of the work is clear.
  • The paper is easy to follow.
  • The paper is honest about its limitation of not proposing a method to find the continuous vectors, but rather just showing that they exist.
Critical weaknesses

My main concern for the paper is its rather unsurprising result.

  • from an information-theoretic perspective, a single d=4096 float32 vector (Qwen3-8B) has 4096 x 32 ≈ 131k bits. A 3000-token reasoning chain over a 150k vocabulary needs at most 3000 x log2(150k) ≈ 52k bits. So even a single latent vector has the capacity to encode a couple of full discrete reasoning chains, and k=256 vectors have the capacity for several hundred. That a frozen LLM can actually decode this much from one trained input vector has also been shown empirically (Kuratov et al., 2025, “Cramming 1568 Tokens into a Single Vector and Back Again”, not cited).
  • practically, several works do teacher distillation and train only a small number of parameters (e.g. LoRA). The paper trains a separate set of parameters (their continuous vectors) for each prompt. I don’t find it surprising that this matches the teacher.

The challenge of latent COT is how to train the LLM to generate these vectors. Showing that they can exist is rather clear from the above 2 points.

I also find the “generalization” claim a bit misleading. The validation set is, per prompt, just another 128 samples from the same teacher on the same prompt. These samples are not very different from the training samples: on HumanEval+ the 2048 training samples contain only a few hundred distinct answers, so most validation answers are contained in the training set. On MATH-Hard the answers differ as strings, but they are essentially the same solution in slightly different wording (the paper’s own n-gram analysis shows this). So a low validation loss only shows that the vectors don’t overfit to the specific training samples, which is expected with this much data per prompt. It does not show that “the latents carry some concept of reasoning”, as the abstract claims. A vector that simply encodes the answer of the prompt (which is not "reasoning") would likely get the same low validation loss.

Questions for authors
  • Can the authors test if the vectors encode more than the output? E.g., train z on a single teacher sample instead of 2048 and compare validation loss.
  • How does a single latent z shared across all prompts perform?
  • How does a z trained on one prompt perform on a different prompt?
5 comments

Response to Reviewer Rog3

We thank the reviewer for their clear review. Obviously, our hope was for a higher rating, but we appreciate the clarity with which the reviewer has raised their concerns. We hope that we are able to address these in the comments below. But, if any concerns remain, we are more than open to further discussion.

Inevitability (W1)

Several reviewers raised related points, so we have addressed them in a unified manner in a shared comment. In particular, the Kuratov et al paper which you referenced is discussed; it's certainly highly relevant, and we'll cite it properly.

In short, we agree that the result is not surprising, but don't feel that it's inevitable. For one, no latent-reasoning method has had success - meaning both performant and short. The lack of success is what motivated us to separate existence from (the harder question of) generation.

Prompt tuning, for example, address 'offline' evaluations, such as (teacher-forced) reconstruction of a piece of text; it's not used as an inference-time prompt in the way that latent reasoning would be.

Regardless, multiple reviewers raise this point, so we'll make it clearer in the revised version of the paper.

Reasoning/Generalisation vs Memorisation (W2/Q1)

Another reviewer raised similar points around the framing and (low-)diversity. We provide a unified answer in a shared comment.

In short, many of the HumanEval+ question are not low-diversity, and the latents' performance is hardly affected by the question's diversity.

Regarding "carry[ing] some concept of reasoning", stronger evidence, in our opinion, comes from §5:

  • The latents are evaluated on unseen prompts, of varying diversity
  • Even a single prompt (a=1a = 1) closes approximately half the CoT-induced gap (Figures 5.1 and D.7)
  • Simply encoding the answers seen during training doesn't naturally explain this performance transfer

Other reviewers raised similar points. We provide more discussion in a shared comment, including outlining further experiments that we'll run.

That said, we have not emphasised this successfully in the paper: §4 reads as if it is the final part; its claims should foreshadow §5. We'll address this in the revised version.

Regarding your specific first question, your proposed experiment sounds sensible. So, yes, we can certainly do it. We'll report it here once it lands, and add to the paper as appropriate.

To best optimise our compute, we'll fix k=64k = 64, say, and a small sweep for nn - likely, n=1,16,256n = 1, 16, 256. This will then be compared with the paper's n=2048n = 2048, of course.

Single Latent Shared Across Prompts (Q2)

We did evaluate this in the past, but didn't include it in the final set of runs. We felt that it diverged from the narrative a bit.

We took 128 HumanEval+ questions (randomly) and trained a single, 'global' latent (with kk slots) across them simultaneously; the 'amortised' capacity was then k′:=k/128k' := k/128.

Click to view plot of global latents

This was an older run, before we had optimised the learning rate, hence the lower-than-expected performance for local latents; the global's learning rate was (lightly) optimised.

In essence, this simply allowed some 'fractional capacity':

  • We already know that HumanEval+ achieves basically the same performance with capacity k=1k = 1 as with k=256k = 256
  • This global variant basically just allowed the amortised capacity k′<1k' < 1

Whilst experimenting with this, we did find that the global latent was very susceptible to overfitting. There were even several scenarios, particularly when evaluating on different prompts, where both the training and validation losses were decreasing, but the accuracy had collapsed to below the no-CoT baseline.

Cross-prompt Transfer (Q3)

"How does a zz trained on one prompt perform on a different prompt?" In a word: badly. This is to be expected: after all, it's not trained to do so.

This isn't just a conjecture. Since the reviews came in, we ran an experiment to test this, on MATH-Hard:

  • Concatenate a question and a different question's latent
  • Evaluate against both questions' verifiers

In short, the latent doesn't encode the answer in a form that the model can simply read off:

  • The model tries to answer the original question, and 'mismatched' latent slightly harms accuracy
  • When the question is removed, performance collapses

Other reviewers raised similar questions. We provide more details in a shared comment.

All this said, the GSM-Symbolic experiments in §5, as referenced above, do demonstrate that some transferability is possible.

I thank the authors for the response.

The authors agree that the result is not surprising. The main counterargument is that no method has managed to generate short latents yet. But that is about generation, which the paper does not address. Given the capacity of a single vector, I think existence was never really in question.

On reasoning vs. memorization, I don’t think the GSM-Symbolic results show more than the answer being encoded. The held-out augmentations are the same problem with different names and numbers, and the latent is trained on full worked solutions. So the latent may just encode a worked solution for one variant, which the model then uses as an example when solving the others. A simple baseline would be the discrete version of this, where the model gets one worked solution from a training variant as text in the prompt. The single-sample experiment I suggested would also help here.

The global-latent experiment also shows that validation loss can go down while accuracy drops below the no-CoT baseline. So the low validation loss in the main experiments is not good evidence that the latents capture the teacher’s distribution.

For these reasons, I keep my original rating.

Thanks for engaging quickly. We'd just like to highlight two points where our response lacked clarity.

  1. On your comment "The single-sample experiment I suggested would also help here":

    • This experiment is literally underway - we should've said this explicitly
    • It should be done by the end of the day (hopefully) - we'll update here once it is
    • [Edit: didn't get it done today, should be done tomorrow 🤞]
  2. Our sentence "There were even several scenarios, particularly when evaluating on different prompts, where both the training and validation losses were decreasing, but the accuracy had collapsed to below the no-CoT baseline" mixed two scenarios:

    • When there was a reduction in validation loss and accuracy, this was on different prompts (questions):
      • We trained on 128 (out of 164) HumanEval+ questions
      • We evaluated on the 36 held-out questions
    • There were also situations where the training loss continued to decrease but the accuracy collapsed:
      • This is not surprising, though - it's just normal overfitting
      • We were only trying to highlight that the global-latent set-up was more susceptible than the local-latent one

Apologies for our lack of clarity on these points

Comments on the paper

7 comments

Exceeding CoT Accuracy and Removed Rollouts

For attention of Reviewers sU7R and vqB2

The removed rollouts are (almost) exclusively due to failing to answer within the allotted token budget - which is already generous at 8k/16k (HumanEval+/MATH-Hard). In our opinion, it is appropriate to mark these as incorrect:

  • The goal is for the LLM to provide a correct answer
  • If it doesn't give any answer, then this is still a failure

Else, in a really extreme case, the latent could learn to encode, "This question is too hard; output random text indefinitely" for difficult questions, and only answer the easier ones.

That said, "exceeds CoT accuracy" was never supposed to be a major claim or anything; the primary point is that CoT accuracy is recovered. So, we'll certainly soften the wording. This will ensure that that's no superiority (over-)claim. Rather, the claim will be defensible whether capped rollouts are included or not.

Framing and Answer Diversity

For attention of Reviewers Rog3 and xXQB

Our set-up is that of distribution learning:

  • The objective is to learn/approximate the distribution pq,CoT:=p(⋅∣q)=∑τp(⋅∣q⊕τ)p(τ∣q)p_{q,\text{CoT}} := p(\cdot \mid q) = \sum_\tau p(\cdot \mid q \oplus \tau) p(\tau \mid q) with some pq,z:=p(⋅∣q⊕z)p_{q,z} := p(\cdot \mid q \oplus z)
  • We use KL to measure the difference: KL(pq,CoT∥pq,z)\mathrm{KL}(p_{q,\text{CoT}} \mathrel{\Vert} p_{q,z})

The marginalisation over all reasoning traces makes pq,CoT(y)p_{q,\text{CoT}}(y) too complex to even estimate, but we can (easily) sample from it via the LLM. This gives us access to training data. But, we need to ensure that we don't simply fit pq,zp_{q,z} to the data: the true KL requires fresh rollouts from pq,CoTp_{q,\text{CoT}}.

The low-diversity of some (HumanEval+) questions simply means that the training data covers the majority of pq,CoTp_{q,\text{CoT}}'s support. Then, "generalisation" and "memorisation" are fundamentally the same: memorising almost the entire distribution is a legitimate approximation.

We do appreciate, however, that this is an easier, less interesting case. Not all HumanEval+ questions are low-diversity, though. We hope to alleviate your concerns, at least partially, by plotting accuracy/loss against unique-answer counts; the counts are grouped into five equally weighted bins.

Click to view plot of accuracy and loss against teacher unique-answer counts, binned

This indicates that the distilled latent still performs as well as the on the high-diversity questions - performance isn't dominated by low-diversity questions.

We feel that this supports the relevance of the held-out evaluation. Again, though, it cannot distinguish "generalisation to unseen data" from "memorising the entire distribution" - the former doesn't exist in the low-diversity case.

GSM-Symbolic Transfer

For attention of Reviewers eTc6, NxXo, Rog3, vqB2 and xXQB

§5 differs from §4 both in the training and evaluation:

  • The latent is trained on aa augmentations of a question's template
  • It is evaluated on held-out augmentations - different prompts

Intra- and Inter-template Transfer

Even training on a single augmentation (a=1a = 1) closes approximately half the CoT-induced gap. To take this further, we extend the evaluation:

  • Currently, a latent is trained and evaluated on the same template (Base/Sym/P1/P2)
  • We'll train on one template still, but evaluate on all

Click to view plots of GSM-Symbolic train × evaluation grids:

A substantial amount this single-augmentation (a=1a = 1) generalisation extends to cross-template evaluation.

We fully acknowledge that the sections currently read rather independently, and that this inference is not made sufficiently clear. We'll address this in the revised version.

Few-shot Discrete-CoT Baseline

One reviewer (xXQB) suggested running 'few-shot' baseline, which we think is worth including in this shared comment. Our implementation of this:

  • Sample augmentations q1,...,qaq_1, ..., q_a of a question qq
  • Sample corresponding reasoning traces r1,...,rar_1, ..., r_a and evaluate [q1,r1,q2,r2,...,qa,ra,q][q_1, r_1, q_2, r_2, ..., q_a, r_a, q] without further CoT
  • The evaluation can be intra- and inter-template, as above

The results for this are now in (18th Nov). Colours match the augmentation count:

  • The latent is trained on aa augmentations
  • The CoT gets the same aa question–trace pairs as few-shot context

Both are evaluated on held-out augmentations.

Click to view plot of few-shot discrete CoT baseline

The results are perhaps as expected: increasing aa improves both cases. The few-shot baseline is extremely computationally expensive, and was limited to a≤4a \le 4, otherwise, we ran out of context; indeed, at a median of around 3k tokens, a=16a = 16 is pushing 50k tokens. In contrast, our performance saturates around k=16k = 16, roughly 750× below the a=4a = 4 few-shot CoT length.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.