- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper is an existence study, which aims to answer whether latent thoughts can replicate CoT performance/behavior. It optimizes question-specific latent tokens by knowledge distillation from plenty CoT rollouts. Experiments with Qwen3-1.7B/8B show that just one latent token can restore 98% CoT accuracy on HumanEval+, and even exceed it on MATH-Hard with 256 latent tokens. Held-out rollout diagnostics and GSM-Symbolic augmentation experiments further show that latent thoughts can provide performance gains on unseen augmentations.
Critical strengths
- Recovering near-CoT accuracy with short learned latent tokens provides useful evidence about the capacity of latent reasoning, might be a meaningful reference for future latent generation research.
- Evaluation goes beyond accuracy. Held-out teacher rollouts, output-overlap diagnostics, and unseen augmentations strengthen the evidence that the results extend beyond memorising individual training responses.
Critical weaknesses
- Overstatement on failure mode diagnostics using similarity
Section 4.2 claimed that by comparing cross-similarity with self-similarity, we can diagnose failure modes, but it doesn't strictly hold.
Consider distributions over the same two answers: . Their self-similarities are 0.68 and 0.52, and cross-similarity is 0.56, sitting between them, but the student added no new answers. This is contradictory to the statement in line 367.
Similarly, with , both self-similarities are 0.46, while cross-similarity is 0.37, below both, but the student didn't replace teacher answers with new ones. This is contradictory to the row 2 in Table 4.2.
- Accuracy gain on MATH-Hard weakened by some evidence
In line 315 and 475, the paper highlights that on MATH-Hard, there is a strong upward trend as increases, and CoT accuracy is recovered with 64 latent tokens, and even exceeded with 256. But in Table D.3, with , the student drops 1.2pts for 1.7B and only gets 0.3pts increase for 8B. This weakens the claimed superiority and should feature prominently.
I am willing to raise my score once any of the above are properly addressed.
Questions for authors
See weaknesses.
2 comments
Thanks for the clarification. This helps address my concerns. I will keep my score.
Response to Reviewer sU7R
We thank the reviewer for their time and effort.
For each key point/question raised, we first describe what we're trying to get across in the relevant section, then what we'll change.
We hope that these changes satisfy your concerns. But, if they do not, please do raise this.
Overstatement of Failure-mode Diagnostics (W1)
This is a fair concern. Our intent in Section/Table 4.2 is to highlight 'typical' errors, and to demonstrate what effect that error would have. By not observing that effect, we rule out the error.
But, you are completely correct: a different error may produce the same effect. We (inadvertently) phrased an "if" as an "if and only if".
We'll rectify this - eg, at line 367, as you point out, but also 351.
Removed Rollouts (W2)
Two reviewers mentioned removed rollouts. We respond to this point in a shared comment. In short, we'll soften the phrasing to ensure that there is no superiority or over-claim.