- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper taxonomizes latent reasoning traces into carriers, empty, bare placeholder, laid-out placeholder, assembling, and unstructured using a series of diagnostic tests that measure causal use, encoded content, and state dynamics. The paper resolves disagreements between prior work that completed a subset of these tests. The paper makes a few surprising observations, including that readable final answers and apparent sequential reasoning in the latent reasoning trace do not mean the model needs the trace to get its final answer.
Critical strengths
S1) The ProsQA + Sim-CoT-Coconut result is interesting. It is surprising that you can decode the 'step k written into slot k' and yet you can replace them with noise/mean vector/donor vector and still get the same accuracy. This clearly demonstrates that readability does not imply usage.
S2) The paper does a good job of expanding diagnostic tests from prior work, and demonstrating that multiple diagnostic tests are needed to understand a latent reasoning trace.
Critical weaknesses
W1) The 'unstructured' label for the GSM8k results are likely due to the probe construction. The probe for ProsQA looks for intermediate results, but for GSM8k it only looks for the final answer. The paper says "GSM8k has no candidate list", but why not try the gold label solution for GSM8k? Of course that gold label solution is not unique; there are many valid solutions to a given GSM8k problem, but it seems like a good starting point. If you do find the 'step k written into slot k' result for the GSM8k gold solutions, then the 'unstructured' label should be changed to 'assembling'. Why do you think the unstructured label is correct given the current probe setup?
W2) Limited models, which have been studied by prior work (though less extensively). The paper studies Coconut, CODI, and SIM-CoT Coconut only; the first two have been studied by prior work with similar findings in their Section 4. How would the diagnostic tests and taxonomy apply to more recent latent reasoning architectures, such as full-bandwidth transformers or looped architectures?
Questions for authors
Suggestion 1) Figure 5 should be in the main paper.
Suggestion 2) "Where prior causal tests apply a single corruption (deletion (Dilgren & Wiegreffe, 2026)or noise(Cui et al., 2026)) each, we pair deletion with three replacements (noise, the dataset mean, and another problem’s vectors)" is not true; Dilgren & Wiegreffe also do the replace with dataset mean test.