acceptodds
Withdrawn from ICLR 2027

Latent Role Separation for Continuous CoT via Hierarchical Latent Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continuous chain-of-thought (CoT) compresses a verbal trace into a short sequence of vectors. Those vectors must both discover a decomposition and carry out the computation, yet existing methods leave their computational roles unspecified. We introduce Hierarchical Latent Reasoning (HLR). A small student first emits one plan vector and then K detail vectors. Every detail is generated from a prefix that already contains the plan, so the plan-to-detail edge is present in the student forward pass and remains active at inference. The projected hierarchy is read by a 7B teacher. Training replaces gold plan and detail text from left to right with the corresponding projected student states, aligns each latent to its role, and scores one answer path per example. We compare HLR with six latent-reasoning baselines on two 7B backbones and five reasoning benchmarks, under a shared five-slot budget for every continuous method. Under this protocol, HLR reaches state-of-the-art accuracy. The gains are largest where one strategy is reused across successive state updates, so the plan stays available while the details execute. In the hierarchy ablation, removing the plan-to-detail edge accounts for the larger drop, and replacing the exclusive answer path accounts for a smaller one. CODI and SemCoT lead the remaining Mistral cells.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent reasoning, implicit chain-of-thought, continuous thoughts, hierarchical representations, large language models, multi-step reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_DFHtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents Hierarchical Latent Reasoning, which trains a small student to emit a plan vector and KK detail vectors, that are then fed to a teacher model to produce the final answer. Starting with textual annotations of plans and details, the training proceeds in a curriculum fashion replacing each plan/detail step with a latent vector while student+teacher combinations learns to produce the final answer with LoRA params. Experiments on 2 backbones on 5 datasets shows that HLR beats six other latent reasoning baselines.

Critical strengths
  • Tackles an important problem and simplicity of the approach. Assigning roles to latent vectors is an interesting way to supervise latent reasoning methods and authors present a method incorporating roles in curriculum based training. Results and ablations show improvements.
Critical weaknesses
  • Unfair comparison with baselines and approach limitations. HLR uses a student+teacher setup, both trained with LoRA params, while CoCoNUT, CODI are student models being trained to answer. HLR, therefore uses substantially more params than others, this makes the comparison unfair. The proposed approach requires plan+detail annotations of existing problems which are not feasible to obtain for additional datasets. All experiments exhibit high variance, possibly due to small training data size (<1k). Authors are suggested to use a larger dataset (like GSM8k-Aug) and train till convergence.

  • Potential overclaiming and Unclear terminologies.

    • Contribution (i) states "..plan latent is a direct ancestor.." which is just the latent being in context, while Contribution(ii) is the CoCoNUT training method.
    • Training objectives (Sec 3.3 + Appendix A) introduce a lot of additional details without proper justification or reasoning. Figure 1 introduces a "Flat" reasoning baseline which is not mentioned later.
    • Terms like "hierarchical", "counterfactual", are used multiple times in unexpected contexts without defining properly. A list of definitions explaining terms would really help.
Questions for authors
  • Which baseline method does Figure 1, "Baseline" correspond to? Is it right to say that the only difference between the two is what each latent trained to represent?
  • Are Table 2 and Figure 3 same data represented twice?
  • Can you present an example of text plan + detail trace.
  • Do you see training and validation loss converge well?
Reviewer_L77COpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Hierarchical Latent Reasoning (HLR), where continuous reasoning is organized into a plan vector followed by detail vectors. A small student is used to generate the plan, followed by details conditioned on the plan as well as previously-sketched details, which are then projected to a larger (7B) answer model with trained LoRA adapters. Inspired by Coconut, the training procedure progressively replaces annotated plan and detail textual spans with continuous vectors in a curriculum fashion. The experimental validation studies HLR on five benchmarks against various latent reasoning baselines, using student-teacher pairs from the Qwen2.5 and Mistral families, achieving clear performance gains with the former.

Critical strengths

HLR is an interesting instantiation of how reasoning should decompose into planning and details, where the role-specific training ensures there is a learning signal for each to actually be used. This seems like it could be a useful way to organize when there’s a small (latent) budget.

Critical weaknesses
  1. The performance with Mistral is unconvincing, and it is documented that Qwen2.5 models sometimes improve even with random or flipped rewards [1], being untrustworthy as the sole signal of empirical value.

  2. The value of role separation has not been clearly demonstrated beyond the hierarchy ablation in Table 4 (which is just CoinFlip), but there is no direct testing that the vectors perform the respective planning and execution roles as intended.

[1] Spurious Rewards: Rethinking Training Signals in RLVR, Shao et al. 2025

Questions for authors
  1. It appears that most of the baselines use 10 epochs, while HLR is given more (cumulatively over the stages). Please run experiments with a standardized compute budget.
  2. Could you remove the plan-specific supervision with the five-vector model, or try permuting the role assignments? It would be useful to establish whether the first vector actually ends up having a distinct planning role or not.
  3. Furthermore, it is unclear which Mistral model was used, as it is uncited and, to my knowledge, Mistral has not directly released a 1.1B model.
  4. The SemCoT citation is missing; also, please describe in greater detail what the “SemCoT-style processed splits” entail procedurally.
  5. As I understand it, the five-slot constraint matches the number of continuous representations, but this is not the same as equalizing the computation. Clarification on this point would be appreciated.
Reviewer_QSdbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a hierarchical latent reasoning method in which latent tokens are generated by a smaller student LLM and the answer is generated by a larger teacher LLM. The latent reasoning trace consists of one plan token and multiple detail tokens. Unfortunately, because of the ambiguous notation, I could not fully understand how these are generated. The generated latent tokens are projected through an MLP into the hidden-state space of the teacher model, which then uses them to predict the answer. The method uses a Coconut-like curriculum to train the student LLM, the MLP projection, and the LoRA weights over the teacher model. Training objective has four different components that also not discussed in the main text of the pape, so it is not completely clear how the whole system is trained. The methodd does require a specific plan-detail data annotation, making it harder to compare it to previous latent reasoning apporaches.

Critical strengths

Adding hierarchical structure to latent reasoning is an interesting direction. However, the implementation and evaluation in this paper are unclear, so I cannot determine how the method works or whether it works at all.

Critical weaknesses
  1. The writing is very poor - too dense and difficult to follow. The paper uses many terms without enough explanation, important definitions and training details are scattered between the main text and the appendix.

  2. The evaluation does not provide important baselines: full CoT and no CoT. Furthermore, relevant latent-reasoning methods such as PCCoT, KaVa, and MUX are missing. Some reported baseline results also look unexpectedly low. In particular, the CODI results on GSM8K and SVAMP are much lower than I expected from previous papers. It is also unclear which model is used for CODI. HLR uses a smaller student model and a 7B teacher model, while CODI uses one shared model, but the paper does not state which of the two are used for CODI.

  3. The method and the contribution of its components are difficult to assess. The method requires specially generated plan/detail annotations, but the quality and effect of this supervision are not isolated from the proposed architecture. There is no ablations over hyperparameters, such as number of steps in the plan of number of "detail" tokens K. The paper alos never explain why both the plan and details must be latent or why the student and teacher must be separate models. The main text does not explain different loss components as well as not properly explaining where they come from.

Questions for authors
  1. Why are Full-CoT and No-CoT baselines not included? Why are PCCoT, KaVa, and MUX omitted from the related work and experiments?

  2. Why are the reported CODI results on GSM8K and SVAMP much lower than previously reported results? Could the authors explain which differences in the data, training setup, or implementation account for this gap?

  3. Which model is used for CODI in each backbone block? Does CODI use the smaller student model or the 7B teacher model as its shared backbone?

  4. Why do both the plan and the details need to be latent? Could the authors provide ablations in which only the plan or only the details are latent?

  5. How does performance change with K? The experiments fix K=4 detail tokens, but do not show whether the result depends on this choice.

  6. Why are two different student and teacher models required? Could the same model generate the latent hierarchy and predict the answer?

  7. How was the quality of the generated plan/detail training data assessed? Without a baseline or ablation that removes this supervision, how much of the improvement comes from the annotations rather than the architecture?

  8. Could the authors define all loss components clearly in the main text and explain the role of each component?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.