acceptodds
Withdrawn from ICLR 2027

Hierarchical Variational Chain of Thoughts for Vision and Language Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Chain-of-thought (CoT) vectors offer a compact way to steer reasoning in frozen large models, but representing reasoning with a single deterministic vector shared across inputs limits adaptation to individual problems. We propose Hierarchical Variational Chain of Thoughts (HieraCoT) by modeling the CoT trace as a distribution over latent states. A global state is defined to capture reasoning patterns shared across tasks, while a local state adapts these patterns to the current input. This separation is encouraged by an input-independent global prior and a local prior conditioned on both the input and the global state. During training, posterior networks use CoT traces and target answers as auxiliary information, and a variational objective aligns these posteriors with the corresponding priors. At inference, states sampled from the priors are projected into additive hidden-state shifts, enabling answer generation without explicit rationales or updates to the backbone parameters. Because the intervention operates in hidden space, the same formulation supports both vision-language and language-only reasoning. Experiments across four frozen backbones show that HieraCoT outperforms direct prompting, CoT prompting, and deterministic CoT-vector baselines, improving average accuracy over CoT prompting by 2.32–5.15 percentage points. Ablations support the benefits of the global–local hierarchy, while cross-domain evaluations provide evidence that the learned states transfer beyond their training domains.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Chain-of-Thought, Reasoning, Vector

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UCTaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Variational Chain of Thoughts (HIERACOT), which models the CoT trace as a distribution over latent states, to improve reasoning. Instead of representing reasoning with a single deterministic CoT vector, HIERACOT introduces a variational framework consisting of a posterior network that leverages the input, CoT trace, and answer during training, and a prior network that generates latent states without access to the CoT trace at inference time. Its hierarchical variant further decomposes latent reasoning into a global variable that captures shared reasoning patterns and a local variable that models input-specific variations, with the resulting latent states projected and injected into the intermediate representations of the frozen backbone. Experiments across multiple benchmarks also demonstrate consistent improvements over the considered baselines.

Critical strengths
  1. The extension from a deterministic CoT vector to a hierarchical distribution over latent states is valuable, as it provides a more flexible representation of reasoning and distinguishes shared reasoning patterns from input-specific variations.
  2. Evaluation across language and vision-language reasoning tasks using multiple frozen backbones demonstrates the broad applicability of the proposed intermediate-representation injection framework.
  3. This paper clearly presents its motivation, variational formulation, and hierarchical design, making the overall framework easy to follow.
Critical weaknesses
  1. The technical novelty of HIERACOT appears limited, as it essentially applies standard KL-based posterior–prior matching to distill CoT and answer information into an inference-time prior, without establishing a substantive advance beyond conventional variational distillation.
  2. The experimental validation is limited. The baselines cover only direct prediction, CoT prompting, and two deterministic CoT-vector methods, omitting other latent-reasoning or parameter-matched baselines. Moreover, HIERACOT-S and HIERACOT-H are compared under different configurations, making it difficult to isolate the contribution of hierarchical modeling.
  3. The semantic interpretation of the hierarchy is insufficiently supported. Since the global prior is input-independent and learned separately for each benchmark, it may capture dataset-level bias rather than transferable reasoning patterns. Although this paper provides latent-space visualizations and cross-domain evaluations, the stability of the global variable is largely expected by design while accuracy improvements alone do not establish what information it encodes.
Questions for authors
  1. For datasets without native CoT annotations, particularly TextVQA, how are the CoT traces used for training obtained?
  2. What are the training cost and inference latency of HIERACOT, and how do they compare with those of the baselines under matched settings?
Reviewer_LfQ1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HieraCoT to address the expressive limitations of injecting a single deterministic CoT vector into a frozen model. The CoT state is modeled using hierarchical latent variables, where the global state gg is designed to capture reasoning tendencies shared across problems, while the local state zz adapts to the current input. During training, the posterior networks take pooled hidden-state features extracted from the complete sample (input, CoT trace, and answer) as auxiliary information, while the global prior is input-independent and the local prior is conditioned on both the input and the global state. The posterior and prior distributions are aligned through a variational training objective that also encourages correct answer prediction. During inference, only the priors are used to sample the latent states, which are projected into additive shifts and injected into the hidden states at all token positions of a selected layer in the frozen backbone. This enables answer generation without explicitly generating rationales or updating the backbone parameters.

Critical strengths

S1. The motivation for constructing the problem is sufficient. This paper points out a reasonable limitation of existing CoT-vector methods: sharing a single deterministic vector across different inputs may fail to capture the inference needs specific to a particular instance. Therefore, introducing input-conditioned latent states is a natural and meaningful direction to address this limitation.

S2. The empirical evaluation is comprehensive, and the performance improvement is robust. The study conducts an extensive assessment across four backbone models and eight benchmarks, spanning both linguistic reasoning and vision-language reasoning tasks. These results clearly demonstrate the practical potential of the proposed method.

Critical weaknesses

W1. The core proposition of a "separation between global and local reasoning" lacks sufficient theoretical grounding. A core argument of this paper is that "global priors" independent of input enable global latent states to capture transferable reasoning patterns, while "local states based on input conditions" capture reasoning information specific to a particular instance. However, the proposed decomposition does not theoretically guarantee such a functional separation. Since both types of latent states ultimately contribute to a single, cumulative hidden-state shift, their respective roles are not necessarily distinguishable.

W2. Experimental evidence failed to fully extract the specific benefits of hierarchical variational modeling. Although the experimental results are promising, it remains unclear whether the performance gains stem from the proposed "hierarchical variational modeling" framework or simply from the introduction of input-conditioned steering vectors.

Questions for authors

Thank you for your work. I acknowledge the problems you have identified, as well as the thorough experimental evaluations provided in the paper. However, I have some reservations regarding whether the theoretical framework and empirical evidence are sufficient to support the paper's core arguments. I would be happy to reconsider my rating if the following questions can be addressed satisfactorily.

1. Theoretical basis for the "global-local" separation.

The paper claims that the global state captures transferable reasoning patterns, while the local state captures instance-specific information. It is not entirely clear, however, whether this separation is truly supported by the proposed variational formulation. In particular, the lower variance observed in the global state (as shown in Figure 5) might simply stem from an "input-independent prior" and KL regularization, rather than representing a meaningful, shared reasoning representation. Could the authors provide evidence to support this interpretation? Furthermore, how can this mechanism be distinguished from a scenario where the model has merely learned a near-constant "steering vector"?

2. Necessity of the hierarchical variational formulation.

Although the experimental results show that this method outperforms the deterministic "Chain-of-Thought vector" (CoT-vector) baseline, it remains unclear whether this performance gain stems from the hierarchical variational design or simply from the introduction of "input-conditioned steering." Additionally, in vision-language tasks, the performance improvement of HieraCoT-H over HieraCoT-S is relatively limited. Could the authors provide evidence demonstrating that the "hierarchical variational formulation" offers a distinct advantage over simpler "input-adaptive approaches"?

Reviewer_CBM4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HieraCoT, which learns distributions over activation shifts for reasoning with frozen language and vision-language models. During training, posterior networks use the input, a CoT trace, and the target answer. At inference, the shifts are obtained from priors that do not require the trace or answer. The hierarchical version combines an input-independent global latent with an input-conditioned local latent and injects their projected sum into a backbone layer. Experiments on four backbones and eight benchmarks compare the method with direct prompting, CoT prompting, and fixed or learned CoT vectors. The paper also studies hierarchy depth, global-prior conditioning, posterior inputs, injection layers, and cross-domain transfer.

Critical strengths
  1. The method gives a coherent way to use training-time rationales while keeping the backbone frozen and generating answers without explicit rationales at inference. Posterior access to the answer is clearly separated from prior-only inference. The posterior-input ablation in Table 4 is useful: removing the CoT trace reduces accuracy by 2.2–7.7 points, while removing the separate answer input costs only 0.2–0.8 points. The authors also acknowledge that the trace may itself contain answer information, which keeps this result in perspective.

  2. The improvements appear across both modalities and all four backbones. HieraCoT-H improves average accuracy over CoT prompting by 2.32–5.15 points and also exceeds the learned CoT-vector baseline in each backbone-level average (Tables 1–2). The hierarchy-depth and global-prior ablations go beyond a single comparison with prompting. The transfer results are encouraging as well, especially the MMLU-Pro gains, although they cover a limited set of domain shifts.

Critical weaknesses
  1. The experiments do not isolate the benefit of hierarchical variational modeling from input conditioning and additional capacity. The deterministic baselines use a single shared vector, whereas HieraCoT adds networks that generate a different shift for each input. A deterministic input-conditioned network could address the same limitation, but this comparison is missing. The S-versus-H ablation also changes several factors: in the vision-language setting, S uses 128 latent dimensions, while H uses 128 global plus 32 local dimensions, different KL weights, and separate norm limits; in the language setting, H doubles the total latent dimensionality (Appendices B.1–B.2). The existing ablations are useful, but they do not establish whether a hierarchy is preferable under comparable capacity and regularization. This matters because the additional H-over-S gains average only 0.62 and 0.54 points on the two vision-language backbones, without reported uncertainty.

  2. The evidence for the claimed global/local reasoning roles is weaker than the paper suggests. Figure 5 shows that the global posterior mean varies little across inputs. That is consistent with a shared representation, but also with a constant offset or a branch that contributes little to prediction. The raw latent standard deviations do not measure the effects of the projected states, particularly when the projections are learned. The training objective also needs to be stated more precisely: Section 3.3 says it maximizes Equation 7, but Appendix B.1 uses global and local KL weights of 0.5 and 5.0. This is a different regularized objective and is not generally the ELBO derived in Appendix A. The derivation itself is valid; the concern is that neither the weighted objective nor the variance analysis establishes that the prior structure alone produces the claimed separation of reasoning roles.

Questions for authors
  1. How does HieraCoT compare with a deterministic input-conditioned steering network, for example a learned shared offset plus an input-dependent shift? Does H still outperform S when trainable capacity, latent dimensions, injection norms, and tuning budgets are matched?

  2. What happens to accuracy and transfer performance when the global latent is fixed to its prior mean, or when its projected contribution is removed? An analysis of the projected components and these interventions would help distinguish a useful global reasoning state from a shared bias or an unused branch.

  3. Please specify the exact optimized objective, including KL reductions, weights, and warmup. How does the unit-weight ELBO compare? At inference, how many latent samples are used per prediction, and how are they combined? Please also report paired variability across training and latent-sampling seeds for the H-versus-S comparison.

  4. What are the CoT sources and exact training partitions for each dataset? Appendix B.1 says annotations are used where available; how are missing rationales handled? For the transfer experiments, please provide domain sample counts and clarify whether all hyperparameters were selected using source-domain data only.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.