Hierarchical Variational Chain of Thoughts for Vision and Language Reasoning

Chain-of-thought (CoT) vectors offer a compact way to steer reasoning in frozen large models, but representing reasoning with a single deterministic vector shared across inputs limits adaptation to individual problems. We propose Hierarchical Variational Chain of Thoughts (HieraCoT) by modeling the CoT trace as a distribution over latent states. A global state is defined to capture reasoning patterns shared across tasks, while a local state adapts these patterns to the current input. This separation is encouraged by an input-independent global prior and a local prior conditioned on both the input and the global state. During training, posterior networks use CoT traces and target answers as auxiliary information, and a variational objective aligns these posteriors with the corresponding priors. At inference, states sampled from the priors are projected into additive hidden-state shifts, enabling answer generation without explicit rationales or updates to the backbone parameters. Because the intervention operates in hidden space, the same formulation supports both vision-language and language-only reasoning. Experiments across four frozen backbones show that HieraCoT outperforms direct prompting, CoT prompting, and deterministic CoT-vector baselines, improving average accuracy over CoT prompting by 2.32–5.15 percentage points. Ablations support the benefits of the global–local hierarchy, while cross-domain evaluations provide evidence that the learned states transfer beyond their training domains.