acceptodds
Under review as a conference paper at ICLR 2027

Characterizing Latent Reasoning Dynamics in LLMs

Abstract

Reasoning in LLMs is typically explained through two distinct mechanisms: the autoregressive generation of discrete tokens, and the looped refinement of a continuous latent state. Both improve performance by increasing inference-time computation, yet the continuous computational dynamics underlying an off-the-shelf, token-based model's ordinary reasoning have not been directly observed. We bridge this gap by formalizing token-based reasoning as state transitions over KV-cache prefixes, compressing this high-dimensional space with a KV-cache VAE to map problem-specific error manifolds. Analyzing the resulting state-space dynamics, we find that standard autoregressive LLMs implicitly perform latent reasoning: their trajectories closely mimic gradient descent over basins. Specific tokens drive this descent, some rapidly exploiting the error gradient and others escaping local minima to explore, and trajectories quickly recover their descent path after off-policy perturbation. Applying these mechanistic insights, we causally improve inference-time scaling by optimizing the token-budget tradeoff between trajectory depth and breadth. Specifically, we demonstrate that initializing diverse latent "random seeds," much like in vanilla gradient descent, steers trajectories toward distinct local minima and yields a larger diversity in correct answers. Finally, we find that RL reshapes the latent space to contain fewer basins, along with faster and more likely convergence to basins correlated with improved performance.

Primary area:
interpretability and explainable AI
Keywords:
Reasoning, mechanistic interpretability, LLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UEcSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether ordinary token-based reasoning in off-the-shelf LLMs exhibits continuous latent dynamics resembling gradient descent. The authors represent reasoning states using KV-cache prefixes and train a VAE with reconstruction, continuation, and KL objectives to obtain a compressed state space. They analyze trajectories against an estimated error field, examine token perturbations and phrase-level associations, and propose latent initialization perturbations to improve sampling diversity and pass@8.

Critical strengths
  1. An interesting question with a compelling, intuitive motivation. I initially found this paper particularly promising. Understanding the latent dynamics underlying ordinary LLM reasoning is an interesting problem, and the paper offers insights that could be meaningful to the community. This is the strongest aspect of the work and the main reason I would like to see it succeed.

  2. Clear and intuitive visualizations. The visualizations make an abstract problem accessible and help communicate the motivation and analysis effectively. Although the performance results are relatively modest, the conceptual perspective is interesting. I was glad to read the paper, and without the concerns below, I would have been inclined to give it a much higher rating.

Critical weaknesses
  1. The experimental comparisons are far too weak for the claimed contribution. For a paper targeting a top-tier venue, comparing only against standard temperature sampling is insufficient. This is clearly not the only work concerned with improving pass@8, so why are competitive existing methods absent from the comparison? This limited evaluation makes the claimed improvements difficult to find convincing, especially because the accuracy gains are neither particularly consistent nor compelling.

The inconsistencies between the main text and appendix further undermine my confidence. For example, Appendix B states that the interventional algorithms were not rerun on Nemotron, while Table 1 reports latent-seeding results for that model. The number of RL evaluation problems also differs between Section 4.4 and Appendix G. The more closely I read, the more questionable the experimental setup appears, which reinforces my concerns about its rigor. These reporting discrepancies should be clarified, but the lack of competitive comparisons remains a substantive weakness.

  1. The description of the mechanism is too strong, particularly in its treatment of correlation and causation. Statements suggesting that LLM reasoning implements gradient descent are presented with more certainty than the evidence warrants. The experiments provide empirical evidence, but the framing often makes this interpretation sound established. Not every researcher would agree with that conclusion. The proposed representation and analysis can still be useful, but the wording should be more cautious.

The experimental design also leaves important alternative explanations unresolved. Several analyses demonstrate correlations, yet the discussion reads as though they establish causation. This applies particularly to the phrase analysis and the relationships involving the estimated error field, where other influencing factors have not been sufficiently ruled out. Correlation and causation are not equivalent. In Figure 3, for example, an internal backtracking process could cause the model to write “think again”; the phrase itself need not cause backtracking. Figure 2 intervenes on entire token windows and therefore does not establish the causal role of a particular phrase.

These two concerns are the main reasons for my current low rating. They remain insufficiently addressed in the paper, and I do not think the current version is ready for acceptance. However, I would reconsider my rating if the authors can resolve these concerns.

Questions for authors
  1. Why does the evaluation compare only against standard temperature sampling, rather than competitive existing methods for improving multiple-sample success? Are relevant comparisons already available, or is there a justification for their omission?

  2. Could you reconcile the Nemotron intervention results in Table 1 with Appendix B? Could you also clarify whether the RL evaluation uses 64 problems or 32 problems with 64 traces each?

  3. What existing evidence rules out the possibility that an internal reasoning change produces both the phrase and the trajectory direction in Figure 3? Which conclusions from this analysis are intended to be causal?

  4. Given that the evidence is empirical and the speed prediction fails in Section 4.2, what supports the stronger claim that LLM reasoning implements gradient descent? Would you revise the wording to distinguish the observed properties from the proposed interpretation?

5 comments

Thanks for taking the time to read our paper thoroughly and for your questions and insightful feedback.

For each question posed, we carefully crafted several new experiments at scale, and made sure any claims we make are backed by empirical evidence. We believe going through this extensive characterization process has further deepened both our appreciation and clarity of the problem which we hope to extend to you.

(W1/Q1)

From the framework in our paper, we hypothesized that a method like latent seeds would increase the diversity of answers reached by the model. Our objective was not to present a new sampling strategy to be used in practice, but merely to highlight that by performing latent perturbations, we could intervene in our model and produce results consistent with this framework: namely higher diversity across rollouts. This was one of the factors due to which we stopped short of comparing it to other techniques. However, after your suggestion we compared our method against all of the existing sampling techniques we could find in prior literature (11 of them in total). We were surprised to find that our latent perturbation strategy performed better than most of the existing technique in terms of pass@8 scores. We believe working in a meaningful latent space provided our strategy with an advantage over the existing sampling techniques. Our experiments spanned 3 different tasks (MATH500, GSM8k, AMC23), and three different models (DeepSeek-R1-1.5B, Qwen3-4B, Nemotron-4B). Complete details about the experiments and the results can be found in Tables 9, 10, and 11 (see Appendix C.1). To be sure of our results, we even increased the evaluation to the full MATH-500 dataset and the full GSM8k evaluation dataset for DeepSeek-R1-1.5B. We believe that these results more convincingly indicate that our framework is the right way to think about reasoning, as it gave us greater control over the model.

Please let us know if there are any other methods you had in mind that you would like us to try.

ModelDataset (n)Best baseline nameBaseline pass@8Latent seed pass@8
Qwen3-4BMATH-500 (32)4 Baselines Tie0.9450.953
AMC23 (40)Temperature T=0.8 / Early hot0.9940.969
GSM8K (100)Temperature T=1.00.9600.965
Nemo-NanoMATH-500 (32)CoT-decode/Prefix branch0.9140.938
AMC23 (40)3 Baselines Tie0.9500.931
GSM8K (100)Temperature T=0.60.9560.963
R1-1.5BMATH-500 (500)Arithmetic sampling0.9620.962
AMC23 (40)Prefix branching (4×2)0.9380.942
GSM8K (1,319)CoT-decoding0.9580.961

(W2/Q2)

We sincerely apologize for the typos and the resulting inconsistencies between the Appendix and the main paper. We provide the following clarifications:

  1. The latent seeds based intervention was indeed run on all the models and detailed results for the Nemotron-4B models are in the above table and Table 11 (see Appendix C.1).

  2. All the RL evaluations are performed using 32 problems with 64 rollouts generated per problem.

We have since fixed the above and significantly edited the Appendix to match the same rigor and quality as the main body of the paper.

(W3/Q3)

We ran additional experiments to answer your question about whether the phrases emitted are causally responsible for the gradient alignment. Concretely, we explored if it is the phrase that caused the trajectory to change direction, or an internal state transition that causes both the model to emit a phrase and subsequently the trajectory to change direction.

First, we identified the set of phrases that have the most consistent impact on gradient alignment (regardless of the magnitude of the alignment) across all models and datasets through proper experimentation. In Figure 12 (see Appendix E), we visualize the phrases that correlate with the the largest uphill or downhill effect along the gradient direction.

To determine whether the phrases with a consistent effect (large or small) have a causal effect, we inject them in random positions on held out trajectories, and observe their resulting effect on the gradient alignment. We hope to see that not only do phrases that correlate with uphill directions push the gradient uphill in new settings, but also that the strength of that push is comparable (same for descending phrases). On one hand, if the phrase itself has no causal effect, then inserting it into a context where no internal mechanism caused it would do nothing. On the other hand, if the inserted phrases had causal effect, then they would still push the trajectory the same way (uphill or downhill) along the gradient direction.

In our experiments, we find that phrases that move the trajectory uphill when written by the model, continues to drive the model uphill even when injected into unrelated problems (31/31, 8/8, and 6/6 across the three models). Overall, the sign of the transition direction in the unrelated matches the transition direction of the model's own emission of the phrase (44/47 for R1-1.5B, 15/19 for Qwen3-4B, and 15/19 for Nemotron-4B). Additionally we find that the direction holds for the majority of injected phrases, and that the Spearman correlation between the magnitude of alignment of phrases naturally, as compared to when inserted artificially, is quite high (Spearman 0.95, 0.70, 0.81). Visualization of the complete set of results is provided in Figures 16 and 17 (see Appendix F).

Ultimately, we agree with you and it was not our intention to give the impression or implication that models implement gradient descent. We have gone through the paper again and removed any statements which may imply such a claim. Please let us know if there are any places where our wording is still too strong. We elaborate on this point more in the text below where we address your last question.

ModelSign preserved (all phrases)Climbing phrases still climbDescending phrases still descendCohen's κSpearman ρ
R1-1.5B44/4731/3113/160.850.95
Qwen3-4B15/198/87/110.600.70
Nemotron-4B15/196/69/130.590.81

(W4/Q4)

The goal of our paper was, as the title suggests, to understand and characterize the dynamics of reasoning in token based LLMs.

Our motivation for doing so was to reconcile the differences in perspective in literature surrounding reasoning via continuous iterative refinement and token based CoT. In iterative refinement, models are engineered to descend energy manifolds and slow down at attractor basins. Thus, by characterizing continuous dynamics, we sought to understand if an off the shelf LLM has implicitly learned any aspects of reasoning which are engineered into looped models. We specifically look at two aspects of energy descent: direction and speed.

With respect to direction, we do find that the models policy defines a flow over trajectories which align with the descent of local error gradients beyond chance (see Appendix D for the significant experimental evidence we have added to this effect). So, the flow of trajectories within our dynamical system can be better explained with our error field.

Now getting to your question about the speed of the dynamics, in our analysis we find that within our latent space, unlike some looped models, LLMs do not slow down at regions with lower energy. This observation manifests itself as a deeper issue with reasoning models: they do not know when to stop reasoning. We do not yet understand this behavior, but we hope that with our framework other researchers may further understand and possibly address this issue in the future. Thus, our analysis of speed is to characterize the flow speed and convergence of trajectories.

Ultimately, we wanted to observe and report evidence both in support of and against the possibility that the model implicitly performs an optimization process similar to energy descent. Our broader goal was to characterize reasoning dynamics and gain a better understanding of the underlying computations driving reasoning, an area that, to our knowledge, has not been explored in the literature.

Reviewer_yAkgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper analyzes LLM reasoning by compressing KV-cache states with a VAE and relating their trajectories to empirically estimated error landscapes. It finds the same directional behavior as gradient descent and investigates how token interventions, latent seeding, and reinforcement learning affect these dynamics and reasoning performance.

Critical strengths
  1. KV-cache VAE provides a useful framework for analyzing reasoning states, with nice visualizations.
  2. Token and latent interventions go beyond observational correlations, providing evidence about how perturbations influence reasoning trajectories and answer accuracy.
Critical weaknesses
  1. Evidence from the cross-model dynamics descent remains inconclusive: Figures 6 and 7 report noisy gradient estimates that do not clearly reproduce the main directional result, leaving its robustness across models uncertain.
  2. Evaluation is restricted to mathematical reasoning, leaving it unclear whether the observed dynamics generalize to non-mathematical tasks such as code generation or planning.
Questions for authors

See weakness

3 comments

Thank you for your feedback. Based on your suggestion, we ran additional experiments for all 3 models at scale which we believe greatly improved the rigor of our paper. For Nemotron-4b and Qwen3-4b we increased training data corpora by generating 4 rollouts for each of the 103k prompts in the DeepMath dataset. Crucially, we increased our number of trajectories and error field probe frequency to generate a denser error field. For both 4b models, we generated 128 trajectories per problem, and probed every 32 tokens.

(W1) We regenerated the perturbation plots (originally Figures 6 and 7) as a consolidated Figure 8. Here you can see that the results are consistent with our original Deepseek-R1-1.5B plots. Namely, during the perturbation region shown by the rectangular dotted box, off-policy trajectories falls below our baseline null, indicating a lack of alignment with the local error. However, past the perturbation window, all trajectories progressively descend the error field above chance (see further evidence in Figures 9 and 10). We also included statistical significance checks to prove this behavior in tables 14 and 15.

For convenience, the table below reports the alignment at=cos⁡(ut,−∇E^)a_t = \cos(u_t, -\nabla\hat E) (×10−3\times 10^{-3}) for the two 4B models, during the perturbation window and after it. During the window, the control and on-policy perturbations (temperature 1.0, 1.5) keep descending the gradient (CIs above 0), while the off-policy perturbations drop to chance (CIs include 0), and on Nemotron the two temperature-20 perturbations fall significantly below it. After the window, every perturbation returns to above-chance alignment in both models, with all 95% lower bounds above 0.

perturbationQwen3-4B MATH-500 duringQwen3-4B MATH-500 afterNemotron-4B MATH-500 duringNemotron-4B MATH-500 after
temp 0.6 (control)+14.2 [+9.0, +19.9]+10.1 [+7.2, +13.1]+13.0 [+10.2, +15.7]+12.7 [+10.7, +14.7]
temp 1.0+14.3 [+9.1, +19.8]+9.9 [+7.0, +12.9]+12.6 [+10.0, +15.3]+12.6 [+10.6, +14.7]
temp 1.5+14.2 [+9.0, +19.9]+9.6 [+6.9, +12.5]+12.4 [+9.9, +14.9]+12.8 [+10.8, +14.9]
temp 20, top-50+0.4 [-8.0, +9.5]+10.2 [+7.7, +12.9]-4.1 [-7.8, -0.4]+13.7 [+11.7, +15.7]
temp 20-8.1 [-26.3, +10.9]+7.1 [+0.8, +13.2]-8.0 [-14.2, -1.9]+12.5 [+10.5, +14.7]
another problem's reasoning+4.4 [-5.0, +13.8]+8.7 [+5.9, +11.6]+3.2 [-0.3, +6.6]+9.5 [+7.1, +11.7]

(W2) To validate the generalization of the observed dynamics, we extended our analysis to 2 additional benchmarks. To show general reasoning capabilities beyond math we chose GQPA diamond which comprises of a set of difficult science-based questions that require step by step reasoning. To highlight logical reasoning, we chose StateBench, a long context state tracking benchmark. Our results across both tasks show that during autoregressive generation of reasoning tokens, the residual field aligns closely with the error fields, validating our primary hypothesis. In Figure 8, Table 14, and Table 15 we show this gradient alignment. In Figure 8 we also validate the effect of perturbation corruption. In Figure 12 and 13 we show token scans for phrases with consistent effects. We also reproduce our observations on speed correlations with depth in Appendix Section G.

We hope that the above set of extensive experimental results provides enough confidence in the generalizability and consistency of the observations originally presented in the paper. Thank you once again, and we would be happy to respond to any additional questions you may have.

Thank you for the detailed response and the additional experiments. My concerns have been well addressed, and I will raise my score.

Thank you for taking the time to review our additional experiments. We apologize for submitting our response relatively late in the rebuttal period. We put considerable time into conducting these additional experiments and carefully addressing your concerns, and we are very glad that our response has helped resolve them.

We sincerely appreciate your consideration and constructive feedback. If there are any remaining concerns or questions that might affect your final assessment, please do not hesitate to let us know. We would be more than happy to address them. We will remain attentive and available until the rebuttal period concludes.

Reviewer_bP1jOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

There are two stories about how LLMs reason: discrete CoT tokens, and continuous latent refinement in purpose-built looped or energy-based models. Past work has not well studied at whether an ordinary token-based model is doing latent refinement under the hood. This paper treats the KV-cache prefix as the Markov state of a frozen model and trains a KV-cache VAE (reconstruction, KL, and a continuation loss) to map it to a 1024-d latent z. Over z they define an error field (forced-answer error rate) and a flow (latent step speed and direction). From this they claim that CoT behaves like gradient descent over basins, that phrases like think again climb the gradient while “hence” descends it, that seeding trajectories from different latent starting points improves pass@k, and that RL reduces the number of basins. Experiments use R1-1.5B, Qwen3-4B, and Nemotron-Nano-4B on math benchmarks.

Critical strengths
  • The framing and the probe are original. Using the full KV prefix as the state, instead of text or a single hidden state, is well argued. The VAE design with continuation loss is thoughtful, and the validity checks are good: reconstruction compared against a resampled trace, continuation compared against a resampled continuation, and a closure test with floor and ceiling references.
  • The token-injection experiment is well designed. The arms are graded by how off-policy they are (higher temperature, top-50 high-entropy, another problem’s reasoning, random tokens), and the distance-to-unperturbed-trajectories measure cleanly separates harmless noise from content that permanently corrupts the state. The phrase analysis selects phrases on half the traces and estimates effects on the held-out half, which avoids the obvious selection bias.
Critical weaknesses
  • Latent seeds reduce pass@1 in most cells (e.g., R1 MATH 0.636 → 0.535), and pass@8 drops on Nemotron GSM8K (0.967 → 0.899). The “∆ breadth” gains are partly mechanical, because pass@1 falls.
Questions for authors
  • Can you report gradient alignment against proper nulls, such as shuffled error labels or random directions within the same PC subspace, along with effect sizes and CIs for each model? How much of the step direction does −∇E explain beyond depth?
1 comment

(W1): Thank you for your comments w.r.t the “∆ breadth” column in Table 1. We agree with them and have now removed the column from the table.

To provide further confidence that our method (latent seeds) indeed improves response diversity in a meaningful way, we compared it against the best sampling methods we could find (11 techniques in total). Evaluation results over three different tasks (MATH-500, GSM8K, AMC23) for three different models (Deepseek-R1-Distill-Qwen-1.5B, Qwen3-4B, and Llama-3.1-Nemotron-Nano-4B) are summarized in Tables 9, 10, and 11 respectively (in Appendix Section C.1). We believe this comparison better highlights the improved diversity our algorithm yields as compared to other sampling techniques across multiple models and benchmarks.

(Q1): We thank the reviewer for the insightful question. We broke it down to the following two parts.

(a) Using proper controls (e.g., shuffling error labels associated with latents and sub-sampling 32-dimensions uniformly at random to represent the error direction) show that the residual field aligns with the error gradient with statistical significance.

(b) As reasoning goes on, two things happen together: the model generates more tokens (depth increases) and it often gets closer to the right answer. Since those two things are correlated, the alignment of the residual field with the error gradient could really be alignment with the depth field. Thus, we project the alignment onto the depth field and show that there is non-zero residual alignment with the error gradient.

For (a), we test alignments against the two nulls (random directions and shuffled error labels). We average alignment within each trace and then over traces and report 95% confidence intervals over problems, and test for alignment above the shuffled-label null with a one-sided t-test on per-problem means. We report effect size with Cohen’s d, and find that although effect size is small d≈0.1−0.3d \approx 0.1-0.3, the alignment is consistent and significant. The control is aligned with the gradient field above chance. We also analyze the impact of perturbation on alignment. Summarized in Tables 14, and 15, our results indicate that off policy tokens, within a perturbation window, fall to chance or worse alignment. However, there is a systematic bias to follow the error direction after the perturbation window ends (see Appendix Section D). We find these results consistent with the random direction null you suggested, which we plotted in Figure 8. We also updated our original Figure 3 in the main paper to include the null.

For (b), we project each step's direction utu_t onto the gradient estimate −∇E^-\nabla \hat{E} to obtain the steps direction that aligns downhill, ata_t. We then decompose ata_t into the direction aligned with local depth (⟨at,dt⟩\langle a_t, d_t \rangle) and whats left over, the residual term.

More precisely, at the midpoint xtx_t of a step, let

d_t = \frac{\nabla\hat\tau(x_t)}{\lVert\nabla\hat\tau(x_t)\rVert},$$ where $\nabla\hat\tau$ is the local depth gradient, defined just as the error gradient but with each label $E_j$ replaced by the depth $\tau_j$ of state $z_j$. With $c_t = g_t\!\cdot d_t$ and $r_t$ the unit vector along $g_t - c_t d_t$, which is orthogonal to $d_t$, $$g_t = c_t\, d_t + \sqrt{1-c_t^2}\; r_t ,$$ the alignment of $a_t$ then splits exactly into two terms, $$a_t = \langle u_t, g_t \rangle = c_t \langle u_t, d_t \rangle + \sqrt{1-c_t^2} \langle u_t, r_t \rangle$$ We ran experiments across MATH-500, GPQA-Diamond and a state-tracking task, roughly half of $a_t$ lies in the residual term (see the table below). In other words, there remains substantial (not zero) residual which indicates that the model isn't just moving forward in time and calling it descent. | Model / Dataset | $a_t$ | depth | beyond-depth | beyond-depth share | |:---|---:|---:|---:|---:| | R1-1.5B / MATH-500 | $+0.0262 \pm 0.010$ | $+0.0129 \pm 0.010$ | $+0.0134 \pm 0.005$ | $51 \pm 27\%$ | | R1-1.5B / GPQA-Diamond | $+0.0217 \pm 0.010$ | $+0.0109 \pm 0.010$ | $+0.0108 \pm 0.009$ | $50 \pm 39\%$ | | R1-1.5B / StateBench | $+0.0131 \pm 0.004$ | $+0.0051 \pm 0.004$ | $+0.0081 \pm 0.006$ | $61 \pm 33\%$ | | Qwen3-4B / MATH-500 | $+0.0072 \pm 0.002$ | $+0.0039 \pm 0.002$ | $+0.0033 \pm 0.003$ | $46 \pm 36\%$ | | Nemotron-4B / MATH-500 | $+0.0104 \pm 0.001$ | $+0.0017 \pm 0.001$ | $+0.0087 \pm 0.002$ | $84 \pm 13\%$ | In Figure 9, we explore the impact of perturbations on the depth and error components. We show that as expected, the depth aligned component during the perturbation window remains positively aligned as more tokens are being sampled with each step. However, as you sample off-policy tokens, the total alignment goes below zero, because the residual part ascends the error gradient and is larger than the depth component.

Comments on the paper

1 comment

Thank you to all the reviewers for your feedback!

We have attached an updated version of our manuscript to address all the comments that were brought to our attention. This includes training new VAE models, evaluating on additional benchmark tasks, new causal experiments, additional nulls and baselines, generating denser error fields, and other additional analysis. Furthermore, we have edited the main text and the appendix very carefully to make the paper easier to read (including new visualizations). We hope these changes bring in more rigor and clarity to the presentation.

We also have included a separate supplementary document which outlines all of the new experimental results in one place, for the reviewers' convenience.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.