Characterizing Latent Reasoning Dynamics in LLMs

Reasoning in LLMs is typically explained through two distinct mechanisms: the autoregressive generation of discrete tokens, and the looped refinement of a continuous latent state. Both improve performance by increasing inference-time computation, yet the continuous computational dynamics underlying an off-the-shelf, token-based model's ordinary reasoning have not been directly observed. We bridge this gap by formalizing token-based reasoning as state transitions over KV-cache prefixes, compressing this high-dimensional space with a KV-cache VAE to map problem-specific error manifolds. Analyzing the resulting state-space dynamics, we find that standard autoregressive LLMs implicitly perform latent reasoning: their trajectories closely mimic gradient descent over basins. Specific tokens drive this descent, some rapidly exploiting the error gradient and others escaping local minima to explore, and trajectories quickly recover their descent path after off-policy perturbation. Applying these mechanistic insights, we causally improve inference-time scaling by optimizing the token-budget tradeoff between trajectory depth and breadth. Specifically, we demonstrate that initializing diverse latent "random seeds," much like in vanilla gradient descent, steers trajectories toward distinct local minima and yields a larger diversity in correct answers. Finally, we find that RL reshapes the latent space to contain fewer basins, along with faster and more likely convergence to basins correlated with improved performance.