acceptodds
Under review as a conference paper at ICLR 2027

LaDex: Latent State Divergence Incentivizes Self-Exploration in LLM Reasoning

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for enhancing the reasoning abilities of large language models (LLMs). However, vanilla group-based RLVR relies on sparse final-answer rewards, offering limited exploration guidance. Existing exploration algorithms primarily focus on token-level signals, which may reward superficial variation rather than guide the discovery of genuinely diverse and promising reasoning paths. With the observation that trajectories following different reasoning paths show greater divergence in latent space, we propose LaDex, a novel RL method that incentivizes LLM self-exploration through latent-space divergence. For a given prompt, LaDex first generates a group of reasoning trajectories, then segments each full reasoning trajectory into discrete stages and extracts their latent states. Within each reasoning stage, LaDex computes latent divergence by comparing latent states from different trajectories in the group. To balance exploration and exploitation, LaDex introduces a Latent Classifier Network (LCN) that monitors meaningful exploration within each reasoning stage and adjusts the latent-divergence signals accordingly. These signals are then aggregated into intrinsic rewards to guide policy optimization toward effective self-exploration. Experiments on diverse reasoning benchmarks demonstrate that LaDex significantly improves exploration efficiency and consistently outperforms strong baselines. Our code is available at \url{https://anonymous.4open.science/r/LLMRL-CDE2/

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning, Reasoning, Exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_DoPCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LaDex, a method that evaluates response diversity in latent space. It computes latent distances among the responses within a group at stage level. Then weights them by correctness through a Latent Classifier Network (LCN), and uses the result as an additional reward to guide learning. The method gives consistent improvements across multiple benchmarks and settings.

Critical strengths

The method is clearly structured, the problem is well defined, the experiments are thorough, and the appendix provides plenty of additional results.

Critical weaknesses

I do have some concerns:

  1. I’m not convinced that computing distances in latent space is necessary. The paper does not show whether a simple token-level distance would also work.
  2. The paper should briefly explain how temporal resampling is done. It directly addresses how stages are divided when responses differ in length, so it deserves a clear description. I'd also like some other details clarified. For example, how is splitting stages by “\n\n” guaranteed to work? The model’s use of this delimiter for segmentation is not consistent, and how the model produces the content of each stage is itself a form of diversity.
  3. Computing the diversity value as the average within-group distance is quite simple. Simple methods can of course be effective, but further explain how this distance behaves under different distributions, for example, when all responses in a group converge to the same paradigm, or when the distances between different paradigms are uneven...
  4. The paper shows that the average latent distance increases, but under the designed objective this is almost an inevitable outcome. It does not show a direct link between latent distance and actual diversity. Some examples, such as a clear increase in the number of distinct response paradigms, would help.
Questions for authors

See weaknesses above.

Reviewer_MuBDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LaDex, an exploration method for reinforcement learning with verifiable rewards. LaDex divides sampled responses into reasoning stages and pools their hidden states. At each aligned stage, it measures a trajectory’s average latent distance from other trajectories. A Latent Classifier Network (LCN) that is trained using final-answer correctness further gates these distances according to predicted success likelihood. The aggregated intrinsic reward is incorporated into GRPO through a bounded advantage adjustment. Experiments on math and webshop demonstrate that LaDex improves exploration efficiency and outperforms strong baselines.

Critical strengths
  1. The method is conceptually clear and integrates naturally with an existing RLVR pipeline. Combining diversity with a learned success-likelihood gate is a reasonable way to discourage unproductive deviations.

  2. The evaluation extends beyond a single metric or backbone. The authors report pass@k and mean accuracy, and explore a multi-turn agent environment.

Critical weaknesses
  1. Novelty Concerns. The broader ideas of latent-space exploration, correctness prediction from latent representations, and joint quality–diversity optimization already have closely related precedents. ReLaX [1] uses latent dynamics derived from the last-layer hidden states to regulate exploration and combines this with advantage-dependent control, while LTO [2] uses a latent classifier to predict correctness and guide optimization. This paper appears to be largely derived from a combination of the two previous works. A direct discussion and comparison is missing despite the conceptual overlap.

  2. The validity of latent distance as a reasoning-path diversity measure remains insufficiently established. Mean-pooled hidden states may reflect wording, formatting, or local token content in addition to reasoning strategy. Moreover, temporal resampling aligns relative progress rather than semantic operations. Two responses following the same solution may reach an intermediate result at different relative positions, producing substantial distance without representing distinct reasoning paths.

  3. The current ablations do not show whether latent divergence is necessary for the improvement. Table 2 shows that adding the LCN improves performance from 49.23 to 52.71. This demonstrates that the classifier helps when combined with divergence. However, it does not show whether divergence adds value beyond using the classifier alone.


[1] ReLaX: Reasoning with Latent Exploration for Large Reasoning Models

[2] Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts.

Questions for authors
  1. How are groups with identical extrinsic rewards handled?

  2. Why do you choose the last-layer hidden states? Have you tried the hidden states from other layers? Or perhaps the average across all layers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.