acceptodds
Under review as a conference paper at ICLR 2027

SLPO: Scaling Latent Reasoning via a Surrogate Policy

Abstract

Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: a differentiable surrogate policy interface over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across two continuous latent reasoners, two backbones, and three held-out benchmarks, SLPO improves Pass@8 and Pass@16 in all 12 backbone–dataset settings, with gains of up to 12.07 percentage points. SLPO further transfers to soft-token inference and learns difficulty-adaptive computation, allocating longer latent trajectories to harder instances.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent reasoning, reinforcement learning, test-time scaling, chain-of-thought, large language models, policy optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_g5feOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SLPO, a training method for autoregressive latent reasoning models that use continuous vectors instead of discrete tokens for intermediate reasoning. It introduces a stopping gate and uses final-answer rewards to optimize a joint objective with three terms: a Gaussian surrogate score for latent transitions, stopping-time log-probabilities, and answer-token log-probabilities.

Critical strengths
  1. The Gaussian surrogate offers a way to train continuous latent transitions using final-answer rewards. The method can be applied to existing COCONUT and CODI models.

  2. The learned stopping gate lets the model adjust its reasoning length for each input. The extra forward passes used to estimate the Gaussian are only needed during training.

  3. Pass@8 and Pass@16 improve across all tested combinations of latent models, backbones, and datasets. The method also works with different policy optimizers and soft-token inference.

Critical weaknesses
  1. The paper does not show how much the latent Gaussian surrogate contributes beyond answer-token RL and stopping-gate training. I would like to see an ablation that removes only the surrogate term while keeping the rest of the setup unchanged.

  2. The argument in Appendix E relies on the surrogate and true policy score gradients being sufficiently close, but the paper neither establishes this condition nor checks it empirically. It is therefore unclear why the proposed isotropic Gaussian is a suitable surrogate for latent policy optimization.

  3. The COCONUT/CODI baselines use 6 latent steps, while SLPO allows up to 12 and sometimes uses nearly all of them. Without comparisons at matched inference budgets, it is difficult to tell how much of the improvement comes from the method itself and how much comes from additional computation.

  4. Appendix D.2 reports per-step training costs but leaves out the stopping-gate cold start, which requires decoding answers at multiple prefix lengths. Total training GPU-hours, including this stage, would give a clearer picture of the method’s computational cost.

  5. All evaluated backbones are small (<= 3B parameters). Could the authors also evaluate the method on a larger LLM?

Questions for authors

See the weaknesses above.

Reviewer_SK4FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on latent reasoning with reinforcement learning and propose SLPO. First, a stopping head is introduced to decide the end step of the latent reasoning. With supervised cold start and reinforce learning, the stopping head learns to assign different computation budget for input problems. Second, they use gaussian distribution to estimate the likelihood of latent tokens, which enables reinforcement learning to directly optimize the latent steps. Experiments on multiple mathematical benchmarks demonstrate the effectiveness of the proposed SLPO on different backbones and latent reasoning methods.

Critical strengths

1.This paper focus on two important problems in latent reasoning. I think adaptive length and calculating the likelihood of latent steps are valuable research directions

2.Estimating the likelihood of latent steps by gaussian model is interesting. This provide a general way for directly optimizing the latent reasoning process by reinforce learning methods.

3.The final results of SLPO outperforms other methods on most metrics.

4.The paper is well organized and easy to follow.

Critical weaknesses

1.SLPO seems less effective on CODI. In Table.2, SLPO only brings 0.05 improvements on GSM8k. Besides, most compared latent reasoning methods are lower than the baseline (CODI). Only Latent-SFT on MultiArith is better than CODI but CODI+SLPO still can not outperform it (96.82 vs 96.38). Authors should provide more evidence to verify the generalization of their SLPO on SOTA latent reasoning methods .

2.Authors should add ablation studies of each component of SLPO in Table.2: SLPO without stopping head and SLPO without Surrogate transition likelihood.

3.The authors should explain why the proposed stopping head can enable Test-time scaling with adaptive latent steps, as the CODI shows that 6 latent steps achieve its best performance.

Questions for authors

See Weaknesses

1 comment

Dear Reviewer SK4F,

As the discussion phase nears its end, we wanted to gently check in to see if our response has resolved your concerns. Specifically, following your constructive suggestions, we added the component ablations (removing the stopping head vs. removing the surrogate likelihood), clarified CODI's test-time scaling and compute-efficiency gains, and provided an instance-level horizon analysis showing why adaptive stopping improves over a fixed 6-step budget.

We would love to hear if these results address your questions, and we remain happy to provide any further clarifications before the discussion period concludes. Thank you again for your time and support!

Best regards,
The Authors

Reviewer_4SnTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SLPO brings outcome-reward RL to latent reasoning through a reasonable and reusable interface, with consistent empirical gains and a clear demonstration of difficulty-adaptive latent computation. However, its novelty is mainly compositional, its theoretical guarantees are weak, and the main hidden-state setting still lacks a direct Latent-GRPO baseline and broader validation.

Critical strengths

The paper addresses an important and well-motivated problem: latent reasoning lacks both an adaptive stopping interface and a tractable latent-transition likelihood for outcome-reward RL. SLPO provides a practical interface combining a stopping gate with a Gaussian surrogate likelihood.

The empirical results are consistent and reasonably broad. SLPO improves both Pass@8 and Pass@16 across all 12 backbone-dataset settings, and the gains transfer to soft-token latent inference.

The difficulty-adaptive computation analysis is convincing. The learned stopping gate allocates longer latent trajectories to harder problems (Pearson r ≈ 0.26–0.30), and the latent geometry analysis adds useful interpretability.

Critical weaknesses

The method is largely a composition of existing components—MC-dropout stochasticity, a stopping head, an isotropic Gaussian surrogate, and RLOO/GRPO. The core contribution is interface construction rather than a fundamentally new optimization principle.

The theoretical guarantees rely on strong and hard-to-verify conditions, such as score alignment and closeness between the surrogate direction and the true expected-reward gradient. These results provide limited practical assurance.

The main hidden-state recurrence experiments do not include a direct Latent-GRPO baseline, although the paper discusses Latent-GRPO and LEPO in related work and compares with LEPO in the soft-token setting. In addition, training is limited to GSM8K-Aug and evaluation is concentrated on grade-school math benchmarks, so broader generalization remains to be shown.

Questions for authors

Can the authors add a “fixed 12-step, no stopping gate” control to separate the effect of a larger budget from the effect of adaptive stopping?

How sensitive are the results to the inference stop threshold and to Pass@k sampling? Please report the gap between the fixed threshold 0.5 and the tuned threshold, and provide confidence intervals for Pass@k.

Can the authors ablate the variance path in the surrogate and report the cost of the stopping-gate cold start, for example in GPU hours and as a fraction of total training cost?

1 comment

Dear Reviewer 4SnT,

As the discussion period is drawing to a close, we wanted to gently follow up and check if our response and additional experiments have adequately addressed your questions. In particular, following your suggestions, we added the direct comparison with Latent-GRPO, the fixed 12-step control, the surrogate variance-path (σθ2\sigma_\theta^2) ablation, threshold/Pass@kk confidence intervals, and the cold-start GPU-hour breakdown, alongside clarifying the role of our theoretical results.

Please let us know if you have any remaining questions or if any part of our response could be further clarified—we would be more than happy to discuss!

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Rebuttal Additions and Consensus

Dear Area Chairs and Reviewers,

We thank all reviewers (4SnT, g5fe, and SK4F) for their feedback. We are grateful that Reviewer g5fe raised their rating after reviewing our new experiments, resulting in a unanimous positive assessment across all three reviewers (3 / 3 / 3, all acceptance).

Continuous latent reasoners carry intermediate computation in hidden vectors rather than token chains, matching or surpassing explicit Chain-of-Thought at shorter horizons. Yet while explicit CoT models improve beyond imitation through outcome-reward RL, latent reasoners remain imitation-bound because vector transitions lack a tractable likelihood for credit assignment and run under a fixed step budget. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL and test-time scaling to latent reasoning. By coupling a surrogate policy over stochastic latent transitions with a learned stopping gate, SLPO lets final-answer rewards optimize intermediate representations and adapt reasoning depth to problem difficulty—improving both test-time scaling and efficiency across latent reasoning paradigms and RL optimizers.

In the revision, we made four additions in response to the reviews:

  1. Component & Mechanism Ablations: We ablated the Gaussian surrogate, the stopping gate, and the surrogate variance gradient path. Each ablation underperforms full SLPO in both accuracy and Pass@kk—either halting prematurely without transition gradients or doubling latent steps without the gate—confirming that all three components are necessary.

  2. Matched Inference Budgets & Efficiency: Under matched inference budgets, SLPO outperforms fixed-length baselines across backbones, whereas running baselines for more steps degrades performance—showing that gains come from policy optimization rather than extra compute. An instance-level horizon analysis confirms that optimal reasoning depth varies across problems, explaining how SLPO improves test-time scaling while reducing average latent steps.

  3. Broader Scaling & SOTA Comparisons:

    • Scaling to Larger Backbones: On a 4B latent reasoner, SLPO improves accuracy and Pass@kk while reducing average latent length by 30%.
    • Comparison with Latent-GRPO: On competition math benchmarks, SLPO outperforms Latent-GRPO and other latent RL baselines across all metrics.
  4. Theoretical Clarity, Robustness & Training Cost: We clarified our two-layer theoretical analysis—separating assumption-free properties of the surrogate objective from local sensitivity bounds. We also verified that inference is robust to the stopping threshold across seeds, and reported training GPU-hours showing that the one-time gate cold start is a moderate fraction of training cost.

We thank the Reviewers and Area Chairs again for their time and support.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.