acceptodds
Under review as a conference paper at ICLR 2027

Next Thoughts Are Distributions: Generative Autoregressive Reasoning in the Latent Space

Abstract

Natural-language chain-of-thought makes reasoning observable, but it forces computation through a discrete and verbose interface. Continuous thoughts provide a more compact substrate, yet existing methods commonly predict them with mean squared error, yielding a unimodal Gaussian model. This is a poor fit for reasoning: the same context admits multiple valid strategies, and their average is useful for none. In this paper, we introduce Autoregressive Continuous Thought, a framework that models continuous next-thought prediction as a generative problem. A causal transformer provides the reasoning context, while a conditional diffusion head samples the next continuous thought from a multimodal distribution. Our approach preserves the autoregressive structure of language models while avoiding the mode collapse of regression-based continuous reasoning. Experiments on mathematical reasoning, planning, and algorithmic tasks demonstrate that our method improves accuracy, condenses reasoning traces, and produces diverse latent reasoning paths.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Rht7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Autoregressive Thought Flow (ATF) for continuous latent reasoning. A causal Transformer provides the context, and a lightweight flow head samples the next latent thought. Each sampled thought is fed back into the model. A frozen Causal-Register VAE provides training targets. Experiments show accuracy gains over a matched regression baseline and broader solution coverage through latent sampling.

Critical strengths
  1. ATF performs repeated denoising in a small flow head rather than repeatedly running the full backbone. The lightweight generation design has practical value.

  2. The matched MSE baseline shares the backbone, latent targets, and stopping rule with ATF. Greedy answer-decoding experiments also show that latent sampling alone can produce useful alternative answers. These results support generative next-thought prediction beyond text sampling. ATF improves the accuracy–latent-length trade-off in several settings.

Critical weaknesses
  1. The additional benefit of learning a flow generator is not fully isolated. The matched MSE comparison does not separate multimodal expressiveness from stochasticity, leaving unclear whether simpler dropout or noise-based sampling could provide similar gains. The LaDiR comparison also involves different training and RL procedures. Moreover, the mixed majority-vote results against MSE do not establish a consistent advantage in final-answer selection.

  2. The interpretation of the learned latent modes is not sufficiently established. The paper motivates its approach through multiple plausible next thoughts, but the evidence connecting latent modes to distinct, valid reasoning continuations is mainly qualitative. The PCA plots and decoded examples illustrate variation, but do not establish how consistently this variation reflects meaningful reasoning alternatives.

Questions for authors
  1. Does the CoLaR comparison isolate the prediction head? Under matched training and inference budgets, what additional benefit does learning a flow generator provide over adding dropout or Gaussian noise to the matched regression baseline? What evidence links this benefit to distinct, valid reasoning modes?

  2. Under the same answer-selection rule and total inference budget, do the Pass@k gains translate into higher accuracy for the selected answer?

  3. How are the discrete CoT budgets in Table 1 enforced, particularly the transition from reasoning to answer generation? Why does Qwen3-4B achieve 72.18% on GSM8K with a 500-token budget but only 47.84% with a 1,000-token budget? (I am mainly curious about the reason for this result, and this is not a criticism of the core method.)

4 comments

We thank Reviewer Rht7 for the thoughtful review and for recognizing our lightweight Flow Head and matched MSE design. Below we address all concerns via additional experiments on stochastic MSE baselines, latent modes, and matched-budget selection, plus clarifications on CoLaR/LaDiR and Table 1 CoT budgets.


W1. Demonstrating the Isolated Benefits of Learning a Flow Generator

Additional matched noise-based MSE comparisons. We thank the reviewer for suggesting the MC-dropout and Gaussian-noise baselines. Controlled experiments on our SFT-only matched MSE model (Q1 below) highlight two advantages of ATF: (1) mutually reinforcing per-path correctness (Pass@1) and multi-sample exploration (Pass@16) with no test-time tuning, whereas both MSE perturbations face an accuracy–exploration trade-off; and (2) providing a learnable multimodal policy, whereas neither heuristic perturbation defines a trainable distribution, making extension to latent RL difficult.

Aligned comparison with LaDiR. For a direct RL-free comparison in Table 2 (LLaMA-3.1-8B), we report Pass@100 for SFT-only ATF (Flow SFT) alongside LaDiR:

Method (Table 2, Pass@100)MATH-500GSM8KOlympiadBench
LaDiR w/o Stage 235.862.610.5
LaDiR63.793.715.3
ATF (Flow SFT)84.698.955.9
ATF (+ RL, Table 2)83.297.652.4

Before any RL, Flow SFT already surpasses both LaDiR w/o Stage 2 and full LaDiR (as well as all other Table 2 baselines) in Pass@100, while RL subsequently concentrates probability onto correct paths to raise Pass@1.

Expanded majority-vote comparison against MSE. Beyond Table 11, evaluating majority voting across the three Table 1 benchmarks (GSM8K, MATH-500, OlympiadBench) on Qwen3-4B ([Q2, W1] below) shows that ATF consistently outperforms matched deterministic MSE in Maj@16.


W2. Quantitative Analysis of Learned Latent Modes

To move beyond qualitative examples and isolate variation arising purely from continuous latent thoughts, we evaluate Qwen3-4B across all 500 MATH-500 problems under greedy answer decoding (T=0T=0, K=16K=16), comparing ATF against deterministic MSE and MSE + Gaussian Noise. Beyond exact-string diversity, an LLM judge (Qwen3-30B-A3B-Instruct, T=0T=0) groups each problem's correct rollouts by underlying mathematical strategy (ignoring wording or reordered arithmetic; counted conservatively only when two randomized presentation orders agree, 90.6% agreement):

Method (Qwen3-4B, MATH-500, greedy T=0)Pass@1Pass@16≥2 Distinct Correct Texts (%)≥2 Distinct Approaches (%, LLM Judge)
Deterministic MSE76.876.80.00.0
MSE + Gaussian Noise77.283.470.226.0
ATF (Flow Head)78.788.883.033.8

With text-decoding randomness turned off (T=0T=0, where deterministic MSE yields a single trajectory per problem), ATF consistently translates latent variation into distinct valid reasoning paths across the dataset: it produces ≥2\ge 2 distinct correct solutions on 83.0% of problems (+12.8 over Gaussian noise, paired 95% CI excluding 0) and ≥2\ge 2 substantively different mathematical approaches on 33.8% of all problems (+7.8 over Gaussian noise, paired 95% CI [+2.6, +13.0]). We will add this dataset-level quantitative analysis to Section 4 in the revision.


Q1. CoLaR Head Isolation and Flow vs. Dropout / Gaussian Noise on Matched MSE

CoLaR (Table 3) compares complete systems under a shared backbone and dataset rather than isolating the prediction head alone, as CoLaR uses its own pooled-embedding targets and training recipe rather than our frozen CR-VAE.

To strictly isolate the prediction head, we evaluate additive Gaussian noise and MC-dropout (p=0.03p=0.03 and 0.200.20, following You et al., 2026) on our matched Qwen3-4B MSE baseline (K=16K=16, T=0.1T=0.1):

Method (Qwen3-4B, SFT-only, no RL)MATH-500OlympiadBench
Pass@1Pass@16Pass@1Pass@16
Deterministic MSE77.087.840.858.7
MSE + Gaussian Noise77.590.040.459.6
MSE + MC-Dropout (p=0.03)76.590.839.960.8
MSE + MC-Dropout (p=0.20)66.193.231.259.2
ATF (Flow Head)78.792.241.962.5

Compared with adding MC-dropout or Gaussian noise to matched MSE, learning a Flow Head provides two key benefits:

  1. Mutually reinforcing per-path correctness (Pass@1) and multi-sample exploration (Pass@16): Over deterministic MSE (77.0 / 40.8 Pass@1; 87.8 / 58.7 Pass@16), ATF simultaneously achieves the highest Pass@1 (78.7 / 41.9) and significantly raises Pass@16 (92.2 / 62.5; at greedy T=0T=0, MATH-500 Pass@1 → Pass@16 rises 76.8 → 88.8 vs. flat 76.8 for MSE). By contrast, Gaussian noise and mild dropout (p=0.03p=0.03) under-explore (90.0 / 59.6 and 90.8 / 60.8), while strong dropout (p=0.20p=0.20) raises MATH-500 coverage only at a 10.7–14.5 point drop in Pass@1 (and still trails ATF on OlympiadBench Pass@16: 59.2 vs. 62.5).
  2. Providing a learnable multimodal policy for RL: MC-dropout lacks a tractable trajectory likelihood, and Gaussian noise adds only fixed, unlearnable isotropic variance around a single MSE prediction. ATF's Flow Head instead parameterizes a trainable conditional policy over next thoughts that latent-trajectory RL directly optimizes, lifting 8B Pass@1 by +5.0 to +12.5 points (W1) and more than doubling answer-only RL's gain on 1B GSM8K (+3.8 vs. +1.8).

Link to distinct, valid reasoning modes. While strong dropout (p=0.20p=0.20) can raise MATH-500 Pass@16 at a steep cost to single-sample accuracy, ATF achieves strong Pass@16 alongside the highest per-path Pass@1 (78.7 / 41.9), confirming valid paths, and the greedy-decoding analysis in W2 shows that these trajectories yield genuinely distinct mathematical approaches (33.8% vs. 26.0% for the Gaussian noise baseline).


[Q2, W1] Selected-Answer Accuracy and Comparison with MSE Under Matched Budget

Yes, the Pass@kk gains consistently translate into higher selected-answer accuracy. We adopt majority voting (Maj@16) as the shared answer-selection rule, and to isolate the contribution of the Flow generator without RL confounds as pointed out in W1 (extending Table 11), we compare Qwen3-4B (SFT-only, T=0.1T=0.1) against matched deterministic MSE across the three large Table 1 benchmarks (GSM8K, MATH-500, OlympiadBench), reporting Maj@16 alongside sequential 16-rollout inference time (Time (s), following Appendix E Table 9 on 1 RTX PRO 6000 at batch size 1):

MethodGSM8KMATH-500OlympiadBench
Maj@16Time (s)Maj@16Time (s)Maj@16Time (s)
Deterministic MSE91.062.179.8113.744.9128.4
ATF (Flow Head)91.663.982.4121.847.2139.4
  • Consistent selected-answer gains over deterministic MSE: ATF achieves higher Maj@16 across all three benchmarks.
  • Strictly smaller or matched total inference budget: Of the 1.8–11.0 s total difference across 16 rollouts, the 24 Flow Head steps add <1.3<1.3 s (<2%<2\% of runtime), with the rest coming from 3–46 slightly longer answer tokens per rollout. Moreover, even at half the sampling budget (Maj@8: 32.0 / 60.9 / 69.7 s, ∼0.53×\sim 0.53\times of MSE's 16-rollout time), ATF reaches 91.6 / 82.1 / 46.4, already surpassing MSE's Maj@16 across all three benchmarks.

Q3. Discrete CoT Budget Enforcement in Table 1 (500 vs. 1,000 Tokens on GSM8K)

We thank the reviewer for pointing out an insightful detail that helps us refine our budgeted-CoT evaluation protocol! For capped discrete CoT (≤500\le 500, ≤1,000\le 1{,}000), our evaluator runs two phases: Phase 1 generates up to the thinking budget (appending </think> to transition to the answer only if </think> has not yet appeared), and Phase 2 generates up to 1,024 final-answer tokens.

Upon inspection, we found that when a problem is simple enough for the model to naturally finish both thinking and its final answer (EOS) within Phase 1, Phase 2 still continued generating past EOS, appending extra text whose trailing \boxed{} could overwrite the already-completed answer. Because GSM8K requires much shorter reasoning traces than the harder competition benchmarks (804 thinking tokens unbounded at 4B vs. 2,020–3,721 elsewhere), raising the cap from 500 to 1,000 caused most 4B GSM8K problems (958 / 1,319, vs. 136 at ≤500\le 500) to finish inside Phase 1, producing this reversal on GSM8K. Stopping cleanly at each item's first EOS restores the expected monotonic trend on 4B GSM8K (79.15% at ≤500\le 500 → 91.74% at ≤1,000\le 1{,}000, matching unbounded 91.21%). We will update Table 1 with this refined protocol in the revision, and all conclusions regarding ATF remain unchanged.

Thank you for the additional experiments, which help alleviate my main concerns. I have increased my score accordingly. I would encourage the authors to add majority-vote results for the stochastic MSE baselines under the same inference budget, and to update the paper with the corrected CoT results and revise the related conclusions as appropriate.

Reviewer_LwKSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Autoregressive Thoght Flow (ATF) to address two limitations of existing continuous reasoning methods, their inability to capture multiple modes of the next thought, and the loss of autoregressive generation in diffusion-based continuous reasoning approaches. Instead of regressing continuous thought latents directly with MSE or simple unimodal gaussian, ATF models each next thought with a diffusion head conditioned on the backbone’s hidden state hth_t, which allows it to represent multimodal distributions. Because thoughts are generated one at a time and fed back into the model, autoregressive generation is preserved. This leads to improvements on reasoning benchmarks, and Table 2 shows particularly large gains in Pass@100 over prior methods.

Critical strengths
  • Clear and well-motivated problem. The paper precisely addresses both the multimodality problem and the need to preserve autoregressive generation in previous continuous reasoning methods.

  • Simple design that preserves the pretrained backbone. By attaching a diffusion head without changing the backbone architecture, ATF can be easily applied by fine-tuning a pretrained model.

  • Credit assignment for RL. The paper also shows how the continuous thought process can be used for RL, separating credit assignment between the latent trajectory and the decoded text answer.

  • Strong results. ATF outperforms the matched MSE baseline in most settings, and shows especially large improvements in Pass@100.

Critical weaknesses
  • No ablation on the RL coefficients. The RL objective combines a latent term and a text term weighted by λz\lambda_z and λy\lambda_y, but no ablation over these coefficients is reported.

  • No ablation on the number of diffusion steps. Diffusion models allow the computation of the generation process to be adjusted at inference time, so varying the number of denoising steps in the diffusion head could serve as a form of test-time compute scaling. However, ablation study about denoising step results are not reported.

  • Missing SFT-only results in Table 2. The ATF model in Table 2 uses joint continuous latent & text RL, while at least some baselines (e.g., Coconut, Soft Thinking) do not use RL. To isolate the contribution of RL to the Pass@100 gap, SFT-only ATF results under the same setting are needed.

  • Unclear effectiveness of RL in Table 2. Only post-RL results are reported, and Pass@100 is very high. When Pass@100 is high, correct rollouts are frequently sampled, so RL should receive dense informative reward signals and could be expected to substantially improve Pass@1. However, Pass@1 remains comparable to or lower than the baselines (e.g., 42.2 vs. 46.2 for LaDiR on MATH-500). The paper does not explain why RL does not translate this coverage into better single-sample accuracy.

Questions for authors
  • How sensitive are the results to the RL coefficients λz\lambda_z and λy\lambda_y? An ablation over these would clarify which term drives the improvement.

  • What happens when the number of diffusion steps is varied? It would be helpful to see whether more steps improve accuracy or diversity.

  • Could you provide the SFT-only results under the Table 2 setting? This result may help to understand the pass@100 gain separately.

  • Why does RL yield relatively small gains in Pass@1 despite the very high Pass@100?

4 comments

We thank Reviewer LwKS for the positive review and for recognizing our motivation, backbone-preserving architecture, RL credit assignment, and multi-sample gains. Below we address all four weaknesses and questions via controlled experiments on RL coefficients λz,λy\lambda_z, \lambda_y [Q1·W1], inference denoising steps [Q2·W2], SFT-only Table 2 results [Q3·W3], and how RL and lightweight voting translate coverage into single-answer accuracy [Q4·W4].


[Q1·W1] Ablation and Sensitivity of RL Coefficients (λz\lambda_z and λy\lambda_y)

Following the reviewer's suggestion, we ablate λz\lambda_z and λy\lambda_y in Eq. (7): the latent-policy term (λz\lambda_z) is the primary driver of the RL gain, while joint optimization achieves the highest accuracy and remains robust across a wide range of coefficient ratios.

We sweep (λz,λy)(\lambda_z, \lambda_y) on the Table 3 backbone and dataset (LLaMA-3.2-1B-Instruct, GSM8K-Aug), training each setting for 4,000 RL iterations from the shared Flow SFT checkpoint (greedy answer decoding):

Methodλz\lambda_zλy\lambda_yGSM8K Acc.
Flow SFT––25.7
+ answer-only RL00.127.5
+ latent-only RL0.5029.2
+ latent and answer RL0.50.129.8
0.50.529.3
1.00.129.7
  • Directly shaping stochastic latent trajectories with latent RL is essential: Refining answer decoding from latent thoughts alone (+ answer-only RL) yields +1.8 (25.7 → 27.5), whereas directly shaping the stochastic latent trajectories (+ latent-only RL) nearly doubles the gain to +3.5 (25.7 → 29.2), and combining both reaches 29.8 (+4.1).
  • Low sensitivity: Across all settings with λz>0\lambda_z > 0—spanning a 5×5\times increase in λy\lambda_y and 2×2\times in λz\lambda_z—accuracy stays tightly within 29.2–29.8.

[Q2·W2] Ablation on the Number of Inference Denoising Steps

We vary inference denoising steps from 8 to 96 under the Table 3 setup (LLaMA-3.2-1B-Instruct, GSM8K, greedy answer decoding over 3 seeds; trained at 24 steps): increasing denoising steps up to ~24–48 improves accuracy on the Flow SFT model, whereas after RL accuracy stabilizes and diversity shows no significant trend across step counts.

(a) Accuracy (mean GSM8K accuracy):

Method81224 (default)4896
Flow SFT23.624.325.726.426.1
+ latent and answer RL29.029.430.030.030.5

Under Flow SFT, the learned next-thought distribution remains broad and diffuse across diverse training continuations, so few steps can land in ambiguous intermediate regions, whereas finer denoising resolves valid modes more accurately. Once RL sharpens probability mass onto high-reward modes, the transport paths become well-concentrated, making sampling insensitive to step count.

(b) Diversity (mean number of distinct predicted answers per question across 3 seeds, max = 3):

Method81224 (default)4896
Flow SFT2.492.472.392.382.39
+ latent and answer RL2.262.262.212.212.19

Across both Flow SFT and + latent and answer RL, rollout diversity remains nearly flat as denoising steps increase.


[Q3·W3] SFT-Only Results Under the Table 2 Setting

Before any RL, SFT-only ATF (Flow SFT) already establishes the full Pass@100 coverage advantage, outperforming every Table 2 baseline.

Evaluating the pre-RL Flow SFT checkpoint under the exact Table 2 protocol (LLaMA-3.1-8B, DART-MATH, 100 latent paths per question, Table 2 evaluation):

MethodMATH-500GSM8KOlympiadBench
Pass@1Pass@100Pass@1Pass@100Pass@1Pass@100
Flow SFT29.784.673.598.97.755.9
+ latent and answer RL42.283.284.697.612.752.4

Without RL, Flow SFT already exceeds both RL-free baselines (e.g., Coconut 39.3 / 74.3 / 6.3, Soft Thinking 46.7 / 86.6 / 13.1) and the strongest Table 2 baseline (63.7 / 93.7 / 17.8) by +20.9 / +5.2 / +38.1 points in Pass@100, while RL plays a complementary role by concentrating probability mass onto correct trajectories to substantially improve Pass@1. We will add this Flow SFT row to Table 2.


[Q4·W4] Effectiveness of RL on Pass@1 and Comparison with LaDiR

Effectiveness of RL. As shown in Q3, RL raises Pass@1 from 29.7 → 42.2 (+12.5) on MATH-500, 73.5 → 84.6 (+11.1) on GSM8K, and 7.7 → 12.7 (+5.0) on OlympiadBench, demonstrating that RL effectively concentrates probability mass onto valid trajectories.

Comparison with LaDiR. ATF already matches LaDiR's Pass@1 on GSM8K (84.6 vs. 84.8) and OlympiadBench (12.7 vs. 12.9), trailing slightly only on MATH-500 (42.2 vs. 46.2). This reflects a natural coverage–concentration trade-off: step-wise autoregressive sampling preserves a broader space of reasoning paths (+19.5 to +37.1 higher Pass@100 than LaDiR), whereas block-level denoising concentrates more tightly on fewer modes. Moreover, ATF offers much higher compute efficiency: whereas LaDiR runs the full 8B backbone at every denoising step (50 full-backbone passes), ATF runs the 8B backbone only once per thought and executes all 24 denoising steps inside a lightweight 233M Flow Head. Leveraging this efficiency, we evaluate majority voting under the Table 2 setting:

MethodGFLOPs / thoughtMATH-500GSM8KOlympiadBench
LaDiR697.946.284.812.9
ATF25.242.284.612.7
ATF + Majority Voting @ 4100.846.686.815.1

With just 4 samples (Maj@4), ATF surpasses LaDiR across all three benchmarks while still using 6.9× fewer per-thought latent FLOPs than a single LaDiR run.

Thank you for the detailed response and the additional experiments. W1–W3 are addressed, but my main concern in W4 remains.

W1–W3. These are resolved by the coefficient sweep, the denoising-step ablation, and the Flow SFT results under the Table 2 setting.

W4/Q4. My concern is not about how much RL improves Pass@1, but that the resulting numbers themselves are hard to explain. RL generates multiple answers per problem and reinforces the trajectories with high reward. Therefore, when Pass@100 is high, RL receives a dense learning signal, which should raise Pass@1 substantially. Prior work also shows that RL mainly sharpens the policy toward correct trajectories that the model can already sample, so the model's Pass@k largely determines how far RL can raise Pass@1 [1]. From this perspective, ATF, whose Pass@100 is much higher than that of LaDiR (84.6 before RL and 83.2 after RL, vs. 63.7 on MATH-500), should be expected to reach a higher Pass@1 after RL. However, ATF after RL has a lower Pass@1 than LaDiR (42.2 vs. 46.2). This result is not logically convincing, and it raises the concern that the proposed RL method may be flawed or insufficient. Could you explain why ATF reaches a lower Pass@1 than LaDiR after RL, despite its much higher Pass@100?

Overall, I maintain my score for now. W4 questions a fundamental assumption, namely whether the proposed RL method works properly, so I consider it important. My final assessment will depend on the response to W4.

[1] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025.

AuthorsOpenReview ↗

Thanks the reviewer for the follow-up and for clarifying your concern in W4.

Thank you for the follow-up and for clarifying your concern in W4.

Clarification of “RL Algorithm” in LaDiR. First, we have to clarify that LaDiR's Stage 2 [2] is not an outcome-reward RL algorithm: instead of optimizing a reward, Stage 2 generates its own latents and supervises the text part with ground-truth answers, which is more like distillation than RL. Consequently, compared with Stage 1, Stage 2 elevates both Pass@1 accuracy (30.7 →\to 46.2 on MATH-500) and Pass@100 coverage (35.8 →\to 63.7), matching the distillation behavior in [1] (Sec. 4.2) referred by the reviewer. In contrast, ATF's RL follows the distribution-sharpening behavior of RLVR in [1] (Sec. 3.1 & 4.1), where Pass@1 rises substantially (29.7 →\to 42.2) while Pass@100 slightly decreases (84.6 →\to 83.2). We will clarify in the revision that LaDiR uses supervised Self-Rollout Training rather than outcome-reward RL.

Limitation and Revision Plan. We also thank the reviewer for this question, which inspires us to consider similar self-distillation methods as a candidate to more effectively sharpen the distribution. In the revision, we will note the limitation of relying solely on a sparse sequence-level reward for the text part, and leverage the correct traces discovered by ATF's high-Pass@KK rollouts, using their token-level entropy as a dense reward on the text part to further improve Pass@1.

Additional Comparison with Baseline with RL (LEPO). Moreover, to further verify the effectiveness of our RL method, we also compare ATF with a similar outcome-reward RL method for latent reasoning, LEPO [3]:

MethodMATH-500GSM8KOlympiadBench
Pass@1Pass@16Pass@100Pass@1Pass@16Pass@100Pass@1Pass@16Pass@100
LEPO [3]34.172.487.279.496.498.59.136.456.2
ATF39.574.787.282.696.598.911.637.252.5

As shown above, ATF consistently outperforms even LEPO's best checkpoint in Pass@1 and Pass@16 across all three benchmarks, confirming the effectiveness of our proposed RL method.


References

  • [1] Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, NeurIPS 2025.
  • [2] Kang et al., LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning, ICLR 2026.
  • [3] Zhou et al., LEPO: Latent Reasoning Policy Optimization for Large Language Models, Findings of ACL 2026.
Reviewer_s6NKOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Autoregressive Thought Flow (ATF), which generates the next latent thought from a conditional distribution. It compresses reasoning traces using a CR-VAE and autoregressively generates the resulting latent representations with a Flow Head to explore multiple reasoning trajectories. Experiments on mathematical reasoning benchmarks show improvements particularly in Pass@100.

Critical strengths

The paper gives the limited exploration of deterministic latent reasoning through a concrete framework for learning next-thought distributions in a compressed latent space. The Pass@100 gains that persist under greedy answer decoding support the contribution of latent trajectory sampling to successful reasoning. Evaluating coverage across multiple attempts, alongside single-attempt accuracy, also provides useful evidence about the method’s exploration capabilities.

Critical weaknesses

The respective contributions of flow-based generation and reinforcement learning to the main Pass@100 gains are not sufficiently isolated. Without direct comparisons against LEPO [1], which combines stochastic latent reasoning with reinforcement learning, or parallel latent exploration methods [2], the added value of CR-VAE plus Flow remains unclear. Furthermore, limited evaluation at small sampling budgets, after candidate selection, and under matched inference time makes it difficult to assess whether the substantial Pass@100 gains translate into practical benefits.

[1] Yuyan Zhou, Jiarui Yu, Hande Dong, Zhezheng Hao, Hong Wang, Jianqing Zhang, and Qiang Lin. 2026. LEPO: Latent Reasoning Policy Optimization for Large Language Models. Findings of ACL 2026. https://aclanthology.org/2026.findings-acl.707/

[2] Runyang You, Yongqi Li, Meng Liu, Wenjie Wang, Liqiang Nie, and Wenjie Li. 2026. Parallel Test-Time Scaling for Latent Reasoning Models. ACL 2026. https://aclanthology.org/2026.acl-long.2069/

Questions for authors
  1. How much does reinforcement learning contribute to the gains?
    Under the Table 2 setting, could you compare the model before reinforcement learning, after answer-only reinforcement learning, and after reinforcement learning of both latent reasoning and answer generation? Reporting both Pass@1 and Pass@100 would clarify which training stage expands the range of problems the model can solve.

  2. Do the gains persist with only a few attempts? Could you report Pass@K curves under the Table 2 setting, including K = 2, 4, and 10? Pass@1 and Pass@100 alone do not reveal whether the benefits emerge at small sampling budgets or require many attempts.

  3. Do the Pass@100 gains translate into higher final-answer accuracy under the same inference budget? In light of prior work on parallel latent exploration and candidate selection [2], could you report accuracy after selection, such as majority voting, together with total inference time under the Table 2 setting? Accounting for repeated Flow Head evaluations and answer generation would clarify how improved candidate coverage translates into practical accuracy and efficiency.

  4. Could you add a direct comparison with LEPO [1]? LEPO also combines stochastic latent thought generation with reinforcement learning of both latent representations and discrete tokens, and is already cited in the manuscript, making it an important baseline for assessing the proposed architecture’s added value. At minimum, please report Pass@1 and Pass@K using a common backbone and evaluation protocol; a comparison of complete systems would still be informative if differences in training conditions are clearly documented.

4 comments

We thank Reviewer s6NK for the constructive feedback and for recognizing the exploration capabilities of ATF. Below we address all concerns with controlled experiments covering Flow SFT vs. RL isolation (Q1), small-budget Pass@KK scaling (Q2), candidate selection and inference time (Q3), direct comparisons with LEPO [1] (Q4), and parallel latent exploration [2].


Q1. How do Flow SFT and RL each contribute? (before RL, answer-only RL, joint RL)

Following the reviewer's suggestion, we ablate Flow SFT, answer-only RL, and latent & answer RL across the Table 2 (8B) and Table 3 (1B) settings. Our conclusion is: Flow SFT and RL play complementary roles: Flow SFT establishes broad Pass@100 solution coverage through multimodal exploration, while RL concentrates probability mass onto correct trajectories to substantially improve Pass@1 (mean) accuracy.

(a) Table 2 setting (LLaMA-3.1-8B, DART-MATH, 100 latent paths per question, Table 2 evaluation):

MethodMATH-500GSM8KOlympiadBench
Pass@1Pass@100Pass@1Pass@100Pass@1Pass@100
Flow SFT29.784.673.598.97.755.9
+ latent & answer RL42.283.284.697.612.752.4
  • Before any RL, Flow SFT's Pass@100 already exceeds the best Table 2 baseline on every benchmark (63.7 / 93.7 / 17.8).
  • RL raises Pass@1 by +12.5 / +11.1 / +5.0 points while keeping Pass@100 within 1.3–3.5 points. On MATH-500, questions solved by all 100 samples rise from 2 to 59.

(b) Answer-only vs. latent & answer RL. Due to time and compute constraints, we cannot retrain 8B RL in Table 2 during the rebuttal period, so we ablate all three stages on the Table 3 backbone and dataset (LLaMA-3.2-1B-Instruct, GSM8K-Aug), training each RL variant for 4,000 iterations from the shared Flow SFT checkpoint and evaluating under the Table 2 multi-sample protocol (100 latent paths, T=0.7T=0.7):

MethodGSM8K
Pass@1Pass@100
Flow SFT25.579.5
+ answer-only RL27.379.8
+ latent & answer RL29.378.5

Consistent with 8B, Flow SFT already establishes the full Pass@100 coverage (~79.5%), while RL leaves the solvable set unchanged and instead improves Pass@1: refining answer decoding alone (+ answer-only RL) yields +1.8, whereas directly shaping the stochastic latent trajectories (+ latent & answer RL) more than doubles the gain to +3.8.


Q2. Do the gains persist with only a few attempts (K=2,4,10K = 2, 4, 10)?

Yes, substantial coverage gains emerge immediately at K=2,4,10K = 2, 4, 10, surpassing the 100-sample ceiling (Pass@100) of every Table 2 baseline within just 2 to 10 attempts.

Since prior methods in Table 2 do not report intermediate KK, we compare ATF's unbiased Pass@KK curve (from the 100 Table 2 trajectories per question, T=0.7T=0.7) directly against the strongest baseline at K=100:

BenchmarkK=1K=2K=4K=8K=10K=16K=100K=100 (Best Baseline)
MATH-50042.251.359.165.867.771.583.263.7 (LaDiR, Kang et al., 2025)
GSM8K84.688.891.793.794.295.297.693.7 (LaDiR, Kang et al., 2025)
OlympiadBench12.718.023.829.631.535.652.417.8 (Disc. Latent, Su et al., 2025)

Because prior latent methods suffer from mode collapse and gain little over 100 attempts, ATF's diverse trajectories already surpass every baseline's K=100 score by K=2 on OlympiadBench (18.0 vs. 17.8; LaDiR: 15.3) and by K=8–10 on MATH-500 (65.8 / 67.7 vs. 63.7) and GSM8K (93.7 / 94.2 vs. 93.7).


Q3. Do Pass@100 gains translate into higher accuracy after selection with total inference time?

Yes. Even with verifier-free majority voting, ATF surpasses every Table 2 baseline's single-answer accuracy by K=4K=4, while its lightweight Flow Head adds acceptable latency overhead relative to answer decoding.

(a) Majority-voting accuracy and total inference time, Table 2 setting (LLaMA-3.1-8B on 1 RTX PRO 6000 GPU, all KK trajectories per question generated in one parallel batch):

Method / BudgetMATH-500GSM8KOlympiadBench
Maj@KTime (s)Maj@KTime (s)Maj@KTime (s)
Best Baseline (K=1)46.2 (LaDiR, Kang et al., 2025)–85.9 (TaH+, Fu et al., 2025)–13.3 (Disc. Latent, Su et al., 2025)–
ATF (K=1)42.27.984.65.012.79.4
ATF (K=4)46.611.586.86.515.114.1
ATF (K=8)49.113.687.67.816.817.7
ATF (K=10)49.715.087.98.517.219.6
ATF (K=16)50.617.688.18.717.823.2
ATF (K=100)52.673.788.637.917.9107.5
  • By K=4, majority voting already beats every Table 2 baseline's Pass@1 while still using 6.9× fewer per-thought latent FLOPs than a single LaDiR run (100.8 vs. 697.9 GFLOPs/thought, Table 10). Furthermore, by K=16, ATF's oracle-free Maj@16 on MATH-500 (50.6) exceeds the 100-sample oracle ceiling (Pass@100) of all six non-diffusion baselines (best: TaH+ at 49.4), and on OlympiadBench (17.8) matches or beats every baseline's Pass@100 including LaDiR (15.3), proving that giving those baselines the same multi-sample budget cannot close the gap.
  • Beyond majority voting, as shown in [2] referenced by the reviewer, it is only a conservative lower bound that can be substantially improved by verifier-guided selection (e.g., LatentRM in [2]), and ATF's much higher Pass@100 upper bound (83.2 / 97.6 / 52.4 vs. best baseline 63.7 / 93.7 / 17.8) provides a far larger ceiling for such posterior verification than prior methods.

(b) Per-rollout inference latency breakdown, Table 2 setting (accounting for all 24 Flow Head evaluations per latent step and full answer decoding):

BenchmarkMean LengthLatency Breakdown per Rollout (s)
LatentAnswerLatent (24× flow steps)Answer DecodeTotal
MATH-50091.9314.93.224.667.90
GSM8K63.9177.42.292.664.97
OlympiadBench106.7395.33.485.889.38

Unlike the prior SOTA method LaDiR, which runs the full 8B backbone at every denoising step (estimated ~207 ms / 697.9 GFLOPs per thought at 50 steps shown in Appendix E, Table 10), our decoupled lightweight Flow Head (233M) allows ATF to run all 24 denoising steps much faster (35.8 ms / 25.2 GFLOPs, which is ~5.8× faster and 27.7× fewer FLOPs). Consequently, the entire latent stage accounts for 37–46% of single-rollout latency (~21–26% from the 24 Flow Head steps themselves, with answer decoding taking the remaining 54–63%), and parallel batching keeps K=4 / K=16 wall-clock latency to just 1.3–1.5× (6.5–14.1 s) and 1.8–2.5× (8.7–23.2 s) of K=1K=1 (7.6–11.4× even at K=100K=100).


Q4. Additional comparison with LEPO [1] under a common backbone and protocol

Under aligned recipes in the Table 2 setting, ATF consistently outperforms LEPO [1] in Pass@1 and small-KK coverage while using 2.6–4.4× fewer latent steps.

For LEPO [1], we use the official code under the matched Table 2 setup (LLaMA-3.1-8B, DART-MATH, shared 500-step SFT warmup and RL hyperparameters, T=0.7T=0.7, latent =280= 280 matching ATF's ≤280\le 280 cap, answer ≤1024\le 1024; 100 samples/question, Table 2 evaluation). Due to the limited rebuttal window (>1 week for full 8B RL), both methods are compared at 200 RL steps. Since LEPO's training became unstable after ~150 steps, we report both matched step 200 and its best checkpoint across the sweep (step 100) (with full Table 2 ATF for reference):

MethodMATH-500GSM8KOlympiadBench
Pass@1Pass@16Pass@100Pass@1Pass@16Pass@100Pass@1Pass@16Pass@100
LEPO [1] (RL step 200)18.438.952.061.580.485.83.113.424.2
LEPO [1] (RL step 100, best ckpt)34.172.487.279.496.498.59.136.456.2
ATF (RL step 200)39.574.787.282.696.598.911.637.252.5
  • Higher accuracy across stages: Even against LEPO's best checkpoint (step 100), matched-step ATF improves Pass@1 by +5.4 / +3.2 / +2.5 and beats LEPO at Pass@16 across all three benchmarks; continuing training to the full Table 2 model further raises Pass@1 to 42.2 / 84.6 / 12.7 (+8.1 / +5.2 / +3.6 over best LEPO).
  • Compact adaptive latent budget: LEPO executes all 280 soft-token full-backbone steps on every problem followed by long explicit CoT text (~442–610 tokens on MATH-500 / OlympiadBench, which accounts for its high large-KK Pass@100 despite trailing on Pass@1 and Pass@16), whereas ATF's CR-VAE + adaptive stopping averages only 63.9–106.7 latent steps (2.6–4.4× fewer).

Comparison with Parallel Latent Exploration [2]

Most points in Critical Weaknesses are addressed in Q1–Q4 above. To directly address the remaining question on parallel latent exploration methods [2], we evaluate additive Gaussian noise and MC-dropout (p=0.03p=0.03 and 0.200.20, following [2]'s released code) on our matched Qwen3-4B MSE baseline (K=16,T=0.1K=16, T=0.1):

MethodMATH-500OlympiadBench
Pass@1Pass@16Maj@16Pass@1Pass@16Maj@16
Deterministic MSE77.087.879.840.858.744.9
MSE + Gaussian Noise77.590.079.240.459.643.7
MSE + MC-Dropout (p=0.03)76.590.881.039.960.845.4
MSE + MC-Dropout (p=0.20)66.193.283.031.259.245.3
ATF (Flow Head)78.792.282.441.962.547.2

Compared with the test-time perturbations in [2], the added value of our autoregressive Flow Head is twofold:

  1. Mutually reinforcing per-path correctness and multi-sample exploration: Over deterministic MSE (77.0 / 40.8 Pass@1; 87.8 / 58.7 Pass@16), ATF simultaneously achieves the highest mean per-path correctness (Pass@1: 78.7 / 41.9), significantly raises Pass@16 (92.2 / 62.5), and improves Maj@16 (82.4 / 47.2). By contrast, [2]'s perturbations face an accuracy–exploration trade-off: Gaussian noise under-explores (Pass@16 90.0 / 59.6 vs. 92.2 / 62.5; Maj@16 79.2 / 43.7 vs. 82.4 / 47.2) and mild dropout (p=0.03p=0.03) also trails ATF (90.8 / 60.8), while strong MC-dropout (p=0.20p=0.20) raises MATH-500 coverage only at the cost of a 10.7–14.5 point drop in Pass@1 (and still trails ATF on OlympiadBench Pass@16: 59.2 vs. 62.5).
  2. Providing a learnable multimodal policy for RL: MC-dropout has no tractable likelihood, and Gaussian noise yields a fixed-variance, unimodal distribution around a single MSE prediction; ATF's Flow Head instead learns its distribution over next thoughts from data, which latent-trajectory RL directly optimizes (Q1).

We will incorporate all of the above ablations, Pass@KK / Maj@KK curves, latency breakdowns, and comparisons with [1, 2] into the revised manuscript.


References

  • [1] Zhou et al., LEPO: Latent Reasoning Policy Optimization for Large Language Models, Findings of ACL 2026.
  • [2] You et al., Parallel Test-Time Scaling for Latent Reasoning Models, ACL 2026.
AuthorsOpenReview ↗

Brief Follow-up on Our Response

Dear Reviewer s6NK,

Thank you again for your thoughtful feedback. We wanted to briefly follow up on our response and the additional experiments. Please let us know if any concerns remain or if there is anything we could clarify further. We would also welcome any updated thoughts on your overall assessment in light of these results.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.