acceptodds
Under review as a conference paper at ICLR 2027

Beyond a Single Thought: Learning Diverse Latent Reasoning Paths with SoftPaths

Abstract

Chain-of-Thought (CoT) reasoning improves the problem-solving capabilities of large language models by generating intermediate reasoning steps, but explicitly verbalizing these steps incurs substantial inference costs and often produces redundant content. Latent CoT alleviates this limitation by encoding intermediate reasoning in continuous representations. However, existing methods often rely on a single latent condition or generate multiple variants through simple resampling or perturbation, which may provide limited exploration of the solution space and fail to capture functionally distinct reasoning directions. We propose SoftPaths, a two-stage framework that learns multiple structured latent reasoning paths and adapts the generation policy to reason effectively under these paths. In the first stage, path-specific routers compose a shared set of projection experts to construct multiple input-dependent latent conditions. Answer-only supervision ensures that each path remains predictive of the final answer, while a contrastive diversity objective mitigates representation collapse across paths. In the second stage, the learned latent-path constructor is frozen, and the paths are activated in a round-robin manner to provide diverse latent conditions for GRPO-based policy optimization. A correctness-gated length reward prioritizes answer correctness while discouraging unnecessary verbalization among correct responses. At inference time, SoftPaths supports both efficient single-path reasoning and multi-path test-time scaling through answer aggregation. Experiments across multiple reasoning benchmarks demonstrate that SoftPaths consistently improves the balance between answer accuracy and efficiency over explicit and latent CoT methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
chain of cot, latent space, moe

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_7mxHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SoftPaths, a two-stage method that gives a decoder several different latent conditions for the same question. A frozen LLaMA-3.2-1B assistant produces hidden states, and four path-specific routers mix a shared pool of projection experts to turn them into four soft-prompt sequences for a LLaMA-3.1-8B base model. After a short rationale-supervised LoRA warm-up, the projector is trained with an answer-only loss plus a contrastive loss that pushes the paths apart. The projector is then frozen, and a new LoRA adapter is trained with GRPO. Paths are cycled across steps, and the reward only credits correct answers and penalizes correct responses that are longer than the group's average correct response. At test time the model either decodes from one path, or decodes once from each of several paths and takes a majority vote. On GSM8K, ASDiv-Aug, MATH, CommonsenseQA and StrategyQA, the paper reports the best average accuracy at M=1 and M=4 with lower visible-token usage than the matched CoT baselines. It also includes analyses of candidate scaling, path complementarity and ablations.

Critical strengths
  1. Good results at matched candidate counts. SoftPaths has the highest five-benchmark average at both budgets: 72.20 at M=1 (strongest external baseline: Self-SoftCoT, 70.79) and 76.51 at M=4 (strongest external baseline: Assist-CoT-SC, 72.69). The visible-token metric includes all four responses at M=4. The paper is also upfront that fewer visible tokens does not mean lower end-to-end latency.

  2. The analysis goes beyond a main table. The GSM8K ablations show drops when the contrastive loss, the MoE projector or the length reward is removed. Figure 5 breaks the voting gain down into mean-path accuracy, Oracle@4, mixed correctness and answer disagreement. The correctness-gated reward is a sensible choice, since a short but wrong answer gets nothing.

Critical weaknesses
  1. The comparisons do not isolate the contribution of latent paths beyond decoder RL. SoftPaths fine-tunes the base model (a warm-up LoRA, then GRPO with a correctness-gated length reward). The explicit CoT baselines are training-free, while SoftCoT trains its projector with the base decoder frozen. Self-SoftCoT is trained with RL, but only through the projector, and it is reported only at M=1. There, it beats SoftPaths on ASDiv-Aug (92.35 vs 89.88) and CSQA (75.18 vs 75.02), and its ASDiv-Aug score is above SoftPaths' own M=4 result (92.00). The w/o-GRPO row shows what GRPO adds to SoftPaths. It does not show what the latent paths add on top of GRPO. A no-latent model trained with the same warm-up, LoRA-GRPO and reward is the missing control, and a good part of the accuracy gain and the token reduction could come from the decoder training and the length reward alone. This matters because latent-path construction is a central contribution, yet the current controls do not separate its benefit from the combined decoder-training recipe. The tables provide no uncertainty estimates, and Appendix B.2 does not specify the number of decoding seeds or independent training runs. Small gains, such as the 0.23-point GSM8K improvement from GRPO at M=1 and the 0.68-point length-reward ablation, therefore cannot be confidently distinguished from variation.

  2. The claim of functionally diverse paths is not sufficiently isolated from sampling diversity. The contrastive loss encourages separation in representation space, but does not by itself establish different reasoning strategies or decoder-relevant information. Such separation could reflect path-specific signals with little effect on decoder behavior. Figure 1 is an illustration and Appendix C states that its case study is not causal evidence. Figure 5 shows disagreement and a voting gain, but lacks a within-model sampling control: the same metrics for four samples from one fixed path of the same checkpoint are needed to distinguish path-conditioned complementarity from stochastic decoding. SoftCoT++ already builds multiple latent thoughts with specialized initial tokens and contrastive learning. The advance over it therefore needs to be established through shared-expert path construction and latent-conditioned policy optimization, rather than representation diversity alone. The shared-linear-projector ablation does not distinguish the benefit of MoE composition from that of having multiple path-specific mappings. Independent projectors of comparable size, trained with the same losses, would provide a more informative control. This matters because functional path diversity is a central claim, and the current controls do not establish that distinct latent conditions help beyond sampling from one condition.

Questions for authors
  1. For the same trained SoftPaths checkpoint, how do four different paths compare with four samples from one fixed path, under matched decoding and voting? Please report accuracy and the Figure 5 metrics. Which path is used for M=1, and how does its GSM8K accuracy of 84.31% relate to Figure 5a's mean-path accuracy of 84.51%?
  2. How does a no-latent model perform with the same warm-up, LoRA-GRPO and correctness-gated length reward? If feasible, a four-candidate Self-SoftCoT result would also help the matched-budget comparison.
  3. What are the expert and router architectures? How does the shared-expert projector compare with independent path projectors of similar size trained with the same answer and diversity losses?
  4. How do you transition from eight latent vectors during projector training to one in Stage II and inference? What happens to the warm-up LoRA when the fresh GRPO adapter is introduced? Please also give the training data and splits, the rationale source, the reward and diversity coefficients, and how ties and invalid outputs are handled in voting.
  5. How many decoding seeds and independent training runs are behind the reported numbers? Can you provide uncertainty estimates for the main comparisons and key ablations, say which baseline numbers were reproduced and which were taken from prior work, and say whether code will be released?
3 comments

[1/3]

We thank the reviewer for the constructive feedback. We conducted additional controlled experiments and provide clarifications below.

Q1 / W2. Do multiple latent paths provide benefits beyond stochastic sampling?

(1)Controlled sampling comparison. We directly compare four distinct latent paths with four samples from a single fixed path, using the same SoftPaths checkpoint, prompts, decoding settings, and candidate budget. The single-path results are averaged over all four path choices.

Table R1. Multi-path versus single-path sampling (M=4).

DatasetSampling strategyMP (%) ↑Vote@4 (%) ↑Oracle@4 (%) ↑MC (%)AD (%)
GSM8KFour paths84.5190.6795.2226.0829.80
One path × 484.1389.2094.2024.1328.39
ASDiv-AugFour paths89.9192.0096.0513.3915.41
One path × 489.0490.7995.1112.4815.70

These controlled results demonstrate that learned latent paths provide additional predictive complementarity beyond repeated stochastic sampling from a single latent condition.

(2)Predictive complementarity. We further compare predictive complementarity and visible reasoning diversity with representative baselines.

Table R2. Predictive complementarity across methods (M=4).

DatasetMethodMP (%) ↑Vote@4 (%) ↑Oracle@4 (%) ↑MC (%)AD (%)All correct (%)All incorrect (%)
ASDiv-AugSoftPaths89.9192.0096.0513.3915.4182.663.95
SoftCoT-SC84.7188.5490.7516.2822.6474.479.25
SoftCoT++86.5888.6390.9410.1216.1880.839.06
Assist-CoT-SC87.9190.1792.8710.8915.4181.987.13
CommonsenseQASoftPaths75.7077.8186.9824.0027.1162.9813.02
SoftCoT-SC64.5068.4785.5044.5553.4840.9514.50
SoftCoT++66.0970.1186.3242.6750.1243.6513.68
Assist-CoT-SC66.5071.9186.5946.8554.4639.7413.41

MP: mean candidate accuracy; Vote@4: voting accuracy; Oracle@4: at least one correct candidate; MC: mixed correctness (1–3 correct candidates); AD: at least two distinct normalized answers (unparseable outputs form one category).

SoftPaths achieves the highest voting accuracy and correct-answer coverage on both benchmarks, supporting the effectiveness of latent-path complementarity across different reasoning tasks.

[2/3]

(3)Reasoning Diversity.

Table R3. Lexical and semantic diversity of visible reasoning (M=4).

DatasetMethodDistinct-2 ↑Length-matched D2 ↑Semantic distance ↑Reasoning tokens/candidate ↓
ASDiv-AugSoftPaths0.48680.56970.153960.3
SoftCoT-SC0.44580.48390.1075100.9
SoftCoT++0.48020.53810.108190.2
Assist-CoT-SC0.41450.43670.082498.7
CommonsenseQASoftPaths0.72190.75460.248046.9
SoftCoT-SC0.71610.74890.2461109.9
SoftCoT++0.66160.73490.2293111.8
Assist-CoT-SC0.65760.68420.1864164.8

Metrics exclude final answers and fixed templates. Distinct-2 measures unique-bigram proportion; length-matched D2 truncates each question's four rationales to the shortest common length; semantic distance averages pairwise MiniLM cosine distances; reasoning tokens/candidate measures average rationale length.

SoftPaths achieves the highest lexical and semantic diversity with shorter reasoning traces, providing behavioral evidence that latent-path conditioning induces more varied visible reasoning outputs.

Together, Tables R1–R3 support observable decoding diversity and predictive complementarity, rather than merely geometric separation of latent representations.

M=1 evaluation clarification (about 84.31% in Table 1 and 84.51% in Figure 5). Table 1 uses balanced random path assignment, yielding 84.31% (1,112/1,319). Figure 5(a) averages four paths evaluated separately on the full test set, yielding 84.51%. We will clarify both protocols in the revision.

Q2 / W1. Are the gains merely attributable to decoder-side GRPO?

No-latent GRPO baseline. To isolate the contribution of latent paths, we introduce LoRA-GRPO, a no-latent baseline trained using SFT followed by GRPO, with matched backbone, training budget, reward, and evaluation settings.

Table R4. Comparison with the no-latent GRPO baseline (ASDiv-Aug, M=4).

MethodVote@4 (%) ↑Oracle@4 (%) ↑Visible tokens/question ↓
SoftPaths92.0096.05333.2
LoRA-GRPO90.7595.18366.4

SoftPaths outperforms matched decoder-only GRPO, providing evidence that learned latent paths contribute beyond the decoder-training recipe.

Self-SoftCoT comparison. We acknowledge that Self-SoftCoT performs better on some individual M=1 benchmarks. However, SoftPaths achieves higher five-benchmark average M=1 accuracy (72.20% vs. 70.79%). At M=4, SoftPaths reaches 76.51%, compared with 72.69% for Assist-CoT-SC. We will clarify the candidate-budget differences.

Q3 / W2. Does shared-expert routing outperform independent projectors?

SoftPaths uses four shared 2048→4096 linear experts and path-specific routing (2048→16), with Gumbel-Softmax during training. We compare it with four independent linear projectors under matched supervision and training budgets.

Table R5. Shared-expert routing versus independent projectors (GSM8K, Stage I, M=4).

MethodParameters ↓Vote@4 (%) ↑
SoftPaths33,587,20089.76 ± 0.35
Independent projectors33,554,43287.96 ± 0.42

With nearly identical parameter counts, shared-expert routing achieves higher voting accuracy and correct-answer coverage, supporting the benefit of SoftPaths.

[3/3]

Q4. How are the two stages connected, and what are the implementation details?

Latent-slot transition. Our implementation follows the prefix-length adjustment used in the official SoftCoT/SoftCoT++ code, which uses 32 soft-token positions during training and 4 during evaluation. Importantly, SoftPaths generates only one latent vector per path, which is repeated across eight positions in Stage I and inserted into a single position in Stage II. Thus, no independent latent vectors are discarded. We evaluate the same Stage-I SoftPaths-SFT checkpoint before GRPO, with the warm-up LoRA removed, using identical decoding settings.

Table R6. Effect of the 8-to-1 latent-slot transition.

Dataset8→1 Accuracy (%)8→8 Accuracy (%)
ASDiv-Aug87.3887.48
CommonsenseQA75.1075.10

The nearly identical accuracy indicates that reducing the repeated latent prefix to one position does not materially affect answer-level performance in the evaluated settings.

About warm-up LoRA lifecycle. The warm-up Stage-I LoRA is trained with supervised reference responses and then frozen to provide a stable decoder for projector training. Before Stage II, it is removed, and a fresh LoRA is initialized for GRPO, avoiding the formatting bias learned during supervised warm-up.

Data and supervision. Both stages use the same task-specific training partitions.

Table R7. Training and evaluation partitions.

DatasetStage I / Stage II trainingEvaluation
GSM8KTrain (7,473)Test (1,319)
ASDiv-AugTrain (4,183)Test (1,038)
MATHTrain (7,348)MATH500 (500)
CommonsenseQATrain (9,741)Labeled Validation (1,221)
StrategyQAFirst 80% (1,832)Remaining 20% (458)

Training details. Both stages use the training partitions above, with no exact normalized question overlap with evaluation sets. Stage-I contrastive weight/temperature/dropout: 0.1/0.05/0.1. Stage-II GRPO uses G=8, a formatting coefficient of 0.1, and a correctness-gated length penalty of 0.05.

We will provide the complete implementation details in the revision.

Q5 / W1. Are the improvements reliable and reproducible?

Length-reward ablation. We reevaluate the GSM8K length-reward ablation using fixed checkpoints and three decoding seeds.

Table R8. Three-seed GSM8K length-reward ablation (M=4).

Decoding seedFull SoftPaths Vote@4 (%) ↑No length reward Vote@4 (%) ↑Gain (pp)Full tokens/question ↓No length reward tokens/question ↓
Mean ± SD90.37 ± 0.7089.11 ± 0.94+1.26 ± 0.39782.07 ± 11.93815.23 ± 15.10

The correctness-gated length reward improves voting accuracy and reduces visible tokens across all three decoding seeds, supporting its effectiveness beyond a single evaluation run.

We additionally report three-seed results on ASDiv-Aug and CommonsenseQA.

Table R9. Multi-seed accuracy and visible-token efficiency (M=4, mean ± sample SD).

DatasetMethodVote@4 (%) ↑Visible tokens/question ↓
ASDiv-AugSoftPaths92.00 ± 0.48323.72 ± 1.99
SoftCoT-SC89.76 ± 1.07473.45 ± 8.69
LoRA-GRPO90.75 ± 0.24366.57 ± 1.66
CommonsenseQASoftPaths77.81 ± 0.64249.82 ± 1.11
SoftCoT-SC66.53 ± 0.70675.05 ± 10.84
LoRA-GRPO76.74 ± 0.36277.32 ± 1.01

SoftPaths achieves the highest mean voting accuracy and lowest visible-token consumption on both benchmarks, providing additional evidence of its advantages over repeated sampling and no-latent decoder optimization.

Reproducibility. Coconut and Self-SoftCoT results are taken from their original papers; other external baselines are locally reproduced using official implementations. We will clarify baseline sources and release our code, configurations, reward functions, and voting implementation upon acceptance.

We appreciate the reviewer's suggestions and will incorporate these additional results and clarifications into the revised manuscript.

Reviewer_Zy74OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SoftPaths, a two-stage framework for learning multiple latent reasoning conditions for large language models. The motivation is that conventional chain-of-thought (CoT) methods incur substantial inference costs by generating intermediate reasoning as visible tokens, while existing latent reasoning approaches may rely on a single latent condition or produce variants that do not necessarily induce functionally distinct reasoning behavior. SoftPaths addresses this problem by learning multiple input-dependent latent paths through a shared pool of projection experts and path-specific routers. In Stage I, the latent-path constructor is trained using answer-only supervision and a contrastive diversity objective intended to preserve answer-relevant information while discouraging representational collapse. In Stage II, the latent-path constructor is frozen, and a LoRA adapter on the base language model is optimized using Group Relative Policy Optimization (GRPO). The paths are activated in a round-robin manner during training, and a correctness-gated length reward encourages concise responses without rewarding incorrect answers for being short. At inference time, the method supports single-path generation and multi-path generation with majority-vote aggregation. The authors evaluate SoftPaths on GSM8K, ASDiv-Aug, MATH, CommonsenseQA, and StrategyQA. They report an average accuracy of 72.20% with a single candidate and 76.51% with four candidates. The corresponding SoftPaths-SFT results are 70.81% and 75.09%, respectively. The paper also reports lower visible-token usage than matched explicit CoT baselines and includes ablations of the MoE projector, path-diversity objective, and reward components. The paper's central contribution is a unified framework for learning multiple latent reasoning conditions and adapting a shared generation policy to use them. The results are promising, although the strength of the efficiency claims and the extent to which the learned paths represent genuinely distinct reasoning strategies require further examination.

Critical strengths

Strength 1: A coherent framework for learning multiple latent conditions The paper combines shared projection experts, path-specific routers, answer-only supervision, and contrastive diversity regularization to construct multiple latent conditions for the same input. This is a well-motivated approach to the limitation of latent reasoning methods that rely on a single condition or unstructured resampling. The separation between learning latent paths and subsequently optimizing the generation policy is also conceptually useful. It allows the authors to investigate the contribution of the learned paths independently of the additional GRPO stage.

Strength 2: Promising accuracy and visible-token results, supported by ablations SoftPaths reports competitive or improved average accuracy across five mathematical and commonsense reasoning benchmarks. With four candidates, it achieves an average accuracy of 76.51%, compared with 75.09% for SoftPaths-SFT and 72.69% for Assist-CoT-SC. The single-candidate results also suggest that the method's benefits are not solely attributable to majority voting. The ablations provide additional evidence that the proposed components contribute to performance. In particular, removing the contrastive path-diversity objective or replacing the MoE projector with a shared linear projector reduces the reported four-candidate GSM8K accuracy. The reward ablations also illustrate a trade-off between correctness and visible-token usage.

Together, these results make the approach promising and provide a reasonable basis for further investigation.

Critical weaknesses

Weakness 1: The efficiency claims are not yet supported by end-to-end cost measurements The paper emphasizes improved inference efficiency, but its main efficiency metric is the number of visible tokens generated. This is a useful metric for measuring the reduction in explicit reasoning text, but it does not capture the full computational cost of the proposed method. SoftPaths uses a frozen assistant model to extract representations, a learned projector to construct latent conditions, and a base model to generate the response. The multi-path setting also requires separate decoding passes for each selected path. Consequently, fewer visible tokens do not necessarily translate into lower wall-clock latency, higher throughput, or lower total computational cost. The authors acknowledge this distinction, which is appropriate, but it also limits the strength of the paper's practical efficiency conclusions. A more complete evaluation should report end-to-end latency, throughput, peak memory, and total inference computation, including assistant-model and projector overhead. The comparison should use consistent hardware, batch sizes, decoding settings, and candidate budgets. This is a consequential limitation because efficiency is one of the paper's central motivations rather than a secondary observation.

Weakness 2: The interpretation of latent-path diversity is not sufficiently established The paper uses path-specific routers and contrastive regularization to encourage distinct latent representations. It further reports complementary errors and improvements from majority voting. These findings support the claim that the paths can provide different predictive conditions. However, representation-level separation does not necessarily imply functionally distinct reasoning strategies. Different latent conditions may produce different answers or errors without encoding qualitatively different intermediate computations. The current experiments do not conclusively distinguish these possibilities. The complementarity analysis is useful, but it primarily demonstrates differences in predictive behavior. Stronger evidence could include systematic comparisons of path-conditioned outputs, controlled tests of whether individual paths specialize in different problem types or solution strategies, and measurements of path diversity beyond representation separation and answer disagreement.

This distinction matters because the paper's central conceptual claim is not merely that multiple latent conditions improve accuracy, but that they provide diverse latent reasoning paths.

Questions for authors

Question 1: What is the actual end-to-end inference cost of SoftPaths? Can the authors report wall-clock latency, throughput, peak memory, and total inference computation for SoftPaths and the strongest relevant baselines under matched hardware and decoding conditions? Please include assistant-model execution and latent-path construction costs. For multi-path inference, can the authors quantify the additional computational cost of using four latent paths relative to generating four candidates from a shared condition? This question materially affects the assessment because the paper's efficiency conclusions currently rely primarily on visible-token counts.

Question 2: What evidence demonstrates that the learned paths are functionally distinct reasoning strategies? How do the authors distinguish genuine reasoning-strategy diversity from representation-level diversity or complementary prediction errors? Can they provide controlled analyses showing that different paths consistently exhibit different solution strategies, specialize in different problem types, or contribute distinct information beyond stochastic variation? This question concerns the interpretation of the central contribution. The current results show useful predictive complementarity, but the stronger claim about distinct reasoning paths requires additional support.

Question 3: How robust are the reported improvements across training runs? Were the proposed method and the most relevant trained baselines evaluated across multiple independent training seeds? If so, can the authors report means and standard deviations or confidence intervals for the main accuracy and visible-token results?

The reported improvements are promising, but uncertainty estimates would help determine whether the observed differences are robust, especially for benchmarks where the absolute performance gaps are relatively small.

3 comments

[1/3]

We thank the reviewer for the constructive feedback and for recognizing the contributions of SoftPaths. We conducted additional experiments to address the concerns regarding end-to-end inference efficiency, path diversity, and statistical reliability.

W1 / Q1. What is the end-to-end inference cost of SoftPaths?

(1) End-to-end inference profiling.

We additionally report end-to-end inference latency, peak GPU memory, and dense TFLOPs on an NVIDIA RTX A6000 GPU (BF16, M=4).

Table R1. End-to-end inference efficiency (M=4).

DatasetMethodVote@4 (%) ↑Visible tokens/query ↓Latency (s/query) ↓Peak memory (GB) ↓Dense TFLOPs/query ↓
ASDiv-AugAssist-CoT-SC90.17588.46.14519.26526.387
SoftCoT-SC88.54477.37.01519.27122.355
SoftCoT++88.63458.35.29519.27822.542
SoftPaths92.00333.25.35319.45014.658
CommonsenseQAAssist-CoT-SC71.91813.37.66519.27430.487
SoftCoT-SC68.47677.39.83319.27732.147
SoftCoT++70.11534.06.12619.27625.903
SoftPaths77.81248.45.26119.43613.402

Under this one-question-at-a-time setup, the serial request rate is the reciprocal of mean latency.

SoftPaths achieves the highest accuracy and lowest dense computational cost, with comparable or lower latency than SoftCoT++, even after accounting for the assistant and projector overhead.

(2) Four latent paths versus repeated sampling from one condition.

To isolate the overhead of multi-path conditioning, we compare four distinct paths against four samples from one fixed path using the same SoftPaths checkpoint and candidate budget in a separate timing run.

Table R2. Latency comparison between multi-path and single-path sampling (M=4).

DatasetFour paths (s/query)One fixed path × 4 (s/query)Difference (s/query)
ASDiv-Aug5.3915.133+0.258
CommonsenseQA7.0206.953+0.067

Difference is four-path latency minus fixed-path latency, calculated from the displayed values.

Four-path conditioning incurs only a modest additional latency of 0.258 and 0.067 seconds per question on ASDiv-Aug and CommonsenseQA, respectively, demonstrating that multi-path inference introduces limited overhead at the same candidate budget.

(1) Predictive complementarity under controlled sampling.

We compare four distinct paths against four samples from a single fixed path using the same checkpoint, prompts, decoding settings, and candidate budget. Single-path results are averaged across all four path choices.

Table R1. Four learned paths versus four samples from one path (M=4).

Sampling strategyTaskMPVote@4Oracle@4MCAD
Four pathsGSM8K84.5190.6795.2226.0829.80
One path × 4GSM8K84.1389.2094.2024.1328.39
Four pathsASDiv-Aug89.9192.0096.0513.3915.41
One path × 4ASDiv-Aug89.0490.7995.1112.4815.70

MP is mean candidate accuracy; Vote@4 is majority-vote accuracy; Oracle@4 is the proportion of questions with at least one correct candidate; MC denotes mixed correctness (1–3 correct candidates); and AD measures disagreement among normalized answers, with unparseable outputs grouped into one category.

These results demonstrate that learned latent paths provide predictive complementarity beyond repeated stochastic sampling from a single condition.

[2/3]

(2) Visible reasoning diversity.

We further evaluate lexical and semantic diversity across four generated reasoning traces on ASDiv-Aug and CommonsenseQA.

Table R2. Lexical and semantic diversity of visible reasoning (M=4).

DatasetMethod (M=4)Distinct-2 ↑Length-matched D2 ↑Semantic distance ↑Reasoning tokens/candidate ↓
ASDiv-AugSoftPaths0.48680.56970.153960.3
ASDiv-AugSoftCoT-SC0.44580.48390.1075100.9
ASDiv-AugSoftCoT++0.48020.53810.108190.2
ASDiv-AugAssist-CoT-SC0.41450.43670.082498.7
CommonsenseQASoftPaths0.72190.75460.248046.9
CommonsenseQASoftCoT-SC0.71610.74890.2461109.9
CommonsenseQASoftCoT++0.66160.73490.2293111.8
CommonsenseQAAssist-CoT-SC0.65760.68420.1864164.8

Distinct-2 measures the proportion of distinct adjacent token pairs. For length-matched D2, we truncate the four traces for each question and method to the length of the shortest trace in that set. Semantic distance is the average cosine distance across the six pairs of MiniLM embeddings.

SoftPaths achieves higher lexical and semantic diversity with shorter reasoning traces, providing behavioral evidence beyond latent representation separation.

Together, these results support observable reasoning diversity and predictive complementarity. We will clarify these findings in the revision.

W2 / Q2. Do the learned paths produce diverse and complementary reasoning outputs?

We agree that distinct latent embeddings do not necessarily imply different reasoning strategies. We therefore investigate predictive complementarity and visible reasoning diversity through controlled experiments.

(1) Predictive complementarity under controlled sampling.

We compare four different latent paths against four samples from a single fixed path using the same checkpoint, prompts, decoding settings, and voting rules. Single-path results are averaged across all four path choices.

Table R3. Multi-path versus single-path sampling (M=4).*

DatasetSampling strategyMP (%) ↑Vote@4 (%) ↑Oracle@4 (%) ↑
GSM8KFour paths84.5190.6795.22
GSM8KOne path × 484.1389.2094.20
ASDiv-AugFour paths89.9192.0096.05
ASDiv-AugOne path × 489.0490.7995.11

MP is mean candidate accuracy. Vote@4 is majority-vote accuracy. Oracle@4 is the percentage of questions for which at least one of the four candidates is correct.

Four-path sampling improves both voting accuracy and correct-answer coverage, demonstrating predictive complementarity beyond repeated stochastic sampling from a single condition.

[3/3]

(2) Direct measurement of visible reasoning diversity.

We further evaluate lexical and semantic diversity across four generated reasoning traces on ASDiv-Aug and CommonsenseQA.

Table R4. Lexical and semantic diversity of visible reasoning (M=4).

DatasetMethodDistinct-2 ↑Length-matched D2 ↑Semantic distance ↑Reasoning tokens/candidate ↓
ASDiv-AugSoftPaths0.48680.56970.153960.3
SoftCoT-SC0.44580.48390.1075100.9
SoftCoT++0.48020.53810.108190.2
Assist-CoT-SC0.41450.43670.082498.7
CommonsenseQASoftPaths0.72190.75460.248046.9
SoftCoT-SC0.71610.74890.2461109.9
SoftCoT++0.66160.73490.2293111.8
Assist-CoT-SC0.65760.68420.1864164.8

Distinct-2 measures the proportion of unique bigrams. For length-matched D2, we truncate the four traces for each question and method to the length of the shortest trace in that set. Semantic distance is the mean cosine distance across the six pairs of MiniLM embeddings.

SoftPaths achieves higher lexical and semantic diversity with shorter reasoning traces, providing behavioral evidence of diverse path-conditioned reasoning outputs.

Together, these results support observable reasoning diversity and predictive complementarity beyond representation-level separation.

Q3. How robust are the reported improvements across different seeds?

We evaluate SoftPaths and representative baselines using three decoding seeds with fixed checkpoints on ASDiv-Aug and CommonsenseQA, reporting mean ± sample standard deviation. We additionally include LoRA-GRPO, a no-latent baseline trained with SFT followed by GRPO, to distinguish latent-path contributions from decoder-side optimization.

Table R5. Multi-seed accuracy and visible-token efficiency (M=4, mean ± sample SD).

DatasetMethodVote@4 (%)Visible tokens/question
ASDiv-AugSoftPaths92.00 ± 0.48323.72 ± 1.99
ASDiv-AugSoftCoT-SC89.76 ± 1.07473.45 ± 8.69
ASDiv-AugLoRA-GRPO90.75 ± 0.24366.57 ± 1.66
CommonsenseQASoftPaths77.81 ± 0.64249.82 ± 1.11
CommonsenseQASoftCoT-SC66.53 ± 0.70675.05 ± 10.84
CommonsenseQALoRA-GRPO76.74 ± 0.36277.32 ± 1.01

For each method, we evaluate a fixed checkpoint with three decoding seeds and report the mean and sample standard deviation. The visible-token count includes all four responses per question.

SoftPaths consistently achieves higher mean voting accuracy and lower visible-token usage than the evaluated baselines, supporting its robustness to decoding randomness.

We appreciate the reviewer's suggestions, which have helped us strengthen the efficiency evaluation, and provide more direct evidence of path-conditioned behavior.

Reviewer_DmQmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for generating diverse latent reasoning paths by learning a set of shared experts and routers that map the input representation to different paths. A frozen assistant model extracts features from the input, which are then used by the experts to transform into multiple different base-model-compatible embeddings. In the first stage, only the projectors are trained to learn to map the input into a diverse set of representations, followed by a second phase of GRPO to fine-tune the base model. At inference, the model can use one path or aggregate answers from several paths through voting. Experiments on multiple reasoning benchmarks show strong improvement

Critical strengths
  1. The method is intuitive and easy to understand.

  2. Strong improvement over multiple baselines shows the efficacy of the proposed method.

  3. Relevant ablations are provided to show how different component of training interacts.

Critical weaknesses

I have the following concerns:

  1. In the first stage, the projectors are trained only with answer-only supervision and diversity loss; a different embedding does not necessarily mean a different reasoning path. Can the authors provide more evidence for this?

  2. In the second stage, the projectors are kept frozen while the base model is fine-tuned with GRPO. This might prevent the latent path from adapting to the changes in the decoder. An ablation where both are optimized together will be useful.

Questions for authors
  1. What's the benefit of using a different assistant model than the base one? Can we use the base model to extract the embedding from the input as well?
3 comments

[1/2]

We thank the reviewer for the constructive feedback and for recognizing the intuitive design, empirical improvements, and component ablations. We conducted additional experiments to address the concerns regarding path diversity, projector adaptation, and assistant-model selection.

W1. Do the embeddings learned in Stage I lead to different reasoning behavior?

Changing the latent path affects both the shared decoder's predictions and the reasoning text it generates for the same question. We evaluate the paths learned with answer-only supervision and a contrastive loss in Stage I. The projector that constructs these paths stays frozen during GRPO (Section 3.4).

(1) Predictive complementarity under controlled sampling.

We compare four distinct paths against four samples from a single fixed path using the same checkpoint, prompts, decoding settings, and candidate budget. Single-path results are averaged across all four path choices.

Table R1. Four learned paths versus four samples from one path (M=4).

Sampling strategyTaskMPVote@4Oracle@4MCAD
Four pathsGSM8K84.5190.6795.2226.0829.80
One path × 4GSM8K84.1389.2094.2024.1328.39
Four pathsASDiv-Aug89.9192.0096.0513.3915.41
One path × 4ASDiv-Aug89.0490.7995.1112.4815.70

MP is mean candidate accuracy; Vote@4 is majority-vote accuracy; Oracle@4 is the proportion of questions with at least one correct candidate; MC denotes mixed correctness (1–3 correct candidates); and AD measures disagreement among normalized answers, with unparseable outputs grouped into one category.

These results demonstrate that learned latent paths provide predictive complementarity beyond repeated stochastic sampling from a single condition.

(2) Visible reasoning diversity.

We further evaluate lexical and semantic diversity across four generated reasoning traces on ASDiv-Aug and CommonsenseQA.

Table R2. Lexical and semantic diversity of visible reasoning (M=4).

DatasetMethod (M=4)Distinct-2 ↑Length-matched D2 ↑Semantic distance ↑Reasoning tokens/candidate ↓
ASDiv-AugSoftPaths0.48680.56970.153960.3
ASDiv-AugSoftCoT-SC0.44580.48390.1075100.9
ASDiv-AugSoftCoT++0.48020.53810.108190.2
ASDiv-AugAssist-CoT-SC0.41450.43670.082498.7
CommonsenseQASoftPaths0.72190.75460.248046.9
CommonsenseQASoftCoT-SC0.71610.74890.2461109.9
CommonsenseQASoftCoT++0.66160.73490.2293111.8
CommonsenseQAAssist-CoT-SC0.65760.68420.1864164.8

Distinct-2 measures the proportion of distinct adjacent token pairs. For length-matched D2, we truncate the four traces for each question and method to the length of the shortest trace in that set. Semantic distance is the average cosine distance across the six pairs of MiniLM embeddings.

SoftPaths achieves higher lexical and semantic diversity with shorter reasoning traces, providing behavioral evidence beyond latent representation separation.

Together, these results support observable reasoning diversity and predictive complementarity. We will clarify these findings in the revision.

[2/2]

W2. Does freezing the projector prevent adaptation to the decoder during Stage II?

Although the projector remains frozen, the decoder adapts to different latent paths through GRPO-trained LoRA updates. We further compare our default setting with jointly optimizing the projector and base LoRA, starting from the same Stage-I projector.

Table R3. Frozen versus jointly optimized projector (GSM8K).

Stage-II trainable componentsM=1 accuracy (%)Vote@4 (%)Oracle@4 (%)
Projector + base LoRA85.1489.4695.07
Base LoRA only (frozen projector)84.3190.6795.22

While joint optimization slightly improves single-path accuracy, freezing the projector yields higher voting accuracy and correct-answer coverage, supporting its suitability for multi-path inference.

Freezing the projector also reduces the number of trainable parameters and preserves the learned latent conditions during decoder optimization. We will include this ablation.

Q1. Is a separate assistant model necessary, or can the base model extract the features?

A separate assistant is not strictly necessary. We evaluate a shared-base variant that uses LLaMA-3.1-8B-Instruct for both feature extraction and decoding.

Table R4. Separate assistant versus shared-base feature extraction (GSM8K, Stage I, M=4).

Feature extractorFeature dimensionparametersVote@4 (%)Oracle@4 (%)
Separate 1B assistant2,04833,587,20089.7695.00
Shared 8B base4,09667,174,40090.0795.00

Both configurations achieve comparable accuracy, while the separate assistant reduces projector parameters by approximately 50%.

We adopt the smaller assistant to decouple feature extraction from decoder optimization and reduce projector size, while the shared-base alternative remains feasible. We will clarify this design choice in the revision.

We appreciate the reviewer's suggestions, which have helped strengthen our empirical analysis and clarify the architectural choices.

Thank you for providing additional results; I will keep my initial positive assessment and review score.

Reviewer_RWDTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SoftPaths, a two-stage framework for learning diverse latent reasoning paths and improving their utilization through reinforcement learning. In Stage I, a mixture-of-experts (MoE) projector with shared projection experts and path-specific routers constructs multiple latent conditions from a frozen assistant model. Answer-only supervision encourages each path to retain task-relevant information, while a contrastive diversity objective discourages representation collapse. In Stage II, the learned projector is frozen, and a shared LoRA policy is optimized using GRPO, with latent paths activated in a round-robin manner. The authors further introduce a correctness-gated length reward to encourage concise responses without rewarding incorrect answers. At inference time, the framework supports both single-path generation and multi-path answer aggregation.

Experiments are conducted on five reasoning benchmarks, including GSM8K, ASDiv-Aug, MATH, CommonsenseQA, and StrategyQA, using LLaMA-3.1-8B-Instruct as the base model. SoftPaths achieves average accuracies of 72.20% and 76.51% under single-response and four-candidate settings, respectively, outperforming the reported baselines in the corresponding settings. Additional experiments investigate path complementarity, visible-token efficiency, and the contributions of the MoE projector, contrastive diversity loss, and reward components.

Critical strengths
  1. An interesting approach to latent reasoning diversity. The paper addresses an important limitation of existing latent CoT methods, which often rely on a single latent condition or generate multiple candidates through stochastic resampling. Learning multiple input-dependent latent conditions through path-specific routing over shared experts is a reasonable and potentially useful direction.

  2. Promising empirical results with a relatively comprehensive evaluation. SoftPaths achieves competitive results across five reasoning benchmarks under both single-path and multi-path inference. The ablation studies also provide initial evidence for the contributions of the MoE projector, contrastive diversity regularization, and correctness-gated length reward. In particular, the improvements under the four-candidate setting suggest that learned latent conditions may provide useful alternatives to repeated sampling from a shared condition.

Critical weaknesses
  1. The claimed functional diversity of latent reasoning paths is not sufficiently established. The proposed contrastive objective explicitly separates latent representations in embedding space, but representational separation does not necessarily imply functionally distinct reasoning processes. All paths are optimized using the same answer-only supervision, while the diversity objective operates on flattened latent representations with dropout-based positive pairs. Consequently, the learned paths could differ geometrically without capturing meaningfully different reasoning strategies. Although Figure 5 provides some evidence of complementary predictions, this analysis is limited to GSM8K and does not sufficiently establish functional diversity across different reasoning tasks. More direct path-wise behavioral analyses are needed to support the paper's central claim.

  2. There are unexplained inconsistencies between the path-learning and downstream optimization stages. Appendix B.1 states that eight continuous thought vectors are used per path during projector training, whereas only one vector is used during GRPO and inference. In addition, Stage I first trains a temporary LoRA adapter on full responses and then optimizes the projector with this adapter frozen, while Stage II discards it and initializes a fresh LoRA adapter. These changes introduce potential distribution shifts in both the latent representation and the decoder used to exploit it. The paper does not sufficiently explain why the learned paths remain effective under these changed settings, nor does it provide controlled ablations to establish that these design choices do not undermine the learned path quality.

Questions for authors
  1. Functional path diversity: How do the authors distinguish genuinely different reasoning functions from merely different latent representations? Since the diversity objective operates directly on flattened latent vectors, can the authors provide path-wise behavioral analyses or complementary prediction statistics across benchmarks beyond GSM8K? It would also be helpful to clarify the definitions of mixed correctness and answer disagreement in Figure 5.

  2. Latent sequence length: Why are eight continuous thought vectors used during projector training, while only one is used during GRPO and inference? How are the learned representations expected to remain consistent under this change? Can the authors provide an ablation comparing consistent latent sequence lengths across training and inference?

  3. Single-path inference: Section 3.5 states that one latent path is selected for single-path inference, but the path-selection strategy is not specified. Are the M=1 results obtained using a fixed path, a randomly selected path, or an average across all paths? Additionally, Table 1 reports 84.31% for SoftPaths on GSM8K, whereas Figure 5 reports 84.51% for mean-path accuracy. Could the authors clarify this difference?

  4. Training data and statistical reliability: What are the exact datasets, sample sizes, and data splits used for Stage-I projector training and Stage-II GRPO? How is potential overlap with the evaluation benchmarks handled? Furthermore, Appendix B.2 mentions using fixed checkpoints across decoding seeds. Were multiple independent training runs conducted, or are the reported results based on a single training run?

  5. Inference efficiency: The efficiency analysis primarily measures visible-token counts. However, SoftPaths introduces an additional assistant-model forward pass and projector computation, while multi-path inference requires multiple base-model decoding passes. Can the authors provide end-to-end inference latency, computational cost, or FLOPs comparisons against the corresponding CoT and self-consistency baselines to better establish the practical efficiency advantage?

5 comments

[1/5]

We thank the reviewer for the constructive feedback. To address the concern that representational diversity does not necessarily imply functional diversity, we conducted additional controlled experiments examining predictive complementarity, path-conditioned decoding behavior, and reasoning-text diversity.

Q1 / W1. Do learned latent paths induce meaningful decoding diversity beyond representational separation?

We agree that geometric separation alone does not establish functional diversity. We therefore examine predictive complementarity, path-conditioned decoding behavior, and visible reasoning diversity.

(1) Predictive complementarity under controlled sampling.

We compare generating one response from each of four latent paths against generating four responses from a single fixed path. Both settings use the same trained checkpoint, prompts, decoding settings, voting rule, and candidate budget. For the single-path control, we evaluate all four fixed paths separately and average their results.

Table R1. Multi-path versus single-path sampling (M=4).

TaskSampling strategyMPVote@4Oracle@4MCAD
GSM8KFour paths84.5190.6795.2226.0829.80
GSM8KOne path × 484.1389.2094.2024.1328.39
ASDiv-AugFour paths89.9192.0096.0513.3915.41
ASDiv-AugOne path × 489.0490.7995.1112.4815.70

All values are percentages. MP is mean candidate accuracy; Vote@4 is majority-vote accuracy; Oracle@4 is the proportion of questions with at least one correct candidate; MC denotes mixed correctness (1–3 correct candidates); and AD measures disagreement among normalized answers, with unparseable outputs grouped into one category.

These results demonstrate that learned latent paths provide additional predictive complementarity beyond repeated stochastic sampling from a single latent condition.

(2) Extended predictive complementarity across reasoning tasks.

We extend the analysis in Figure 5 to ASDiv-Aug and CommonsenseQA, comparing SoftPaths against representative baselines under the same four-candidate budget.

DatasetMethod (M=4)MP (%) ↑Vote@4 (%) ↑Oracle@4 (%) ↑MC (%)AD (%)All correct (%)All incorrect (%)
ASDiv-AugSoftPaths89.9192.0096.0513.3915.4182.663.95
SoftCoT-SC84.7188.5490.7516.2822.6474.479.25
SoftCoT++86.5888.6390.9410.1216.1880.839.06
Assist-CoT-SC87.9190.1792.8710.8915.4181.987.13
CommonsenseQASoftPaths75.7077.8186.9824.0027.1162.9813.02
SoftCoT-SC64.5068.4785.5044.5553.4840.9514.50
SoftCoT++66.0970.1186.3242.6750.1243.6513.68
Assist-CoT-SC66.5071.9186.5946.8554.4639.7413.41

All correct and all incorrect denote the proportions of questions for which all four candidates are correct or incorrect, respectively. By definition, Oracle@4 = 100% − All incorrect, and MC = Oracle@4 − All correct.

SoftPaths achieves the highest voting accuracy and correct-answer coverage across both benchmarks, supporting the effectiveness of learned latent-path complementarity across different reasoning tasks.

[2/5]

(3) Direct measurement of visible reasoning diversity.

To assess whether different latent paths produce distinct visible reasoning traces, we measure lexical and semantic diversity on ASDiv-Aug and CommonsenseQA, excluding final answers and fixed templates.

DatasetMethod (M=4)Distinct-2 ↑Length-matched D2 ↑Semantic distance ↑Reasoning tokens/candidate ↓
ASDiv-AugSoftPaths0.48680.56970.153960.3
SoftCoT-SC0.44580.48390.1075100.9
SoftCoT++0.48020.53810.108190.2
Assist-CoT-SC0.41450.43670.082498.7
CommonsenseQASoftPaths0.72190.75460.248046.9
SoftCoT-SC0.71610.74890.2461109.9
SoftCoT++0.66160.73490.2293111.8
Assist-CoT-SC0.65760.68420.1864164.8

All metrics exclude final answers and fixed templates. Distinct-2 measures unique-bigram proportion; length-matched D2 truncates the four rationales to their shortest common length; semantic distance averages six pairwise MiniLM cosine distances; reasoning tokens/candidate is the average visible rationale length.

SoftPaths consistently achieves higher lexical and semantic diversity with shorter reasoning traces, providing behavioral evidence that learned latent paths induce distinct visible reasoning outputs.

These results show that SoftPaths generates lexically and semantically more varied visible reasoning traces, even after controlling for response length. This provides behavioral evidence beyond the geometric separation enforced by the contrastive objective.

Together, these results support both predictive complementarity and observable reasoning diversity beyond representational separation alone. We will clarify the corresponding claims and analyses in the revised manuscript.

[3/5]

Q2 / W2. Why does Stage I use eight soft-token positions while Stage II uses one? How is the transition between the two LoRA adapters justified?

Thank you for raising these concerns. We clarify the two transitions separately and provide controlled evaluations of their effects.

(1) Why eight positions during training but only one during inference?

Our implementation follows the training/inference prefix-length adjustment used in the official SoftCoT/SoftCoT++ implementation, which specifies num_thought_tokens=32 during training and 4 during evaluation.

Importantly, SoftPaths does not construct eight independent latent thoughts. The frozen assistant produces one contextual feature, and each path-specific projector maps it to a single latent vector. During Stage I, this same vector is repeated across eight input positions of the backbone to facilitate supervised projector optimization. During Stage II and inference, the vector occupies only one position.

We evaluate the same Softpath-SFT before GRPO, comparing one-slot and eight-slot inference under the same decoding settings.

Table R4. Effect of reducing eight latent positions to one.

Dataset8→1 Accuracy (%) ↑8→8 Accuracy (%) ↑
ASDiv-Aug87.3887.48
CommonsenseQA75.1075.10

The nearly identical accuracy confirms that reducing the repeated latent prefix to one position preserves answer-level performance in the evaluated settings.

(2) Why discard the Stage-I LoRA and initialize a fresh adapter for GRPO?

The two adapters serve different optimization objectives.

During Stage I, a warm-up LoRA is first trained using supervised reference responses and then frozen to provide a stable decoder for projector learning. Before Stage II, it is removed, and a fresh LoRA is initialized for GRPO, avoiding inherited warm-up formatting preferences during policy optimization.

We evaluate the same trained projector with and without the warm-up LoRA before GRPO.

Table R5. Effect of retaining the warm-up LoRA (StrategyQA, Stage-I checkpoint).

StrategyQA (Stage-I checkpoint)Accuracy (%) ↑Vote@4 (%) ↑
Retain warm-up LoRA71.8372.49
Remove warm-up LoRA72.2773.36

Removing the warm-up LoRA substantially improves generation performance without modifying the projector, supporting the separation of latent-path learning and downstream policy optimization.

We will clarify both transitions and their empirical evaluations in Appendix B.1.

Q3. How is the latent path selected for M=1 inference, and why does Table 1 report 84.31% while Figure 5(a) reports 84.51%?

Thank you for pointing out this ambiguity. Both results use the same trained checkpoint and evaluation settings but differ in how the four latent paths are evaluated.

Table 1 (84.31%, single-path inference). For M=1, we use balanced random path assignment with a fixed seed (41). Each test question is assigned one of the four paths before decoding, and only one response is generated per question. On GSM8K, the four paths are assigned to 330, 329, 330, and 330 questions, respectively, yielding 1,112 correct answers out of 1,319 questions (84.31%).

Figure 5(a) (84.51%, mean-path accuracy). To analyze path complementarity, we evaluate each of the four paths on all 1,319 GSM8K questions. The paths correctly answer 1,121, 1,109, 1,118, and 1,111 questions, respectively. Averaging their individual accuracies yields 84.51%.

Thus, the difference arises from evaluating one randomly assigned path per question versus averaging all four paths over the complete test set. It does not reflect different model checkpoints or an inconsistent evaluation protocol.

We will explicitly describe the balanced random path-selection strategy in Section 3.5 and clarify the mean-path accuracy calculation in the caption of Figure 5(a).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.