acceptodds
Under review as a conference paper at ICLR 2027

Sparse RL Relaxation for LLM Reasoning

24 views

Abstract

Recent work has explored continuous representations in place of discrete tokens during Chain-of-Thought (CoT) reasoning, often referred to as soft CoT. This is motivated by the idea that a mixture of token embeddings can represent a superposition of multiple reasoning possibilities within a single step. However, it remains challenging to train large language models to realize this potential. Existing work has explored reinforcement learning (RL)-style training for soft CoT. However, these approaches treat soft tokens as actions in a Markov decision process rather than as differentiable intermediate computation steps. Thus, they do not propagate gradients across the soft tokens and miss out on the signal of how the choice of one soft token affects the next. In this paper, we propose a sparse relaxation method for RL (called SpaRLax), which leverages the differentiability of soft-CoT reasoning over time steps and makes training more stable and effective. We adopt an entmax sparsity method that adaptively determines the degree of softness, which truncates long-tail noise at confident steps while retaining multiple plausible candidates when uncertain. Across nine reasoning benchmarks, our method achieves the highest average performance compared with baseline methods. Further analyses validate the effective use of soft-token superposition in our approach, and also demonstrate the contribution of adaptive sparsity in our approach.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, large language models, soft thinking, soft tokens, math reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_byezOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an RL algorithm for soft chain-of-thought reasoning. Unlike prior methods, it treats soft tokens as differentiable intermediate computation steps. The authors claim that this design makes the algorithm more stable and effective.

Critical strengths
  1. The paper is clearly written and easy to follow, making the motivation and methodology of SpaRLax relatively easy to understand.

  2. The analyses of KL divergence and hidden-state cosine similarity in Figure 2 are well motivated and helpful for understanding the method.

Critical weaknesses
  1. The motivation for claiming that existing methods are ill-defined under the MDP framework needs further clarification, and an important baseline is missing.

    • The paper argues that methods such as SofT-GRPO and STHT are less well-founded because they use soft-token embeddings during reasoning and discrete tokens during answering. However, MDPs can support hybrid action spaces as long as the corresponding action distributions are properly defined.
    • Multiplex Thinking (Tang et al., 2026) is a relevant baseline that uses discrete token sampling and embedding aggregation to support soft reasoning while retaining a standard discrete-token MDP formulation. Since it also uses DeepSeek-R1-Distill-Qwen-1.5B and GRPO, comparing against it would help clarify the theoretical motivation and advantages of SpaRLax.
  2. The improvements in pass@1 and pass@32 appear modest.

    • In Tables 1 and 2, the w.avg. improvements over discrete GRPO and soft GRPO under the soft-inference setup are approximately 1 percentage point. The lack of variance estimates for pass@32 and the high standard deviations on AIME24 and AIME25 in Table 1 make the robustness of these gains difficult to assess.
    • Reporting means and standard deviations across multiple runs for Table 2, along with AIME24 and AIME25 results under a longer inference budget (e.g., 32k tokens), would help clarify the results.
Questions for authors
  1. Could the authors provide additional experimental results to help clarify the findings in Table 2? Specifically, could they report means and standard deviations for Table 2 and provide results with a longer inference budget for AIME24 and AIME25?

  2. Could the authors clarify precisely why existing soft-RL methods are invalid under the MDP framework? They should further discuss the fundamental advantages of SpaRLax's cross-step differentiable gradients over approaches such as Multiplex Thinking, which aggregates embeddings after discrete sampling, and provide a comparison with Multiplex Thinking under the same experimental settings.

Reviewer_C8CWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpaRLax trains a model to use mixtures of token embeddings during intermediate reasoning steps. It adds a gradient connection between neighboring steps so that the final answer can provide a training signal to earlier reasoning. Across nine benchmarks, it reports the highest weighted-average Pass@1 under both mixture-based and ordinary token-by-token inference.

Critical strengths
  • The paper addresses a clear limitation in training models that use mixtures of token embeddings for intermediate reasoning.

  • It evaluates both inference settings across math, knowledge, and code tasks.

  • Its candidate-size experiment shows that the proposed way of combining tokens remains effective as the candidate pool grows.

Critical weaknesses
  • Figure 3 studies how token candidates are combined, but there is no matched experiment that isolates the benefit of the proposed gradient connection.

  • Some baseline settings differ, making the main performance comparison harder to interpret.

  • The reported variation comes from repeated evaluation of the same checkpoint; variation across independent training runs is not reported.

Questions for authors
  • What happens if the proposed gradient connection is disabled while all other settings remain the same?

  • Could you clarify the differing baseline settings and report results across independent training runs?

Reviewer_2TUQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpaRLax, a sparse reinforcement learning relaxation framework for training large language models with soft chain-of-thought reasoning. The method is motivated by two claimed limitations of prior soft-CoT RL approaches: they do not effectively exploit differentiable dependencies across consecutive soft reasoning steps, and they rely on dense soft-token distributions that may introduce long-tail noise over large vocabularies. To address these issues, the paper combines Gumbel-based reparameterization with entmax-based adaptive sparsity, and further introduces Proxy Gradient Grafting to make the method compatible with the common two-pass RL training pipeline. The paper evaluates the approach on nine reasoning benchmarks spanning math, knowledge, and code tasks, and reports improvements over several discrete- and soft-CoT RL baselines under both hard- and soft-CoT inference. Additional analyses examine superposition utilization and the effect of adaptive sparsity.

Critical strengths
  1. The paper addresses a timely and interesting problem: how to better train LLMs to exploit soft chain-of-thought reasoning. The focus on optimization and credit assignment in soft-CoT RL is meaningful and potentially important for continuous reasoning methods.

  2. The proposed method is reasonably well designed and empirically promising. In particular, combining differentiable soft-token construction with adaptive sparsity is intuitive, and the experiments cover multiple reasoning domains and show consistent gains in weighted-average Pass@1 over the reported baselines.

Critical weaknesses
  1. My main concern is that the paper’s central claim about cross-step gradient propagation is not fully established by the efficient implementation actually used. In Algorithm 2, Proxy Gradient Grafting applies stop-gradient to preceding rollout embeddings, which appears to remove part of the recursive dependency between consecutive soft-token distributions. As a result, it is unclear to what extent the method preserves the full cross-step learning signal emphasized in the paper. This issue is consequential because it directly affects the validity of the paper’s core technical contribution.

  2. The experimental evidence does not sufficiently isolate the contributions of the two main components: the proposed gradient mechanism and the entmax-based adaptive sparsity. The current ablations do not clearly show how much of the gain comes from cross-step differentiable optimization versus sparsity, and the paper also lacks evidence from multiple independent training runs. This makes the empirical support for the main claims less convincing than I would like.

Questions for authors
  1. In Algorithm 2, which gradient terms from the full recursive soft-CoT computation are preserved, and which are discarded due to the stop-gradient operation on preceding rollout embeddings? A precise clarification would help assess whether Proxy Gradient Grafting truly supports the paper’s cross-step credit-assignment claim.

  2. Which implementation is used for the main results in Tables 1 and 2: the full recursive version (Algorithm 1) or the proxy version (Algorithm 2)? If the proxy version is used, can the authors provide either performance comparisons or gradient-similarity evidence relative to the full version?

  3. Can the authors provide controlled ablations that separately evaluate (a) the effect of the proposed differentiable gradient mechanism and (b) the effect of entmax-based adaptive sparsity?

  4. Are the reported means and standard deviations based on repeated evaluations of a single checkpoint or on multiple independently trained runs? If they are based on a single checkpoint, do the authors have evidence that the gains are stable across training seeds?

  5. Were all compared methods trained on exactly the same RL data and under comparable compute/training budgets? Some clarification would help assess the fairness of the baseline comparisons.

Reviewer_gKmtOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sparse RL Relaxation (SpaRLax), a post-training method for soft chain-of-thought reasoning in LLMs. It constructs soft-token distributions using Gumbel-softmax and a proxy gradient grafting technique to allow gradients to flow across reasoning steps in two-pass RL implementations, and replaces softmax with entmax to produce sparse soft-token distributions. Experiments train DeepSeek-R1-Distill-Qwen-1.5B on DAPO-MATH-17k and evaluate on nine math, knowledge, and code benchmarks under hard- and soft-CoT inference, with additional analyses of superposition utilization and candidate-set size effects.

Critical strengths
  1. Specific problem positioning: The paper clearly identifies a concrete limitation in existing soft-CoT RL methods, namely treating soft tokens as continuous RL actions and thereby weakening the differentiable structure and cross-step gradient signal.

  2. Clear method chain: The approach is presented as a coherent sequence, using Gumbel reparameterization to construct soft-token distributions and a proxy gradient grafting mechanism to preserve the forward rollout while enabling gradient flow to the previous step.

  3. Clear presentation: The paper is organized and written in a clear manner, making the motivation, method, and experimental logic easy to follow.

  4. Engineering-oriented design: The method includes practical implementation considerations that improve its feasibility for real training pipelines.

Critical weaknesses
  1. Incomplete cross-step credit assignment: The proxy gradient explicitly propagates only one step back, so the practical coverage of “cross-step learning” remains limited. The authors also acknowledge that gradients do not fully backpropagate along the entire soft-token sequence.

  2. Limited experimental scale: Experimental validation is restricted to a single 1.5B model, and the model is relatively old. This limits the evidence for broader generalization.

  3. Parameter settings need clarification: The authors do not state key inference-time parameters such as the maximum response length, leaving the evaluation setup unclear. This omission may affect the comparability of reported scores and warrants clarification.

  4. Efficiency claims lack quantitative data: The paper emphasizes low latency and parallelizability of proxy gradient grafting, but provides no system-level comparisons such as memory usage, throughput, or training time.

Questions for authors

Please see Weaknesses.

1 comment

Thanks for the authors’ response. I do really like aspects such as the presentation of this paper, but the overall evaluation should be determined more by factors such as technical and experimental credibility (no misclicks). Although the authors provided a detailed response, unfortunately I do not think my concern has been fully resolved, so I have decided to keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.