- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an RL algorithm for soft chain-of-thought reasoning. Unlike prior methods, it treats soft tokens as differentiable intermediate computation steps. The authors claim that this design makes the algorithm more stable and effective.
Critical strengths
-
The paper is clearly written and easy to follow, making the motivation and methodology of SpaRLax relatively easy to understand.
-
The analyses of KL divergence and hidden-state cosine similarity in Figure 2 are well motivated and helpful for understanding the method.
Critical weaknesses
-
The motivation for claiming that existing methods are ill-defined under the MDP framework needs further clarification, and an important baseline is missing.
- The paper argues that methods such as SofT-GRPO and STHT are less well-founded because they use soft-token embeddings during reasoning and discrete tokens during answering. However, MDPs can support hybrid action spaces as long as the corresponding action distributions are properly defined.
- Multiplex Thinking (Tang et al., 2026) is a relevant baseline that uses discrete token sampling and embedding aggregation to support soft reasoning while retaining a standard discrete-token MDP formulation. Since it also uses DeepSeek-R1-Distill-Qwen-1.5B and GRPO, comparing against it would help clarify the theoretical motivation and advantages of SpaRLax.
-
The improvements in pass@1 and pass@32 appear modest.
- In Tables 1 and 2, the w.avg. improvements over discrete GRPO and soft GRPO under the soft-inference setup are approximately 1 percentage point. The lack of variance estimates for pass@32 and the high standard deviations on AIME24 and AIME25 in Table 1 make the robustness of these gains difficult to assess.
- Reporting means and standard deviations across multiple runs for Table 2, along with AIME24 and AIME25 results under a longer inference budget (e.g., 32k tokens), would help clarify the results.
Questions for authors
-
Could the authors provide additional experimental results to help clarify the findings in Table 2? Specifically, could they report means and standard deviations for Table 2 and provide results with a longer inference budget for AIME24 and AIME25?
-
Could the authors clarify precisely why existing soft-RL methods are invalid under the MDP framework? They should further discuss the fundamental advantages of SpaRLax's cross-step differentiable gradients over approaches such as Multiplex Thinking, which aggregates embeddings after discrete sampling, and provide a comparison with Multiplex Thinking under the same experimental settings.
Thanks for the authors’ response. I do really like aspects such as the presentation of this paper, but the overall evaluation should be determined more by factors such as technical and experimental credibility (no misclicks). Although the authors provided a detailed response, unfortunately I do not think my concern has been fully resolved, so I have decided to keep my current score.