- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SofT-GRPO, an extension of GRPO to soft-thinking reasoning. It employs Gumbel-Softmax to generate stochastic soft-thinking trajectories and Gumbel reparameterization to compute action likelihood ratios for policy optimization. Experiments across three models and five mathematical benchmarks show comparable Pass@1 and improved Pass@32 over discrete-token GRPO.
Critical strengths
-
The paper introduces a Gumbel-based action parameterization that enables stochastic soft-thinking trajectories and provides tractable action likelihood ratios for policy optimization.
-
The paper evaluates SofT-GRPO across multiple model sizes, reasoning benchmarks, and additional model families, including out-of-domain generalization and compatibility with Dr.GRPO.
Critical weaknesses
-
Unfair comparison due to inconsistent training and inference configurations. The main experiments do not adequately isolate the contribution of SofT-GRPO from differences in sampling configurations. In particular, the soft-thinking baselines use an inference temperature of 0.6, top-k = 30, and Gumbel temperature = 0.5, whereas SofT-GRPO uses 1.0, top-k = 5, and Gumbel temperature = 0.1. These parameters directly influence the sampling distribution and potentially Pass@K performance. Consequently, the reported improvements cannot be confidently attributed to the proposed optimization method rather than differences in decoding configurations. A controlled comparison using matched or comparably tuned inference configurations is needed to substantiate the claimed gains.
-
Limited practical gains relative to computational overhead. Although SofT-GRPO improves multi-sample reasoning coverage, its practical benefits remain unclear. The average Pass@1 improvement over discrete-token GRPO is only 0.23 percentage points, while Appendix D.3 reports approximately 2–3× higher training wall-clock time. Furthermore, the paper does not provide a systematic comparison of inference latency, throughput, or accuracy under matched inference compute budgets. Without such evaluations, it is difficult to determine whether the modest performance improvements justify the additional computational cost.
Questions for authors
Could the authors provide comparisons under matched or independently tuned decoding configurations (temperature, top-k, and Gumbel temperature) to isolate the contribution of SofT-GRPO?
Given the 2–3× training overhead, how does SofT-GRPO compare with discrete-token GRPO under matched training and inference compute budgets?
Appendix E.7 shows training collapse under certain hyperparameter settings. How sensitive is SofT-GRPO to these choices, and does this sensitivity persist across model scales?
Could the authors report results across multiple training seeds to establish the reliability of the relatively small performance gains?
11 comments
Response to Reviewer Hpyt (2/2)
Weakness 2 & Question 2. Gain vs. training and inference compute: "Given the 2–3× training overhead, how does SofT-GRPO compare with discrete-token GRPO under matched training and inference compute budgets?"
Thanks for your comment. We would like to make two clarifications.
First, we would like to make a clarification on efficiency: the 2–3× training overhead refers to wall-clock time rather than compute. We acknowledge that our original manuscript only reported the wall-clock time. The revised manuscript derives the training FLOPs and estimates them from the training logs (Sec. 3.3 and 4.7); as shown below, they are almost the same as discrete-token GRPO:
| Model | Training FLOPs (SofT/GRPO) | Wall-clock (SofT/GRPO) | Extra wall-clock in rollout |
|---|---|---|---|
| R1-Distill-Qwen-1.5B | 1.03× | 2.96× | 89% |
| LLaMA-3.2-3B-Instruct | 0.63× | 2.08× | 96% |
| R1-Distill-Qwen-7B | 1.07× | 1.92× | 80% |
Second, the 0.23-point Pass@1 improvement is not the main gain we claim. Our claim is comparable Pass@1 and improved Pass@32 over discrete-token GRPO, consistent with your summary.
For inference compute, SofT-GRPO is ahead at equal generated tokens: on 1.5B, its Pass@16 (83.54) beats the Pass@32 of discrete-token GRPO (82.39) with 62% of the tokens. Per generated token, our prototype engine is currently 2.6–4.7× slower at the same FLOPs, which we acknowledge as a limitation of the current implementation (Sec. 4.7 and App. D.3).
Question 3. Sensitivity across scales: "How sensitive is SofT-GRPO to these choices, and does this sensitivity persist across model scales?"
Thanks for your question. The collapse in App. E.7 (=0.25 on R1-Distill-Qwen-1.5B) is model-dependent: on Qwen2.5-1.5B-Math, training with =0.25 is stable and close to the default =0.1 (App. E.7). With the default setting, training is stable in all nine training pairs across six base models from 1.5B to 7B.
Question 4. Multiple training seeds: "Could the authors report results across multiple training seeds to establish the reliability of the relatively small performance gains?"
Thanks for your suggestion. We train both methods with three seeds on Qwen2.5-1.5B-Math (App. E.10). The lowest seed of SofT-GRPO is still higher than the highest seed of GRPO at Pass@16 and Pass@32, and SofT-GRPO has the higher Pass@32 in all nine training pairs of the paper over six base models (sign test, p=0.004). The block-wise tests in App. E.9 also show significant gains over GRPO (p=0.0067/0.022/0.0057 on AIME24 Pass@16 and AMC23 Pass@32/64):
| Qwen2.5-1.5B-Math (mean±std) | @1 | @16 | @32 |
|---|---|---|---|
| Discrete-Token GRPO | 43.69±0.16 | 66.60±0.46 | 71.54±0.51 |
| SofT-GRPO | 44.17±0.70 | 68.36±0.54 | 74.45±0.23 |
We sincerely hope our responses have addressed your concerns, and that you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.
Official Review of Submission1978 by Reviewer Hpyt
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns. However, two issues remain:
Limited practical applicability: Given the relatively niche soft-thinking setting, the modest performance gains (+0.23 Pass@1 and +2.19 Pass@32) seem difficult to justify against the 2.6–4.7× per-token inference latency. Without evidence of benefits under matched wall-clock budgets, I remain unconvinced of the method's practical value.
Hyperparameter robustness: The additional stability experiment only covers another 1.5B model. Have the authors evaluated sensitivity to Gumbel temperature and top-p on larger models (3B/7B)?
Overall, even if these remaining concerns were addressed, I would still find the practical impact and broader significance of the work limited, given its substantial inference overhead and relatively narrow applicability. Therefore, I am not inclined to recommend acceptance.
Response to Reviewer Hpyt (1/2)
Thank you very much for your time and effort in reviewing our work. We address your concerns as follows.
Thanks for your suggestion. In our original manuscript, the baselines are evaluated under temperature=0.6, top-k=30, τg=0.5 (setting A), and SofT-GRPO under temperature=1, top-k=5, τg=0.1 (setting B). We have fixed it by evaluating the baselines under setting B and SofT-GRPO under setting A (
Table 1,App. E.6, and the newSec. 4.8).Table 1, rows marked †; @1/@16/@32, averaged over the five benchmarks):On LLaMA-3B, the two methods are close under setting B (+0.15 Pass@1 and +0.04 Pass@32), while on 1.5B and 7B SofT-GRPO is higher by +1.90 and +1.51 at Pass@32.
Tables 16 and 17). On 1.5B, both are lower than SofT-GRPO under setting B; on 7B and 3B, even with the best temperature (0.6 to 1.4) for each baseline, SofT-GRPO keeps the highest Pass@32 and the highest average over @1/@8/@16/@32 (Avg.):We hope these supplemented temperature ablations sufficiently address your concern about the decoding temperature.
Tables 17 and 18). As shown in the table below, since SofT-GRPO is trained under setting B, setting A is out of its training distribution, and its average drops on 1.5B, although its Pass@16 and Pass@32 are still the highest among all methods under setting A. Therefore, we suggest evaluating SofT-GRPO with its training setting.Under these controlled comparisons, SofT-GRPO maintains its lead compared to baselines, further validating our conclusions.