acceptodds
Under review as a conference paper at ICLR 2027

SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization

Abstract

The soft-thinking reasoning paradigm has demonstrated superior performance over traditional discrete-token Chain-of-Thought (CoT) reasoning in various scenarios. However, unlike discrete-token CoT reasoning that can be reinforced through advanced reinforcement learning with verifiable rewards (RLVR) techniques such as group relative policy optimization (GRPO), extending the soft-thinking reasoning with such strong techniques remains challenging. This difficulty stems from the complexities of injecting stochasticity into soft-thinking tokens and updating soft-thinking policies accordingly. As a result, previous attempts to combine soft-thinking with RLVR typically underperform their discrete-token RLVR counterparts. To fully unlock the potential of soft-thinking, this paper presents a powerful policy optimization algorithm, SofT-GRPO. It injects the Gumbel noise into token probabilities with Gumbel-Softmax for controllable stochasticity, and leverages the Gumbel reparameterization trick to achieve accurate credit assignment to LLM soft-thinking policies. We conduct experiments over LLMs ranging from 1.5B to 7B parameters, where SofT-GRPO enables LLMs with soft-thinking to slightly outperform discrete-token CoT GRPO on Pass@1 (+0.13% on average accuracy), and brings a substantial uplift on Pass@32 (+2.19% on average).The codes are available at https://anonymous.4open.science/r/SofT-GRPO-master-A512.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HpytOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SofT-GRPO, an extension of GRPO to soft-thinking reasoning. It employs Gumbel-Softmax to generate stochastic soft-thinking trajectories and Gumbel reparameterization to compute action likelihood ratios for policy optimization. Experiments across three models and five mathematical benchmarks show comparable Pass@1 and improved Pass@32 over discrete-token GRPO.

Critical strengths
  • The paper introduces a Gumbel-based action parameterization that enables stochastic soft-thinking trajectories and provides tractable action likelihood ratios for policy optimization.

  • The paper evaluates SofT-GRPO across multiple model sizes, reasoning benchmarks, and additional model families, including out-of-domain generalization and compatibility with Dr.GRPO.

Critical weaknesses
  • Unfair comparison due to inconsistent training and inference configurations. The main experiments do not adequately isolate the contribution of SofT-GRPO from differences in sampling configurations. In particular, the soft-thinking baselines use an inference temperature of 0.6, top-k = 30, and Gumbel temperature = 0.5, whereas SofT-GRPO uses 1.0, top-k = 5, and Gumbel temperature = 0.1. These parameters directly influence the sampling distribution and potentially Pass@K performance. Consequently, the reported improvements cannot be confidently attributed to the proposed optimization method rather than differences in decoding configurations. A controlled comparison using matched or comparably tuned inference configurations is needed to substantiate the claimed gains.

  • Limited practical gains relative to computational overhead. Although SofT-GRPO improves multi-sample reasoning coverage, its practical benefits remain unclear. The average Pass@1 improvement over discrete-token GRPO is only 0.23 percentage points, while Appendix D.3 reports approximately 2–3× higher training wall-clock time. Furthermore, the paper does not provide a systematic comparison of inference latency, throughput, or accuracy under matched inference compute budgets. Without such evaluations, it is difficult to determine whether the modest performance improvements justify the additional computational cost.

Questions for authors

Could the authors provide comparisons under matched or independently tuned decoding configurations (temperature, top-k, and Gumbel temperature) to isolate the contribution of SofT-GRPO?

Given the 2–3× training overhead, how does SofT-GRPO compare with discrete-token GRPO under matched training and inference compute budgets?

Appendix E.7 shows training collapse under certain hyperparameter settings. How sensitive is SofT-GRPO to these choices, and does this sensitivity persist across model scales?

Could the authors report results across multiple training seeds to establish the reliability of the relatively small performance gains?

11 comments
AuthorsOpenReview ↗

Response to Reviewer Hpyt (1/2)

Thank you very much for your time and effort in reviewing our work. We address your concerns as follows.

Weakness 1 & Question 1. Decoding configurations: "A controlled comparison using matched or comparably tuned inference configurations is needed to substantiate the claimed gains."

Thanks for your suggestion. In our original manuscript, the baselines are evaluated under temperature=0.6, top-k=30, τg\tau_g=0.5 (setting A), and SofT-GRPO under temperature=1, top-k=5, τg\tau_g=0.1 (setting B). We have fixed it by evaluating the baselines under setting B and SofT-GRPO under setting A (Table 1, App. E.6, and the new Sec. 4.8).

  1. SofT-GRPO still outperforms the soft-thinking baselines under setting B (Table 1, rows marked †; @1/@16/@32, averaged over the five benchmarks):
ModelBase (soft, setting B)GRPO (soft, setting B)SofT-GRPO (setting B)
R1-Distill-Qwen-1.5B58.22/79.66/83.9661.24/81.64/83.2861.39/83.54/85.18
LLaMA-3.2-3B-Instruct25.52/53.20/56.5032.68/53.94/57.0232.83/54.96/57.06
R1-Distill-Qwen-7B73.60/87.63/88.9873.63/87.31/88.6574.04/87.98/90.16

On LLaMA-3B, the two methods are close under setting B (+0.15 Pass@1 and +0.04 Pass@32), while on 1.5B and 7B SofT-GRPO is higher by +1.90 and +1.51 at Pass@32.

  1. SofT-GRPO also outperforms the discrete-token baselines under setting B and different temperatures (Tables 16 and 17). On 1.5B, both are lower than SofT-GRPO under setting B; on 7B and 3B, even with the best temperature (0.6 to 1.4) for each baseline, SofT-GRPO keeps the highest Pass@32 and the highest average over @1/@8/@16/@32 (Avg.):
ModelBase (discrete)GRPO (discrete)SofT-GRPO
R1-Distill-Qwen-1.5B, setting B (@1/@16/@32)58.22/83.06/84.1861.24/81.68/82.5061.39/83.54/85.18
R1-Distill-Qwen-7B, best temperature (Pass@32/Avg.)88.2/84.1889.5/84.1090.16/84.65
LLaMA-3.2-3B-Instruct, best temperature (Pass@32/Avg.)56.7/46.1256.9/48.3357.06/48.71

We hope these supplemented temperature ablations sufficiently address your concern about the decoding temperature.

  1. SofT-GRPO's performance decreases under the OOD setting A (Tables 17 and 18). As shown in the table below, since SofT-GRPO is trained under setting B, setting A is out of its training distribution, and its average drops on 1.5B, although its Pass@16 and Pass@32 are still the highest among all methods under setting A. Therefore, we suggest evaluating SofT-GRPO with its training setting.
R1-Distill-Qwen-1.5B (@1/@16/@32)Setting ASetting B
Discrete-Token GRPO61.28/80.16/82.3961.24/81.68/82.50
SofT-GRPO60.04/82.22/84.5661.39/83.54/85.18

Under these controlled comparisons, SofT-GRPO maintains its lead compared to baselines, further validating our conclusions.

AuthorsOpenReview ↗

Response to Reviewer Hpyt (2/2)

Weakness 2 & Question 2. Gain vs. training and inference compute: "Given the 2–3× training overhead, how does SofT-GRPO compare with discrete-token GRPO under matched training and inference compute budgets?"

Thanks for your comment. We would like to make two clarifications.

First, we would like to make a clarification on efficiency: the 2–3× training overhead refers to wall-clock time rather than compute. We acknowledge that our original manuscript only reported the wall-clock time. The revised manuscript derives the training FLOPs and estimates them from the training logs (Sec. 3.3 and 4.7); as shown below, they are almost the same as discrete-token GRPO:

ModelTraining FLOPs (SofT/GRPO)Wall-clock (SofT/GRPO)Extra wall-clock in rollout
R1-Distill-Qwen-1.5B1.03×2.96×89%
LLaMA-3.2-3B-Instruct0.63×2.08×96%
R1-Distill-Qwen-7B1.07×1.92×80%

Second, the 0.23-point Pass@1 improvement is not the main gain we claim. Our claim is comparable Pass@1 and improved Pass@32 over discrete-token GRPO, consistent with your summary.

For inference compute, SofT-GRPO is ahead at equal generated tokens: on 1.5B, its Pass@16 (83.54) beats the Pass@32 of discrete-token GRPO (82.39) with 62% of the tokens. Per generated token, our prototype engine is currently 2.6–4.7× slower at the same FLOPs, which we acknowledge as a limitation of the current implementation (Sec. 4.7 and App. D.3).

Question 3. Sensitivity across scales: "How sensitive is SofT-GRPO to these choices, and does this sensitivity persist across model scales?"

Thanks for your question. The collapse in App. E.7 (τg\tau_g=0.25 on R1-Distill-Qwen-1.5B) is model-dependent: on Qwen2.5-1.5B-Math, training with τg\tau_g=0.25 is stable and close to the default τg\tau_g=0.1 (App. E.7). With the default setting, training is stable in all nine training pairs across six base models from 1.5B to 7B.

Question 4. Multiple training seeds: "Could the authors report results across multiple training seeds to establish the reliability of the relatively small performance gains?"

Thanks for your suggestion. We train both methods with three seeds on Qwen2.5-1.5B-Math (App. E.10). The lowest seed of SofT-GRPO is still higher than the highest seed of GRPO at Pass@16 and Pass@32, and SofT-GRPO has the higher Pass@32 in all nine training pairs of the paper over six base models (sign test, p=0.004). The block-wise tests in App. E.9 also show significant gains over GRPO (p=0.0067/0.022/0.0057 on AIME24 Pass@16 and AMC23 Pass@32/64):

Qwen2.5-1.5B-Math (mean±std)@1@16@32
Discrete-Token GRPO43.69±0.1666.60±0.4671.54±0.51
SofT-GRPO44.17±0.7068.36±0.5474.45±0.23

We sincerely hope our responses have addressed your concerns, and that you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.

Reviewer_HpytOpenReview ↗

Official Review of Submission1978 by Reviewer Hpyt

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns. However, two issues remain:

Limited practical applicability: Given the relatively niche soft-thinking setting, the modest performance gains (+0.23 Pass@1 and +2.19 Pass@32) seem difficult to justify against the 2.6–4.7× per-token inference latency. Without evidence of benefits under matched wall-clock budgets, I remain unconvinced of the method's practical value.

Hyperparameter robustness: The additional stability experiment only covers another 1.5B model. Have the authors evaluated sensitivity to Gumbel temperature and top-p on larger models (3B/7B)?

Overall, even if these remaining concerns were addressed, I would still find the practical impact and broader significance of the work limited, given its substantial inference overhead and relatively narrow applicability. Therefore, I am not inclined to recommend acceptance.

Reviewer_aRSLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SofT-GRPO extends GRPO to soft thinking, where intermediate steps use mixtures of token embeddings. It introduces Gumbel noise to sample these mixtures and treats the perturbed log-probabilities as continuous actions. Their densities provide the likelihood ratios needed for policy updates. Across the primary mathematical benchmarks, the method achieves comparable Pass@1 and higher Pass@32 than discrete-token GRPO. Additional experiments examine other models, code generation, and integration with Dr.GRPO.

Critical strengths
  1. The paper addresses a concrete challenge in training soft-thinking models. The continuous-action formulation provides a tractable way to evaluate sampled actions during policy updates.
  2. The evaluation extends beyond the main models. Additional model families, code tasks, and the Dr.GRPO experiments support the broader usefulness of the approach.
Critical weaknesses
  1. The Gaussian ablation does not represent Soft Tokens. Table 5 adds Gaussian noise to token probabilities, while Soft Tokens perturbs embeddings. These are different mechanisms, so the ablation does not establish superiority over the closest method.

  2. The computational cost is substantial relative to the gains. Training takes approximately 2–3 times longer than discrete-token GRPO. The primary average gains are modest, particularly the 0.23-point improvement in Pass@1.

  3. Training is sensitive to sampling settings. Some nearby configurations, such as increasing the Gumbel temperature from 0.1 to 0.25, lead to training collapse.

Questions for authors
  1. How much of the improvement comes from soft-policy training versus the decoding configuration? Could you examine this distinction beyond the 1.5B model?
  2. How does SofT-GRPO compare when the additional training time is available to discrete-token GRPO? This would clarify the practical value of the reported gains.
6 comments
AuthorsOpenReview ↗

Response to Reviewer aRSL (1/2)

Thank you very much for your time and effort in reviewing our work. We address your concerns as follows.

Weakness 1. Meaning of the ablation: "The Gaussian ablation does not represent Soft Tokens. Table 5 adds Gaussian noise to token probabilities, while Soft Tokens perturbs embeddings ... the ablation does not establish superiority over the closest method."

Thanks for your comment. We would like to make a clarification: Table 5 is not presented as a comparison with Soft Tokens. In Sec. 5 (l. 466–467 of the original manuscript), we have explicitly stated that "The Gaussian variant here perturbs probabilities; Soft Tokens perturbs embeddings." The ablation in Table 5 answers a different question: within the probability-space parameterization of SofT-GRPO, which kind of noise should be injected. Among the three choices, Gumbel noise achieves the best average Pass@1 and Pass@16.

The only Soft Tokens results are in Table 4, and its caption states that Soft Tokens "was not reproduced in our pipeline". Therefore, the paper does not claim a controlled superiority over Soft Tokens. To make this clearer, we remove the comparative sentence in Sec. 4.5 and retitle it as a reference comparison, and we state at the start of the ablation in Sec. 5 that only the noise distribution is varied.

Weakness 2 & Question 2. Gain vs. compute: "The computational cost is substantial relative to the gains ... particularly the 0.23-point improvement in Pass@1. How does SofT-GRPO compare when the additional training time is available to discrete-token GRPO?"

Thanks for your comment. We would like to make two clarifications.

First, the 0.23-point Pass@1 improvement is not the main gain we claim. Our abstract states that SofT-GRPO "retain[s] comparable Pass@1", as also noted in your summary ("comparable Pass@1 and higher Pass@32 than discrete-token GRPO"). Since prior soft-thinking RLVR methods report a lower Pass@1 than discrete-token GRPO (as reported by Butt et al. [3]), matching it is already non-trivial. The gain we claim is multi-sample coverage: +2.19 Pass@32 on average over three models, and 75.5 vs. 70.9 Maj@32 on 1.5B (Table 3).

Second, we would like to make a clarification on efficiency: the 2–3× refers to wall-clock time rather than compute. We acknowledge that our original manuscript only reported the wall-clock time. In the revised manuscript, we derive the training FLOPs and estimate them from the training logs (Sec. 3.3 and 4.7). As shown below, SofT-GRPO consumes almost the same training FLOPs as discrete-token GRPO (0.63× on LLaMA-3B because of its shorter responses):

ModelTraining FLOPs (SofT/GRPO)Wall-clock (SofT/GRPO)Extra wall-clock in rollout
R1-Distill-Qwen-1.5B1.03×2.96×89%
LLaMA-3.2-3B-Instruct0.63×2.08×96%
R1-Distill-Qwen-7B1.07×1.92×80%

For giving discrete-token GRPO more training time, we believe that a fair comparison of two RL algorithms is at matched training compute rather than matched wall-clock time. Since the FLOPs are already matched, giving GRPO the same wall-clock time would give it 1.8–3.3× more compute than SofT-GRPO. The wall-clock gap mainly reflects the maturity of the inference engines: discrete-token decoding has years of kernel and serving optimization, while soft-thinking rollout does not yet. Moreover, longer discrete RLVR is not expected to recover the coverage: discrete-token GRPO already lowers Pass@32 below the base model on two of three models (Table 1), consistent with the finding of Yue et al. [1]. We acknowledge that the longer wall-clock time is a limitation of the current SofT-GRPO implementation, which we state in Sec. 4.7 and App. D.3.

AuthorsOpenReview ↗

Response to Reviewer aRSL (2/2)

Weakness 3. Sensitivity to τg\tau_g and top-p: "Training is sensitive to sampling settings. Some nearby configurations, such as increasing the Gumbel temperature from 0.1 to 0.25, lead to training collapse."

Thanks for your comment. We agree that SofT-GRPO collapses with τg\tau_g=0.25 or top-p=1.0 on R1-Distill-Qwen-1.5B. However, we would like to highlight that this sensitivity is model-dependent, and the default setting is stable:

  1. Both collapses come with a sharp rise of the KL divergence to the reference policy (Fig. 6a), while the PPO-KL between consecutive policies stays below 1e-5 (Fig. 6b). This suggests that it is not the precision-mismatch collapse reported by Qi et al. [2]. Intuitively, a larger τg\tau_g flattens the Gumbel-Softmax weights, and top-p=1.0 admits the long tail of low-probability tokens into the mixture; both push soft tokens away from the token embeddings (Lemma 3.1).

  2. On Qwen2.5-1.5B-Math, training with τg\tau_g=0.25 is stable, with performance close to the default setting and still higher than discrete-token GRPO (App. E.7 and Table 20; @1 is Mean@32 and @K is Pass@K, averaged over the five benchmarks):

Qwen2.5-1.5B-Math@1@16@32
SofT-GRPO, τg\tau_g=0.1 (3 seeds)44.1768.3674.45
SofT-GRPO, τg\tau_g=0.2544.4267.0873.65
Discrete-Token GRPO (3 seeds)43.6966.6071.54
  1. With the default τg\tau_g=0.1, training is stable in all nine training pairs of the paper, across six base models, Dr.GRPO, and code tasks.

Question 1. Training vs. decoding beyond 1.5B: "How much of the improvement comes from soft-policy training versus the decoding configuration? Could you examine this distinction beyond the 1.5B model?"

Thanks for your suggestion. We address the two parts as follows.

  1. SofT-GRPO training brings gains beyond the decoding setting. In the revised Table 1, we add the soft-thinking baselines under SofT-GRPO's decoding setting (temperature=1, top-k=5, τg\tau_g=0.1; setting B) for all three models (rows marked †). Compared with the baselines' original setting (temperature=0.6, top-k=30, τg\tau_g=0.5; setting A), this separates the effect of decoding from the effect of training (@1/@32, averaged over the five benchmarks):
ModelDecoding only (base, A→B)Training (base B→SofT-GRPO)SofT-GRPO vs. GRPO (soft), both B
R1-Distill-Qwen-1.5B+1.94/+1.55+3.17/+1.22+0.15/+1.90
LLaMA-3.2-3B-Instruct+0.56/+1.11+7.31/+0.56+0.15/+0.04
R1-Distill-Qwen-7B+0.16/−0.40+0.44/+1.18+0.41/+1.51

Decoding alone changes the base model by −0.4 to +1.9 points, while SofT-GRPO training brings further gains on every model and metric.

  1. SofT-GRPO outperforms the discrete-token baselines at every temperature beyond 1.5B. In the original manuscript, the temperature ablation of the discrete-token baselines was only conducted on 1.5B. We extend it to R1-Distill-Qwen-7B and LLaMA-3.2-3B-Instruct with temperatures from 0.6 to 1.4 (App. E.6, Table 16). As shown below (Avg. over @1/@8/@16/@32), SofT-GRPO is higher than both discrete-token baselines at every temperature, and it also keeps the highest Pass@32 (90.16 and 57.06 vs. at most 89.5 and 56.9):
Avg. (@1/@8/@16/@32)Temp. 0.6Temp. 0.8Temp. 1.0Temp. 1.2Temp. 1.4SofT-GRPO
R1-Distill-Qwen-7B, Base84.1883.9583.8082.7876.7084.65
R1-Distill-Qwen-7B, GRPO83.8183.2884.0884.1083.6284.65
LLaMA-3.2-3B-Instruct, Base44.4046.1243.4537.9031.9848.71
LLaMA-3.2-3B-Instruct, GRPO46.9447.7048.3347.7747.2548.71

We hope these supplemented temperature ablations sufficiently address your concern about the decoding temperature.


We sincerely hope our responses have addressed your concerns, and that you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.

References

[1] Yue, Yang, et al. "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" NeurIPS 2025.

[2] Qi, Penghui, et al. "Defeating the Training-Inference Mismatch via FP16." arXiv:2510.26788, 2025.

[3] Butt, Natasha, et al. "Soft Tokens, Hard Truths." arXiv:2509.19170, 2025.

Thank you for the detailed response and additional experiments. I acknowledge that my original comment did not clearly distinguish the purpose of the Gaussian ablation from a direct comparison with Soft Tokens. The FLOPs analysis and the additional training-versus-decoding controls substantially address my concerns. I still have reservations about the longer wall-clock training time and the small additional gains on some models. However, these concerns no longer outweigh the contribution. I am therefore raising my rating from 2 to 3.

Reviewer_TW7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines the soft-thinking idea with RLVR via Gumbel noise injection during GRPO to enable likelihood optimization and eventually benefit training performance. The authors provide ample intuition via math derivation and theorems which are accompanied by numerical results of qwen and llama model families on math/science/coding benchmarks.

Critical strengths

A good extension of vanilla soft thinking, deterministic with no likelihood to optimize, to the GRPO optimization process via the Gumbel-Softmax techniques. The accompanying mathematics is thorough and makes the estimator's properties checkable.

Critical weaknesses
  1. The Pass@32 gain of Soft-GRPO is not significant and the Major@32 tells almost the same story.
  2. No compute-matched training comparison, no Pass@K at equal generated tokens or latency, no measured decoding latency for soft thinking.
Questions for authors
  1. W1: Pass@32 over the base model is +1.95 / +0.34 / +2.09 in Table 1; the block-wise tests show no significant difference from the base; AIME24 Pass@64 equals the base in Table 15; and with majority voting, Major@32 is 75.5 for SofT-GRPO vs 75.4 for the base model in Table 3. Is there any benchmark where SofT-GRPO is significantly above the base model at Pass@K or Major@K? Please report training-seed variance for at least one model, since no training repeats are given.
  2. W1: It seems that SofT-GRPO is evaluated at temperature 1.0 and top-k 5, the baselines at temperature 0.6 and top-k 30. Please report the baselines under SofT-GRPO's settings and SofT-GRPO under the baselines'.
  3. W2: Soft-based training takes 2–3× the wall-clock of the discrete case. It is natural to ask what does discrete GRPO reach at the same training FLOPs, or SofT-GRPO at the baseline's budget?
  4. W2: Indeed that responses are +23% (1.5B) and +3.7% (7B) longer than discrete GRPO. Please report Pass@K and Major@K at equal generated tokens or equal latency.
  5. W2: The asymptotic latency argument in the appendix is not enough for real-world cases. Please report measured per-token latency and throughput for soft vs discrete settings.
6 comments
AuthorsOpenReview ↗

Response to Reviewer TW7x (1/3)

Thank you very much for your time and effort in reviewing our work. We address your concerns as follows.

Weakness 1. Significance of the gain: "The Pass@32 gain of Soft-GRPO is not significant and the Major@32 tells almost the same story."

Thanks for your comment. We would like to clarify that this judgement is based on the comparison with the base model. Our paper claims the gain of SofT-GRPO over discrete-token GRPO, the RLVR baseline it is proposed to improve, and does not claim significance over the base model. Against discrete-token GRPO, the Pass@K gain is statistically significant on AIME24 and AMC23, and Maj@32 tells a different story:

  1. Pass@K. As shown in the block-wise test in App. E.9, SofT-GRPO significantly outperforms discrete-token GRPO on AIME24 Pass@16 (p=0.0067), AMC23 Pass@32 (p=0.022), and AMC23 Pass@64 (p=0.0057).

  2. Compared with discrete-token GRPO, Maj@32 tells a different story (Table 3). Discrete-token GRPO loses 4.5 points of Maj@32 compared with the base model (75.4 to 70.9), while SofT-GRPO keeps it (75.5), 4.6 points higher than discrete-token GRPO.

Question 1. Significance above the base model: "Pass@32 over the base model is +1.95 / +0.34 / +2.09 ... Is there any benchmark where SofT-GRPO is significantly above the base model at Pass@K or Major@K?"

Thanks for your question. Yes, with the same block-wise test as App. E.9, SofT-GRPO is significantly above the base model on AMC23 Pass@64 (p=0.029), where, as in Table 1, the base model uses setting A and SofT-GRPO uses setting B. We complement this part in the revised manuscript Appendix E.9. Beyond single benchmarks, we would like to clarify two aspects.

  1. Our claim against the base model is outperformance rather than significance. We claim that SofT-GRPO outperforms the base model at Pass@K, while discrete-token GRPO improves Pass@1 at the cost of Pass@K, as also observed by Yue et al. [1]. We do not claim significance against the base model on every benchmark, and we report all p-values in the paper for transparency, including those above 0.05.

  2. The lead is supported by consistent evidence across models. As shown below (averaged over the five benchmarks, Tables 1 and 11), discrete-token GRPO is lower than the base model at Pass@32 on three of the six models, while SofT-GRPO outperforms the base model at Pass@32 on all six models, and outperforms discrete-token GRPO at both Pass@1 and Pass@32 on all six models.

Model (@1/@32)BaseDiscrete-Token GRPOSofT-GRPO
R1-Distill-Qwen-1.5B58.09/83.2361.28/82.3961.39/85.18
LLaMA-3.2-3B-Instruct25.79/56.7232.60/55.3232.83/57.06
R1-Distill-Qwen-7B73.62/88.0773.69/88.1274.04/90.16
Qwen2.5-1.5B3.99/34.9427.33/46.8428.78/59.71
Qwen2.5-1.5B-Math36.36/69.3343.83/72.0543.96/74.44
Qwen3-1.7B59.30/80.0658.36/76.6460.42/82.69

Question 1: AIME24 Pass@64 equals the base in Table 15.

For AIME24 Pass@64 in Table 15 of the submission (Table 19 in the revision), with 64 samples, both models reach the same coverage on AIME24 (23.9 of 30 problems), so Pass@64 no longer separates them on this 30-problem benchmark. On AMC23, SofT-GRPO is significantly higher at Pass@64 than the base model and discrete-token GRPO (0.9891 vs. 0.9750 and 0.9688, p=0.029 and 0.0057, App. E.9).

Question 1. Training-seed variance: "Please report training-seed variance for at least one model, since no training repeats are given."

Thanks for your suggestion. We train both methods with three seeds on Qwen2.5-1.5B-Math (App. E.10). As shown below (@1 is Mean@32 and @K is Pass@K, averaged over the five benchmarks), the lowest seed of SofT-GRPO is still higher than the highest seed of GRPO at Pass@16 and Pass@32 (74.22 vs. 72.05 at Pass@32), and their Pass@1 is within one standard deviation.

Qwen2.5-1.5B-Math (mean±std)@1@16@32
Discrete-Token GRPO43.69±0.1666.60±0.4671.54±0.51
SofT-GRPO44.17±0.7068.36±0.5474.45±0.23
AuthorsOpenReview ↗

Response to Reviewer TW7x (2/3)

Weakness 1 & Question 2. Evaluation settings: "Please report the baselines under SofT-GRPO's settings and SofT-GRPO under the baselines'."

Thanks for your suggestion. In our original manuscript, the baselines are evaluated under temperature=0.6, top-k=30, τg\tau_g=0.5 (setting A), and SofT-GRPO under temperature=1, top-k=5, τg\tau_g=0.1 (setting B). We have fixed it by evaluating the baselines under setting B and SofT-GRPO under setting A (Table 1, App. E.6, and the new Sec. 4.8).

  1. SofT-GRPO still outperforms the soft-thinking baselines under setting B (Table 1, rows marked †). As shown below, this holds on all three models (@1/@16/@32):
ModelBase (soft, setting B)GRPO (soft, setting B)SofT-GRPO (setting B)
R1-Distill-Qwen-1.5B58.22/79.66/83.9661.24/81.64/83.2861.39/83.54/85.18
LLaMA-3.2-3B-Instruct25.52/53.20/56.5032.68/53.94/57.0232.83/54.96/57.06
R1-Distill-Qwen-7B73.60/87.63/88.9873.63/87.31/88.6574.04/87.98/90.16

On LLaMA-3B, the two methods are close under setting B (+0.15 Pass@1 and +0.04 Pass@32), while on 1.5B and 7B SofT-GRPO is higher by +1.90 and +1.51 at Pass@32.

  1. SofT-GRPO also outperforms the discrete-token baselines under setting B and different temperatures (Tables 16 and 17). We evaluate the discrete-token baselines under setting B on 1.5B, and extend the temperature ablation (0.6 to 1.4) in App. E.6 to 7B and 3B. Even with the best temperature for each baseline, SofT-GRPO keeps the highest Pass@32 and the highest average over @1/@8/@16/@32 (Avg.):
ModelBase (discrete)GRPO (discrete)SofT-GRPO
R1-Distill-Qwen-1.5B, setting B (@1/@16/@32)58.22/83.06/84.1861.24/81.68/82.5061.39/83.54/85.18
R1-Distill-Qwen-7B, best temperature (Pass@32/Avg.)88.2/84.1889.5/84.1090.16/84.65
LLaMA-3.2-3B-Instruct, best temperature (Pass@32/Avg.)56.7/46.1256.9/48.3357.06/48.71

We hope these supplemented temperature ablations sufficiently address your concern about the decoding temperature.

  1. SofT-GRPO's performance decreases under the OOD setting A (Tables 17 and 18). As shown in the table below, since SofT-GRPO is trained under setting B, setting A is out of its training distribution, and its average drops on 1.5B, although its Pass@16 and Pass@32 are still the highest among all methods under setting A. Therefore, we suggest evaluating SofT-GRPO with its training setting.
R1-Distill-Qwen-1.5B (@1/@16/@32)Setting ASetting B
Discrete-Token GRPO61.28/80.16/82.3961.24/81.68/82.50
SofT-GRPO60.04/82.22/84.5661.39/83.54/85.18
AuthorsOpenReview ↗

Response to Reviewer TW7x (3/3)

Weakness 2 & Question 3. Equal training FLOPs: "Soft-based training takes 2–3× the wall-clock of the discrete case. It is natural to ask what does discrete GRPO reach at the same training FLOPs, or SofT-GRPO at the baseline's budget?"

Thanks for your question. We would like to make a clarification on the efficiency of SofT-GRPO: the 2–3× wall-clock time does not mean 2–3× training FLOPs. We acknowledge that our original manuscript only reported the wall-clock time and did not analyze FLOPs. In the revised manuscript, we derive the training FLOPs in Sec. 3.3 and estimate them from the training logs in Sec. 4.7. As shown below, SofT-GRPO consumes almost the same training FLOPs as discrete-token GRPO, and most of the extra wall-clock time comes from rollout generation in our prototype soft-thinking engine:

ModelTraining FLOPs (SofT/GRPO)Wall-clock (SofT/GRPO)Extra wall-clock in rollout
R1-Distill-Qwen-1.5B1.03×2.96×89%
LLaMA-3.2-3B-Instruct0.63×2.08×96%
R1-Distill-Qwen-7B1.07×1.92×80%

The 0.63× on LLaMA-3B comes from SofT-GRPO's shorter responses. Since both methods are also trained with a similar number of steps (608–629) and the same batch size, Table 1 is a comparison under matched training compute.

Weakness 2 & Question 4. Equal generated tokens: "Please report Pass@K and Major@K at equal generated tokens or equal latency."

Thanks for your suggestion. Using the measured response lengths (Table 15 in the revision), we compare SofT-GRPO with discrete-token GRPO under the token budget of GRPO's 32 samples:

ModelSofT-GRPODiscrete-Token GRPOTokens used by SofT-GRPO
R1-Distill-Qwen-1.5BPass@16=83.54Pass@32=82.3962%
R1-Distill-Qwen-1.5BMaj@16=73.7Maj@32=70.962%
LLaMA-3.2-3B-InstructPass@32=57.06Pass@32=55.3214%
R1-Distill-Qwen-7BPass@32=90.16Pass@32=88.12104%

On 1.5B, SofT-GRPO with only 16 samples already outperforms discrete-token GRPO with 32 samples, using 62% of its tokens. On LLaMA-3B, SofT-GRPO only uses 14% of the tokens. On 7B, SofT-GRPO uses 4% more tokens for 32 samples and is 2.0 points higher at Pass@32.

Weakness 2 & Question 5. Measured latency: "Please report measured per-token latency and throughput for soft vs discrete settings."

Thanks for your suggestion. In our training runs on 8×H200, the soft-thinking rollout takes 3.96×, 4.67×, and 2.60× the time per generated token of discrete-token rollout (1.5B/3B/7B), while the FLOPs per token are the same (App. D.3, Table 9). This is a limitation of our current implementation rather than of the algorithm: existing inference engines are highly optimized for discrete tokens, but not yet for soft-thinking. We have added this discussion in Sec. 4.7, and we expect future inference-system optimization to reduce this overhead.


We sincerely hope our responses have addressed all your concerns, and that you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.

References

[1] Yue, Yang, et al. "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" NeurIPS 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We sincerely thank all reviewers for their constructive comments and valuable suggestions. The current manuscript has received positive evaluations regarding its novelty, effectiveness, and impact:

  • Reviewer TW7x: SofT-GRPO is a good extension of soft thinking to GRPO via Gumbel-Softmax, with thorough and checkable mathematics.
  • Reviewer aRSL: The paper addresses a concrete challenge with a tractable continuous-action formulation, and the additional models, code tasks, and Dr.GRPO experiments support its broader usefulness.
  • Reviewer Hpyt: The Gumbel-based parameterization gives tractable likelihood ratios for soft-thinking, evaluated across model sizes, benchmarks, and model families.

We address some common concerns shared by different reviewers in this response. All revisions are marked in blue in the updated PDF.

Common Concern 1: Matched decoding settings (Reviewer TW7x, Reviewer aRSL, Reviewer Hpyt).

Reviewers pointed out that the decoding settings were not controlled in the original version. In our original manuscript, we mainly use temperature=0.6, and top-k=30, τg\tau_g=0.5 (denoted as setting A) for baselines and temperature=1, top-k=5, τg=0.1\tau_g=0.1 (denoted as setting B) for SofT-GRPO. We have fixed it by evaluating the baselines under setting B (soft-thinking baselines on all three models, discrete-token baselines on 1.5B, plus a temperature sweep on all three models) and SofT-GRPO under setting A.

We update Table 1, App. E.6, and add one new Sec. 4.8 in the revised manuscript. As briefly shown in the table below, under identical decoding, SofT-GRPO remains ahead of the soft-thinking baselines on all three models (by +1.90/+0.04/+1.51 Pass@32 on 1.5B/3B/7B), further validating our conclusions.

Model (@1/@32)Base (soft, setting B)GRPO (soft, setting B)SofT-GRPO (setting B)
R1-Distill-Qwen-1.5B58.22/83.9661.24/83.2861.39/85.18
LLaMA-3.2-3B-Instruct25.52/56.5032.68/57.0232.83/57.06
R1-Distill-Qwen-7B73.60/88.9873.63/88.6574.04/90.16

Common Concern 2: Training cost (Reviewer TW7x, Reviewer aRSL, Reviewer Hpyt).

As a limitation of current SofT-GRPO, it leads to 2–3× more wall-clock time compared to GRPO. However, we would like to make a clarification that SofT-GRPO only uses almost the same training compute (FLOPs) as discrete-token GRPO. As shown in the table below, in terms of FLOPs, SofT-GRPO is actually almost aligned with GRPO; the longer clock-time is a result of the specific infrastructure optimization for discrete-token decoding.

ModelTraining FLOPs (SofT-GRPO/Discrete-Token GRPO)Wall-clock (SofT-GRPO/Discrete-Token GRPO)Extra wall-clock in rollout
R1-Distill-Qwen-1.5B1.03×2.96×89%
LLaMA-3.2-3B-Instruct0.63×2.08×96%
R1-Distill-Qwen-7B1.07×1.92×80%

To show this, our revised manuscript derives the training FLOPs in Sec. 3.3 and estimates them from the training logs in Sec. 4.7. We show that the compute of SofT-GRPO is matched with discrete-token GRPO, and the extra wall-clock comes from rollout generation in our prototype soft-thinking engine.


As contributions of SofT-GRPO: this paper presents a powerful RLVR framework for optimizing soft-thinking trajectories through Gumbel-Softmax group rollout and Gumbel-reparameterized policy optimization, leading to better Pass@K performance compared to discrete-token GRPO.

Point-to-point responses can be found below. We are also glad to continually improve our work to address any further concerns.

Best Regards, Paper1978 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.