acceptodds
Under review as a conference paper at ICLR 2027

Beyond Uniform Soft Thinking: Adaptive Gating and Calibration of Soft Tokens for Latent Reasoning

Abstract

Decoding-time soft thinking extends discrete chain-of-thought reasoning by feeding back a probability-weighted mixture of token embeddings rather than a single-token embedding. However, applying such mixtures at every step does not distinguish between steps with a clear leading token and those with several plausible continuations. Recent selective methods decide when to activate soft mixing, but construct mixtures with a fixed candidate budget or over the full vocabulary. Selecting the steps does not determine how widely to mix, since their next-token distributions can still differ in candidate count and probability concentration. We therefore propose Adaptive Gating and Calibration of Soft Tokens (AGC-ST), a training-free decoding method that determines both when and how widely to mix from the local next-token distribution. Its gate combines entropy and the top-1/top-2 probability margin to select steps where probability mass is broadly distributed and the two most likely tokens have similar probabilities. For each gated step, the Rényi-0.5 effective number, which reflects both candidate count and probability concentration, sets a support cap. A cumulative-mass prefix then selects the final mixing support within this cap. We evaluate AGC-ST on five reasoning models from the Qwen and Llama families, spanning 7B–32B parameters, across five benchmarks covering mathematical, coding, and scientific reasoning. With one shared hyperparameter configuration, AGC-ST achieves the highest accuracy among the evaluated methods in all 25 model–benchmark settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XXZQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AGC-ST, a training-free decoding method that adaptively determines when to apply soft-token mixing and how many candidate tokens to include. The method combines an entropy threshold with the probability gap between the top two candidates, activating soft mixing at positions where the distribution is sufficiently uncertain and no candidate clearly dominates. At selected positions, it uses the Rényi effective number to set a support-size cap, followed by a cumulative-probability rule to determine the final mixing support. Other positions retain standard discrete sampling. The authors evaluate the method on five reasoning models across five benchmarks covering mathematical, scientific, and coding tasks. With a shared hyperparameter configuration, AGC-ST achieves the highest accuracy among the evaluated methods in all 25 model–benchmark settings.

Critical strengths
  1. The method achieves consistent accuracy improvements across five models and five benchmarks, obtaining the highest accuracy among the evaluated methods in all 25 model–benchmark settings.

  2. The appendix provides statistical support for the reported improvements. On the five-benchmark mean, the 95% paired bootstrap confidence intervals exclude zero in 34 of the 35 model–baseline comparisons, including comparisons with CoT sampling for all five models.

Critical weaknesses
  1. Although Table 2 shows that the tightest gate is not the best-performing setting, the ablations do not fully separate the effect of position selection from that of mixing frequency. Activation-rate-matched entropy-only, margin-only, or random controls would help establish the specific benefit of the joint criterion.

  2. The comparison with closely related selective Soft Thinking methods is incomplete. Although SeLaR and SwiReasoning are evaluated, ThinkRouter is discussed but not included in the experiments. Since ThinkRouter also routes between soft and discrete decoding based on model confidence, this omission limits the evidence for AGC-ST’s advantage over existing routing strategies.

Questions for authors
  1. Appendix E.3 explicitly states that the interventions characterize one-step behavior rather than accuracy. Beyond the aggregate support-rule comparisons in Table 3, is there evidence linking these local prediction changes to improved final-answer correctness?

  2. Does the benefit of joint gating come from selecting better positions or simply applying soft mixing less often? Does it still outperform entropy-only gating or random selection when their activation rates are matched?

  3. Could the authors include a comparison with ThinkRouter, or explain why it was not included?

  4. Table 5 uses top-10 mixing and fixed Cold Stop settings for vanilla Soft Thinking, whereas the original paper reports model-dependent top-n settings and searches over Cold Stop parameters. How were these baseline settings selected, and would model-specific tuning change the comparison?

5 comments
AuthorsOpenReview ↗

Response to Reviewer XXZQ (1/3)

Thank you for your detailed review and concrete suggestions. We appreciate your recognition of the consistent gains across 25 settings and the supporting paired bootstrap analysis. We addressed each comment with new experiments:

  • Activation-rate-matched controls (W1, Q2). At approximately matched activation rates, the joint gate outperforms entropy-only gating, margin-only gating and random selection for both models tested on AIME2024/2025.
  • Comparison with ThinkRouter (W2, Q3). AGC-ST is higher than ThinkRouter in all 25 settings.
  • Top-1 substitution in full rollouts (Q1). Replacing the mixture with the top-1 embedding at gated steps lowers accuracy for all three models tested on AIME2024/2025.
  • Model-specific tuning of Soft Thinking (Q4). With top-n or the Cold Stop entropy threshold chosen per model, Soft Thinking is below AGC-ST on all five models.

We address W1 & Q2 here, W2 & Q3 in Part 2, and Q1 and Q4 in Part 3.

W1 & Q2. Separating position selection from mixing frequency

"Although Table 2 shows that the tightest gate is not the best-performing setting, the ablations do not fully separate the effect of position selection from that of mixing frequency. Activation-rate-matched entropy-only, margin-only, or random controls would help establish the specific benefit of the joint criterion."

"Does the benefit of joint gating come from selecting better positions or simply applying soft mixing less often? Does it still outperform entropy-only gating or random selection when their activation rates are matched?"

Thank you for these questions. At approximately matched activation rates, the joint gate still outperforms entropy-only gating, margin-only gating and random selection for both models tested on AIME2024/2025. Its benefit does not come from applying soft mixing less often.

To establish the specific benefit of the joint criterion, we built three controls with activation rates close to that of AGC-ST. We used DS-Qwen-7B and DS-Llama-8B on AIME2024/2025. Support calibration and decoding are unchanged.

  • Entropy only: a higher entropy threshold, without the margin condition.
  • Margin only: a lower margin threshold, without the entropy condition.
  • Random gate: random selection among steps whose calibrated support has at least two candidates, at the activation rate of AGC-ST.

Table XXZQ-1. Gate ablation on AIME2024/2025 at approximately matched activation rates (Avg@16). Same runs as Table Qc6c-2. CoT is shown for reference. Rate is the percentage of gated thinking steps.

SettingDS-Qwen-7B AIME2024RateDS-Qwen-7B AIME2025RateDS-Llama-8B AIME2024RateDS-Llama-8B AIME2025RateMean
CoT54.790%40.630%41.880%31.040%42.09
Entropy only (α\alpha=1.60 / 1.56, β\beta=1.00)52.081.16%40.831.29%42.501.59%31.461.85%41.72
Margin only (α\alpha=0.00, β\beta=0.002)53.332.01%40.212.08%42.711.51%30.001.63%41.56
Random gate55.001.12%38.541.21%39.381.58%31.041.79%40.99
AGC-ST (α\alpha=1.25, β\beta=0.10)56.881.14%42.501.18%44.171.52%32.291.70%43.96

AGC-ST is higher than each control for both models tested on AIME2024/2025, with a mean of 43.96 versus 41.72 (entropy only), 41.56 (margin only), and 40.99 (random gate). The entropy-only and random gates closely match the activation rate of AGC-ST. On DS-Llama-8B, the margin-only gate also matches it and is still 1.46 and 2.29 points lower on AIME2024 and AIME2025. On DS-Qwen-7B, its rate stays near 2% even at β\beta=0.002, about 1.8 times that of AGC-ST.

If the benefit came only from mixing less often, a gate with the same activation rate would match AGC-ST. The random gate has this rate but selects steps at random. It is 2.97 points lower at 40.99. Mixing even less does not help either. The tightest gate in Tab. 2 mixes about a quarter to a third as often as AGC-ST and is 0.78 points lower at 43.18. Mixing frequency therefore does not explain the benefit of the joint criterion. The benefit depends on which steps it selects. We have added these comparisons to block (d) of Tab. 2.

AuthorsOpenReview ↗

Response to Reviewer XXZQ (2/3)

W2 & Q3. Comparison with ThinkRouter

"Although SeLaR and SwiReasoning are evaluated, ThinkRouter is discussed but not included in the experiments. Since ThinkRouter also routes between soft and discrete decoding based on model confidence, this omission limits the evidence for AGC-ST's advantage over existing routing strategies."

"Could the authors include a comparison with ThinkRouter, or explain why it was not included?"

Thank you for this suggestion. To test the advantage of AGC-ST over existing routing strategies, we added ThinkRouter to all 25 model–benchmark settings. With ThinkRouter included, AGC-ST still achieves the highest accuracy among the evaluated methods in all 25 settings.

ThinkRouter uses top-10 soft mixing when the top-token probability is at least a threshold τ\tau, and discrete sampling otherwise. We re-implemented it in our framework following its public code, with the decoding settings, token budget and evaluation protocol of Tab. 1. The thresholds follow the values reported in the ThinkRouter paper where available. Otherwise, we transferred them from the Qwen3 model of the closest size. LCB uses the MBPP value. ThinkRouter thus uses a separate threshold for each benchmark and model size. AGC-ST uses one configuration for all 25 settings.

Table XXZQ-2. ThinkRouter and AGC-ST across five models and five benchmarks (Avg@16 on AIME, Avg@4 otherwise).

ModelMethodAIME2024AIME2025GPQA-DMBPPLCBMean
DS-Qwen-7BThinkRouter54.1739.7949.6285.9938.8953.69
AGC-ST56.8842.5050.6387.4539.6155.41
DS-Llama-8BThinkRouter43.3330.6344.3284.9237.6348.17
AGC-ST44.1732.2946.4686.3839.7049.80
Qwen3-8BThinkRouter74.3864.3858.3394.0756.2769.48
AGC-ST77.5070.0061.1194.8459.9572.68
QwQ-32BThinkRouter77.5063.7561.6297.5762.1072.51
AGC-ST79.3870.2165.6697.7663.3575.27
Qwen3-32BThinkRouter81.2569.3864.3995.9160.2274.23
AGC-ST81.8874.3866.9296.5066.0477.14

Table XXZQ-3. Paired comparison of AGC-ST and ThinkRouter on the five-benchmark mean. Δ is AGC-ST minus ThinkRouter.

ModelAGC-ST MeanThinkRouter MeanΔ [95% CI]
DS-Qwen-7B55.4153.69+1.72 [+0.08, +3.33]
DS-Llama-8B49.8048.17+1.63 [+0.13, +3.16]
Qwen3-8B72.6869.48+3.20 [+1.88, +4.57]
QwQ-32B75.2772.51+2.76 [+1.65, +3.94]
Qwen3-32B77.1474.23+2.91 [+1.65, +4.20]

AGC-ST is higher than ThinkRouter in all 25 settings. On the five-benchmark mean, it exceeds ThinkRouter by 1.63–3.20 points. The paired bootstrap intervals (same procedure as App. E.1) exclude zero for all five models. So the advantage of AGC-ST also holds against an existing confidence-based routing strategy. We have added ThinkRouter to Tab. 1, its settings to App. D.4 (Tabs. 5 and 6), and its paired comparisons to Tab. 9.

AuthorsOpenReview ↗

Response to Reviewer XXZQ (3/3)

Q1. Local prediction changes and final-answer correctness

"Appendix E.3 explicitly states that the interventions characterize one-step behavior rather than accuracy. Beyond the aggregate support-rule comparisons in Table 3, is there evidence linking these local prediction changes to improved final-answer correctness?"

Thank you for this insightful question. To examine this link, we applied the substitution from App. E.3 in full rollouts. Gated steps receive the top-1 embedding instead of the mixture. This lowers final-answer accuracy for all three models tested on AIME2024/2025.

In App. E.3, the AGC-ST mixture and the top-1 input give a different next-step argmax at about 47% of the same gated states. Here the joint gate and the sampling at non-gated steps are unchanged. Only the input at gated steps differs: the top-1 embedding, a fixed top-2 mixture, or the AGC-ST mixture. We used DS-Qwen-7B, DS-Llama-8B and QwQ-32B on AIME2024 and AIME2025.

Table XXZQ-4. Final-answer accuracy with different gated-step inputs on AIME2024/2025, with the joint gate fixed (Avg@16). Same runs as Table rd2a-1. CoT is shown for reference.

SettingDS-Qwen-7B AIME2024DS-Qwen-7B AIME2025DS-Llama-8B AIME2024DS-Llama-8B AIME2025QwQ-32B AIME2024QwQ-32B AIME2025Mean
CoT54.7940.6341.8831.0477.5069.5852.57
Fixed size (top-1, argmax)55.2139.3843.1230.4277.2967.7152.19
Fixed size (top-2)56.2540.6243.7531.4677.5068.5453.02
AGC-ST56.8842.5044.1732.2979.3870.2154.24

AGC-ST is higher than the top-1 input for all three models tested on AIME2024/2025. The mean gain is 2.05 points, from 52.19 to 54.24. The top-2 mixture lies in between in every case (53.02 on average). The mixture that changes next-step predictions in App. E.3 thus also improves final-answer accuracy over the top-1 input in full rollouts. We have added the top-1 and top-2 variants to Tab. 3(b) and their discussion to App. E.3.

Q4. Soft Thinking baseline settings

"Table 5 uses top-10 mixing and fixed Cold Stop settings for vanilla Soft Thinking, whereas the original paper reports model-dependent top-n settings and searches over Cold Stop parameters. How were these baseline settings selected, and would model-specific tuning change the comparison?"

Thank you for this question. The Soft Thinking settings follow its official release. With top-n or the Cold Stop entropy threshold chosen per model, Soft Thinking is still below AGC-ST on all five models.

How the settings were selected. We used the configuration of the official repository [1] and its released evaluation script [2] (Tab. 5). This configuration uses top-10 mixing, Cold Stop entropy threshold 0.01, and patience 256. This Cold Stop setting is one of the combinations searched in the Soft Thinking paper.

Model-specific tuning. To test whether per-model tuning changes the comparison, we varied top-n (5, 10, 15) and the Cold Stop entropy threshold (0.01, 0.05, 0.10) on AIME2024 for all five models. We changed one parameter at a time and kept the other at its default value.

Table XXZQ-5. Soft Thinking with different top-n and Cold Stop entropy thresholds on AIME2024 (Avg@4).

Soft Thinking settingDS-Qwen-7BDS-Llama-8BQwen3-8BQwQ-32BQwen3-32B
Top-n (entropy threshold 0.01)
top-550.0030.0075.8372.5078.33
top-1050.8323.3375.8375.8379.17
top-1544.1725.0074.1778.3380.00
Entropy threshold (top-10)
0.0547.5027.5075.8376.6780.83
0.1046.6725.8371.6773.3376.67
AGC-ST55.0049.1778.3381.6784.17

With its best setting, Soft Thinking is below AGC-ST on all five models. Model-specific tuning of top-n and the Cold Stop entropy threshold therefore does not change the comparison. We have added these results to App. D.4 (Tab. 7).

[1] Soft Thinking official repository

[2] Soft Thinking evaluation script


Thank you again for your concrete suggestions. Following your comments, we conducted activation-rate-matched controls, a comparison with ThinkRouter, the top-1 substitution in full rollouts, and model-specific tuning of Soft Thinking. We have incorporated these results into the revised paper, with all changes highlighted in red. We hope these additional experiments address your concerns. We would be grateful if you would consider raising your score. We would be happy to discuss any remaining questions.

Reviewer_Qc6cOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AGC-ST is a training-free decoder that uses entropy and the gap between the two highest token probabilities to decide when to mix token embeddings. A Renyi-0.5 cap and cumulative-mass rule select how many tokens to mix. The paper evaluates five models on five benchmarks, with ablations and local interventions.

Critical strengths
  1. AGC-ST’s gains across held-out settings are convincing. Table 1 reports the highest accuracy among the evaluated methods in all 25 settings, including 19 excluded from hyperparameter selection (§4.3). Under shared maximum-token and sampling budgets, mean gains over sampled CoT are 1.09–1.71 percentage points (Appendix D3). The nominal 95% paired confidence intervals for these mean gains are above zero for all five models (Table 8).
  2. Changing the number of mixed tokens at fixed model states can change DS-Qwen-7B’s next prediction on AIME2024 and LCB. Norm-matched controls strengthen this observation (§4.4 Fig. 3 and Appendix E3). The results clarify mixture behavior, but one-step changes do not establish accuracy gains.
Critical weaknesses

The ablations provide less general evidence for the individual design choices. Table 3 favors calibrated support over fixed top-4 in all six AIME settings, but these are also the disclosed selection settings (§4.3). In Table 2 single-signal gates mix more often than the joint gate, so the comparisons do not isolate the joint gate’s benefit at a matched mixing rate. The tighter joint gate counters a simple “less mixing is better” explanation.

Questions for authors
3 comments
AuthorsOpenReview ↗

Response to Reviewer Qc6c

Thank you for your thoughtful review and positive assessment of our work. We appreciate your recognition of the convincing held-out gains, paired bootstrap analysis, and same-state interventions. Following your comment on the ablations, we extended the support comparison to benchmarks not used for hyperparameter selection (W1(a)) and compared the gates at approximately matched rates (W1(b)). AGC-ST outperforms the alternatives in both.

W1(a). Support ablation on benchmarks not used for hyperparameter selection

"The ablations provide less general evidence for the individual design choices. Table 3 favors calibrated support over fixed top-4 in all six AIME settings, but these are also the disclosed selection settings (§4.3)."

To provide more general evidence, we evaluated the support ablation on two benchmarks not used for hyperparameter selection. Calibrated support improves over fixed top-4 on both.

We compared AGC-ST with fixed top-4 on GPQA-D and LCB, using the same three models as Tab. 3. Fixed top-4 is the variant in your comment. Only the support rule at gated steps differs between the two rows.

Table Qc6c-1. Support ablation on GPQA-D and LCB, with the joint gate fixed (Avg@4).

SettingDS-Qwen-7B GPQA-DDS-Qwen-7B LCBDS-Llama-8B GPQA-DDS-Llama-8B LCBQwQ-32B GPQA-DQwQ-32B LCBMean
Fixed size (top-4)48.6138.8944.0739.2561.9963.1749.33
AGC-ST50.6339.6146.4639.7065.6663.3550.90

AGC-ST is higher for all three models tested on both benchmarks. The mean gain is 1.57 points, from 49.33 to 50.90. Together with AIME2024/2025 in Tab. 3, AGC-ST outperforms fixed top-4 in all 12 comparisons. Calibrated support is therefore also effective on benchmarks not used for hyperparameter selection. We have added these results to App. E.5 (Tab. 16) and summarized them in Sec. 4.3.

W1(b). Gate comparison at a matched mixing rate

"In Table 2 single-signal gates mix more often than the joint gate, so the comparisons do not isolate the joint gate's benefit at a matched mixing rate. The tighter joint gate counters a simple 'less mixing is better' explanation."

Thank you for acknowledging that the tighter joint gate already rules out a simple "less mixing is better" explanation. At approximately matched mixing rates, the joint gate still outperforms the entropy-only, margin-only and random gates.

To isolate the benefit of the joint gate, we raised the entropy-only threshold and lowered the margin-only threshold. This brings their mixing rates close to that of AGC-ST on DS-Qwen-7B and DS-Llama-8B across AIME2024/2025. Support calibration and decoding are unchanged. Following a suggestion from Reviewer XXZQ, we also included a random gate. It selects thinking steps at random from those with a calibrated support of at least two candidates, at the mixing rate of AGC-ST.

Table Qc6c-2. Gate ablation on AIME2024/2025 at approximately matched mixing rates (Avg@16). CoT is shown for reference. Rate is the percentage of gated thinking steps.

SettingDS-Qwen-7B AIME2024RateDS-Qwen-7B AIME2025RateDS-Llama-8B AIME2024RateDS-Llama-8B AIME2025RateMean
CoT54.790%40.630%41.880%31.040%42.09
Entropy only (α\alpha=1.60 / 1.56, β\beta=1.00)52.081.16%40.831.29%42.501.59%31.461.85%41.72
Margin only (α\alpha=0.00, β\beta=0.002)53.332.01%40.212.08%42.711.51%30.001.63%41.56
Random gate55.001.12%38.541.21%39.381.58%31.041.79%40.99
AGC-ST (α\alpha=1.25, β\beta=0.10)56.881.14%42.501.18%44.171.52%32.291.70%43.96

AGC-ST is higher than each of the other gates for both models tested on AIME2024/2025, with a mean of 43.96 versus 41.72 (entropy only), 41.56 (margin only), and 40.99 (random gate). The entropy-only and random gates closely match the mixing rate of AGC-ST. On DS-Llama-8B, the margin-only gate also matches it and is still 1.46 and 2.29 points lower on AIME2024 and AIME2025. On DS-Qwen-7B, its rate stays near 2% even at β\beta=0.002, about 1.8 times that of AGC-ST. The benefit of the joint gate thus holds at approximately matched mixing rates. We have added these comparisons to block (d) of Tab. 2.


Thank you again for your helpful suggestions. We have added both ablations to the revised paper, with all changes highlighted in red. We would be happy to address any further questions.

Thank you for your response! With a fixed gate, calibration support provides higher accuracy than with a fixed top-4 for all models in GPQA-D and LiveCodeBench. This answers my concerns about the initial AIME comparison. I will keep my assessment unchanged.

Many thanks for checking our new results. We are pleased that the comparison on GPQA-D and LiveCodeBench answers your concern about the initial AIME comparison. Your valuable suggestion prompted us to extend the support ablation beyond the selection settings. This makes the evidence for calibrated support more general. Please let us know if anything else would be helpful. We are grateful for your positive assessment and your time.

Reviewer_rd2aOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free soft-thinking method, AGC-ST. By leveraging the signal from the overall sampling entropy and the top margin, AGC-ST determines when and how many tokens should be mixed in soft-thinking. Comprehensive experiments show the effectiveness of the proposed method on different backbones.

Critical strengths
  • An interesting and insightful study is given, which reveals that soft thinking cannot always outperform deterministic CoT.
  • The evaluation is comprehensive across different benchmarks and backbones.
Critical weaknesses
  • At gated steps, AGC-ST both makes the input soft and removes sampling randomness, and none of the ablations separates these two effects. Please add an ablation that decodes gated steps with argmax (or a lower local temperature) while all other steps keep sampling, and report CoT with different temperatures. It would help understand the significance of the performance gain.
  • Since the sampling randomness presents in the evaluation and the performance gain is not significant, evaluating with more random seeds will help validate the effectiveness of the proposed method.
Questions for authors
  • Will the proposed method still outperform other methods under the settings I mentioned in the weaknesses?
  • Entropy and margin are computed on the temperature-scaled distribution (tau=0.6). How do the gate rate and accuracy change at other temperatures with the thresholds fixed?

I'd like to raise my score if the effectiveness of the method is better proven, since it provides interesting insights.

5 comments
AuthorsOpenReview ↗

Response to Reviewer rd2a (1/3)

Thank you for your detailed review and constructive suggestions. We appreciate your recognition of our findings and comprehensive evaluation. We ran each experiment you suggested. AGC-ST still outperforms the compared methods in each study (Q1):

  • Gated-step decoding ablation (W1). With the same gate, the two discrete variants reach 52.19 (argmax) and 52.36 (sampling at TT=0.3) on average, compared with 54.24 for AGC-ST. AGC-ST is highest for all three models tested on AIME2024/2025.
  • Multi-seed evaluation (W2). Over three runs, AGC-ST has higher mean accuracy than CoT for all five models on AIME2024 and GPQA-D.
  • Temperature study with fixed thresholds (W1, Q2). For DS-Qwen-7B on AIME2024, the best CoT over five temperatures (55.00) is below AGC-ST at TT=0.6 (56.88). AGC-ST outperforms CoT from TT=0.4 to TT=0.8. At TT=1.0, it is 0.62 points below CoT with fixed thresholds. Computing the gate on the TT=0.6 distribution brings AGC-ST to 55.62. This is above CoT (54.37).

These results answer Q1. We address the other comments in three parts: W1 here, W2 in Part 2, and Q2 in Part 3.

W1. Separating soft input from sampling randomness at gated steps

"At gated steps, AGC-ST both makes the input soft and removes sampling randomness, and none of the ablations separates these two effects. Please add an ablation that decodes gated steps with argmax (or a lower local temperature) while all other steps keep sampling, and report CoT with different temperatures."

Thank you for this suggestion. To separate the two effects, we conducted the gated-step decoding ablation and evaluated CoT at five temperatures. Neither removing randomness at gated steps nor lowering the CoT temperature reproduces the gain of AGC-ST.

Gated-step decoding ablation. We kept the joint gate and the sampling at all other steps unchanged. Only the input at gated steps differs between rows. Two rows are discrete variants that remove or reduce randomness: (1) argmax and (2) one token sampled at a local temperature of 0.3 instead of 0.6. Two rows use a soft input: (3) a fixed top-2 mixture and (4) the AGC-ST mixture. We used DS-Qwen-7B, DS-Llama-8B, and QwQ-32B on AIME2024/2025.

Table rd2a-1. Gated-step decoding ablation on AIME2024/2025, with the joint gate fixed (Avg@16). CoT is shown for reference.

SettingDS-Qwen-7B AIME2024DS-Qwen-7B AIME2025DS-Llama-8B AIME2024DS-Llama-8B AIME2025QwQ-32B AIME2024QwQ-32B AIME2025Mean
CoT54.7940.6341.8831.0477.5069.5852.57
Fixed size (top-1, argmax)55.2139.3843.1230.4277.2967.7152.19
Sample one token (TT=0.3)56.4640.0043.5430.0077.5066.6752.36
Fixed size (top-2)56.2540.6243.7531.4677.5068.5453.02
AGC-ST56.8842.5044.1732.2979.3870.2154.24

These rows separate the two effects you raised. On average, the two discrete variants stay at the CoT level of 52.57 (52.19 and 52.36). The two soft inputs are both above CoT (53.02 and 54.24). AGC-ST is highest for all three models tested on AIME2024/2025. Removing randomness at gated steps therefore does not explain the gain.

CoT with different temperatures. For DS-Qwen-7B on AIME2024 (Avg@16), CoT accuracy at each temperature is:

TT0.40.50.60.81.0
CoT52.2955.0054.7954.7954.37

CoT is best at TT=0.5 (55.00). This is still below AGC-ST at its default TT=0.6 (56.88). Lowering randomness at every step does not reproduce the gain either. Part 3 reports AGC-ST at the same temperatures (Q2).

We have added this ablation to Tab. 3 of the revised paper (fixed top-1 and top-2 in block (b), sampling at TT=0.3 in block (e)). CoT results at these temperatures are in App. E.5 (Tab. 15).

AuthorsOpenReview ↗

Response to Reviewer rd2a (2/3)

W2. Evaluation with more random seeds

"Since the sampling randomness presents in the evaluation and the performance gain is not significant, evaluating with more random seeds will help validate the effectiveness of the proposed method."

Thank you for this suggestion. To validate the effectiveness of AGC-ST, we conducted experiments with three random seeds. Across the three runs, AGC-ST outperforms CoT for all five models on AIME2024 and GPQA-D.

Multi-seed evaluation. We ran AGC-ST and CoT three times on AIME2024 and GPQA-D for all five models, with the decoding settings of Tab. 1. Each run uses a different random seed and is a full evaluation (Avg@16 on AIME2024, Avg@4 on GPQA-D).

Table rd2a-2. AGC-ST and CoT across five models and three runs, reported as mean ± standard deviation (Avg@16 on AIME2024, Avg@4 on GPQA-D).

ModelMethodAIME2024GPQA-D
DS-Qwen-7BCoT54.72 ± 1.1549.07 ± 1.67
AGC-ST56.39 ± 0.6850.13 ± 0.51
DS-Llama-8BCoT42.71 ± 0.7545.24 ± 1.25
AGC-ST43.82 ± 0.6146.76 ± 0.86
Qwen3-8BCoT74.79 ± 1.1059.81 ± 0.39
AGC-ST76.11 ± 1.2060.31 ± 1.18
QwQ-32BCoT77.29 ± 0.2163.34 ± 0.76
AGC-ST78.40 ± 1.0564.06 ± 1.58
Qwen3-32BCoT80.69 ± 0.7965.45 ± 0.51
AGC-ST81.46 ± 0.9166.67 ± 0.55

The mean gains range from 0.50 to 1.67 points. The run-to-run variation of AGC-ST is comparable to that of CoT. This shows that the advantage holds across sampling runs.

Paired bootstrap intervals. The submitted paper already includes a paired bootstrap over questions (App. E.1, now Tab. 9). On the five-benchmark mean, AGC-ST improves over CoT by 1.09 to 1.71 points. The 95% intervals exclude zero for all five models. So the gain also holds when questions are resampled.

Comparison with other soft-thinking methods. On the five-benchmark mean, the six other soft-thinking methods in Tab. 1 are at most 0.39 points above CoT. Only 3 of their 30 method–model pairs are above CoT. AGC-ST is the only evaluated method above CoT for all five models. Its gain over CoT is larger than that of any other soft-thinking method.

We have added the multi-seed results to App. E.1 (Tab. 10).

AuthorsOpenReview ↗

Response to Reviewer rd2a (3/3)

Q2. AGC-ST at other temperatures with fixed thresholds

"Entropy and margin are computed on the temperature-scaled distribution (tau=0.6). How do the gate rate and accuracy change at other temperatures with the thresholds fixed?"

Thank you for this question. We selected (α,β)(\alpha,\beta) at TT=0.6, the temperature recommended for all five models. We kept these thresholds fixed and tested TT from 0.4 to 1.0. A higher temperature raises the entropy and shrinks the margin of the scaled distribution. The gate rate therefore rises with temperature. AGC-ST outperforms CoT from TT=0.4 to TT=0.8 and is 0.62 points below CoT at TT=1.0. Computing the gate and the mixture on the TT=0.6 distribution brings AGC-ST above CoT at TT=1.0 as well.

Temperature study with fixed thresholds. We evaluated CoT and AGC-ST with DS-Qwen-7B on AIME2024 at five temperatures, with the AGC-ST thresholds fixed at (α,β)=(1.25,0.10)(\alpha,\beta)=(1.25,0.10). All other settings are the same for both methods.

Table rd2a-3. CoT and AGC-ST at different temperatures with fixed thresholds (DS-Qwen-7B, AIME2024, Avg@16). The CoT row is the same as in W1. Gate rate is the percentage of gated thinking steps. Mean support size is the average number of tokens mixed at gated steps. The last column computes the gate statistics, the support and the mixture weights at TT=0.6 while sampling at TT=1.0.

TT=0.4TT=0.5TT=0.6TT=0.8TT=1.0TT=1.0, decision TT=0.6
CoT52.2955.0054.7954.7954.37—
AGC-ST53.5455.4256.8856.4653.7555.62
AGC-ST gate rate0.30%0.70%1.14%3.31%8.05%2.01%
AGC-ST mean support size4.294.434.895.967.915.06

The gate rate rises from 0.30% at TT=0.4 to 8.05% at TT=1.0, because the gate statistics are computed after temperature scaling. The mean support grows from 4.29 to 7.91 tokens. From TT=0.4 to TT=0.8, AGC-ST is above CoT at every temperature. Over this range, the gate rate changes more than tenfold. The gain thus holds over a wide range of gate rates.

What changes at TT=1.0. With fixed thresholds, AGC-ST is 0.62 points below CoT at TT=1.0. At this temperature the distribution becomes flatter, so the gate admits about seven times as many steps as at TT=0.6. Each gated step also mixes 7.91 candidates on average instead of 4.89. To test this explanation, we kept the decision temperature at 0.6 (last column). The gate statistics, the support and the mixture weights are computed on the TT=0.6 distribution, while non-gated steps still sample at TT=1.0. This lowers the gate rate to 2.01%. AGC-ST reaches 55.62 and outperforms CoT (54.37). Since non-gated steps still sample at TT=1.0 in this column, the result suggests that the accuracy drop comes from making these decisions on the flatter TT=1.0 distribution. With the decision temperature kept at 0.6, soft mixing at TT=1.0 again improves over CoT.

Although the gate depends on the temperature, the default TT=0.6 works well for AGC-ST. At this temperature AGC-ST reaches its highest accuracy in Table rd2a-3 (56.88), above CoT at every temperature tested. The main results in Tab. 1 also use TT=0.6, and AGC-ST achieves the highest accuracy among the evaluated methods in all 25 settings there. We have added the fixed-threshold study to App. E.5 (Tab. 15).


Thank you again for your constructive suggestions. Following them, we ran the gated-step decoding ablation, the multi-seed evaluation, and a temperature study with fixed thresholds. We have incorporated these results into the revised paper, with all changes highlighted in red. We hope they better prove the effectiveness of AGC-ST. We would be grateful if you would consider raising your score. We would be happy to discuss any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful reviews and constructive suggestions. The reviewers noted the following strengths:

  • Comprehensive evaluation with consistent gains (rd2a S2, XXZQ S1). The evaluation covers five models and five benchmarks. AGC-ST achieves the highest accuracy among the evaluated methods in all 25 model–benchmark settings.
  • Convincing held-out gains with statistical support (Qc6c S1, XXZQ S2). Qc6c found the gains on the 19 settings not used for hyperparameter selection convincing. On the five-benchmark mean, the 95% paired bootstrap intervals for the gains over CoT exclude zero for all five models.
  • Insightful analysis (rd2a S1, Qc6c S2). rd2a found the study showing that soft thinking does not always outperform CoT interesting and insightful. Qc6c noted that the same-state interventions clarify mixture behavior.

We conducted new experiments to address each reviewer's comments. All results are in the revised paper, with changes highlighted in red.

Comparison with related methods

  • ThinkRouter (XXZQ W2, Q3). AGC-ST outperforms ThinkRouter in all 25 model–benchmark settings. On the five-benchmark mean, the 95% paired bootstrap intervals for the difference exclude zero for all five models.
  • Soft Thinking with model-specific settings (XXZQ Q4). Tuning top-n or the Cold Stop entropy threshold per model does not change the comparison. On AIME2024, Soft Thinking with the best setting for each model is still below AGC-ST on all five models.

Ablations

  • Gates at approximately matched rates (Qc6c W1(b), XXZQ W1, Q2). The benefit of the joint gate does not come from mixing less often. At approximately matched gate rates, it outperforms entropy-only, margin-only and random gates for both models tested on AIME2024/2025.
  • Separating soft input from sampling randomness (rd2a W1). Neither removing randomness at gated steps nor lowering the CoT temperature reproduces the gain of AGC-ST. With the same gate, decoding the gated steps with argmax or with sampling at TT=0.3 is below AGC-ST for all three models tested on AIME2024/2025. For DS-Qwen-7B on AIME2024, the best CoT over five temperatures is also below AGC-ST at TT=0.6.
  • Final-answer effect of the mixture (XXZQ Q1). In full rollouts, the mixture that changes next-step predictions in the same-state interventions also improves final-answer accuracy over the top-1 input. Replacing it with the top-1 embedding at gated steps lowers final-answer accuracy for all three models tested on AIME2024/2025.
  • Support on held-out benchmarks (Qc6c W1(a)). Calibrated support also outperforms fixed top-4 on benchmarks not used for hyperparameter selection. AGC-ST is higher on GPQA-D and LCB for all three models tested.

Seeds and temperatures

  • Multi-seed evaluation (rd2a W2). The advantage over CoT holds across sampling runs. Over three runs, AGC-ST has higher mean accuracy than CoT for all five models on AIME2024 and GPQA-D.
  • Temperature with fixed thresholds (rd2a Q2). For DS-Qwen-7B on AIME2024, the gain holds over a wide range of gate rates. With fixed thresholds, AGC-ST outperforms CoT from TT=0.4 to TT=0.8. Over this range, the gate rate changes more than tenfold. At TT=1.0, AGC-ST is 0.62 points below CoT with fixed thresholds. Computing the gate and the mixture at TT=0.6 brings it above CoT.

Our response to each reviewer gives the details and the locations in the revised paper.

AuthorsOpenReview ↗

Summary of Rebuttal Discussion

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback and engagement during the discussion. We briefly summarize the reviewer feedback and the new results.

ReviewerInitial ratingUpdated ratingReviewer feedback
rd2a23"Most of my concerns are addressed."
Qc6c44"With a fixed gate, calibration support provides higher accuracy than with a fixed top-4 for all models in GPQA-D and LiveCodeBench. This answers my concerns about the initial AIME comparison."
XXZQ33"…additional experiments, which address most of my empirical concerns. I remain positive about the consistent improvements…"

As detailed in our general response, the revised paper adds the following results:

  • Comparison with related methods: ThinkRouter in all 25 model–benchmark settings, and Soft Thinking with model-specific settings.
  • Ablations: gates at approximately matched rates, separating soft input from sampling randomness, the final-answer effect of the mixture, and support on held-out benchmarks.
  • Seeds and temperatures: multi-seed evaluation and temperature with fixed thresholds.

These results strengthen the support for the two distribution-conditioned decisions in AGC-ST: whether to apply soft mixing and how widely to mix.

We hope this summary and the reviewers' follow-up feedback will be helpful to the Area Chair in the final assessment. We thank the reviewers again for their engagement, and the Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.