acceptodds
Under review as a conference paper at ICLR 2027

Bounded Exponential Reward Shaping for Adaptive Reasoning Efficiency

Abstract

Reasoning models often generate lengthy responses even when additional computation offers limited benefit. Uniform length penalties encourage conciseness but do not account for differences in problem difficulty. We study DASE (Difficulty-Adaptive Shaping for Efficiency), a reinforcement learning approach that uses the current rollout group's solve rate to adapt a capped exponential penalty on reasoning length. The penalty encourages stronger compression on reliably solved prompts and relaxes it on lower-success prompts, without requiring a separate difficulty predictor or an inference-time controller. Experiments with Qwen3.5-122B-A10B across science, mathematics, and coding benchmarks show an approximately 38.4% reduction in average response length relative to an unpenalized reference and the highest mean Overthinking-Adjusted Accuracy area among the compared methods. Ablations examine the influence of the shaping parameters, while trajectory analysis reveals longer reasoning on lower-success groups. These findings support bounded, difficulty-adaptive reward shaping as a practical approach to improving reasoning efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient reasoning, reinforcement learning, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FgyFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies DASE, a reinforcement learning approach for reducing excessive reasoning. DASE combines first-K candidate selection with a capped length penalty whose coefficient depends exponentially on the retained rollout group's solve rate. The intended behavior is to encourage stronger compression on reliably solved prompts while allowing more reasoning on lower-success prompts. Experiments use Qwen3.5-122B-A10B across five science, mathematics, and coding benchmarks. Relative to an unpenalized training reference, the selected configuration reduces average response length by 37.9%, while average accuracy decreases from 87.12% to 85.35%. It also achieves a higher mean Overthinking-Adjusted Accuracy area than GFPO and the reported DLER configuration.

Critical strengths

The paper demonstrates a substantial reduction in generated tokens on a large reasoning model, with evaluation covering several task domains. Within the reported configurations, DASE achieves both shorter average responses and higher average accuracy than the two length-control baselines. These results suggest potential practical value, although the comparison and evaluation limitations below make the extent of the improvement uncertain.

Critical weaknesses
  1. The methodological contribution beyond closely related work is limited and insufficiently established.
  • The central motivation and mechanism substantially overlap with existing approaches. ALP [1] already uses online rollout success rates to adapt length penalties, applying stronger compression to easier prompts. LASER-D [2] likewise performs difficulty-aware length reward shaping based on rollout correctness. The manuscript also explicitly attributes the capped exponential penalty to prior work in Sections 2.3 and 3.2. The remaining contribution therefore appears primarily to be the combination of these ideas with first-K selection and particular optimization settings.

  • The paper does not establish why this combination constitutes a substantive advance. In particular, it lacks controlled comparisons against a fixed penalty coefficient, a coefficient that varies linearly with solve rate, an uncapped penalty, and the same training procedure without first-K selection. Table 4 changes both penalty parameters simultaneously, demonstrating different operating points rather than isolating the contribution of individual design choices. Moreover, Section 4.5 reports that only approximately 0.005%–0.023% of training trajectories reach the penalty cap, leaving the practical importance of the “bounded” component unclear. Without these comparisons, the observed gains cannot be attributed specifically to the proposed design.

[1] Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning
[2] Learn to Reason Efficiently with Adaptive Length-based Reward Shaping

  1. The experimental coverage and evaluation protocol are insufficient to establish generality or reliable superiority over existing methods.
  • All experiments use a single model, Qwen3.5-122B-A10B, with one training run per configuration. Evaluation across several tasks is useful, but it does not establish whether the method transfers across model families, model sizes, or different initial reasoning capabilities.

  • The baseline comparison has further limitations. The closest difficulty-aware methods, particularly ALP and LASER-D, are not evaluated. In addition, for DLER, the explicit reward-zeroing threshold (81,919 tokens) exceeds the training response limit (65,536 tokens), so in effect the truncation penalty is applied only through the generation limit at a single, untuned budget. Together with differences in sampling, filtering, and advantage normalization, this prevents the experiments from isolating the effect of DASE's reward design.

  • Finally, Appendix D selects the penalty parameters using accuracy on the same five benchmarks at checkpoints 100 and 110, then reports checkpoint-100 results on those benchmarks. Although this overlap is disclosed, it creates a risk of optimistic selection bias and does not provide an independent test of the selected configuration. The 0.64-percentage-point average accuracy improvement over GFPO is not accompanied by uncertainty estimates that account for repeated responses to the same questions or variability across training runs. The lower accuracies on both additional HMMT 2025 benchmarks, which were excluded from parameter selection, also indicate that the main-table advantage is not consistent across the available evaluations. Together, these limitations make the evidence insufficient to support a robust comparative advantage.

Questions for authors
  • Clarification of novelty: What specific methodological advance distinguishes DASE from ALP, LASER-D, and the previously introduced capped exponential penalty? Which reported result demonstrates the benefit of this distinction, rather than the benefit of length-aware training more generally?

  • Baseline validity: Since DLER's truncation effectively occurs at the 65,536-token generation limit, how was this budget chosen? Are results available for DLER with other budgets, and for the closest difficulty-aware baselines under comparable training settings?

  • Evidence for the proposed components: Are there controlled ablations that isolate the exponential mapping, the penalty cap, and first-K selection while holding filtering and optimization settings fixed? Such evidence would materially affect my assessment of whether the combination provides a meaningful methodological improvement.

  • Generalization and evaluation robustness: Are there results on additional base models, independent evaluation benchmarks not used for parameter selection, or repeated training runs? Please distinguish any existing evidence from proposed future experiments, and clarify how uncertainty from repeated responses to the same questions is handled.

7 comments
AuthorsOpenReview ↗

Response to Reviewer FgyF

Summary of the Reviewer’s Concerns

Thank you for carefully reviewing the paper and for your specific and helpful suggestions on how the method differs from prior work, the ablation design, and the reliability of the evaluation.

Your concerns focus on four aspects: the specific distinction between DASE and prior methods and the evidence for it; whether the DLER generation budget and reward threshold are well chosen and whether valid controls have been added; whether controlled ablations isolate the mapping, cap, and first-K selection; and whether the evidence from a second model, independent benchmarks, and repeated runs is sufficient, including how statistical dependence among repeated responses to the same question is handled. We address these four aspects below.

Q1. Clarification of novelty: What specific methodological advance distinguishes DASE from ALP, LASER-D, and the previously introduced capped exponential penalty? Which reported result demonstrates the benefit of this distinction, rather than the benefit of length-aware training more generally?

DASE combines an exponential mapping from the retained group’s solve rate to a shared per-token coefficient with a bounded reward deduction and group-centered updates without division by the within-group reward standard deviation. When the penalty is below the cap, the per-token coefficient and its corresponding group-centered length signal are:

\qquad a_{\mathrm{length}}^{(k)}=-w(p)\bigl(\ell^{(k)}-\overline\ell\bigr).$$ This explicitly shows how the observed solve rate scales the relative-length signal. The controlled linear alternative is: $$w_{\mathrm{linear}}(p)=\frac{2\alpha}{L_{\mathrm{ref}}} \exp\!\left(-\frac{0.5}{\beta}\right)p,$$ This linear mapping matches the exponential coefficient at p = 0.5. The DASE, fixed-coefficient, and linear-mapping configurations in Table S1 share initialization, training data, actor settings, the length definition, cap, filtering, and 16/16 sampling, differing in how the coefficient depends on the observed solve rate. The table reports the mean ± sample standard deviation across 3 independent training runs with different seeds: ### Table S1. Additional Controls on Qwen3.5-122B-A10B at Checkpoint 50 | Method | Training runs | Accuracy (%) | Response tokens (k) | OAA × 100 | | --- | ---: | ---: | ---: | ---: | | DASE, 16/16 | 3 | 86.61 ± 0.243 | 26.61 ± 0.217 | **33.22 ± 0.286** | | Uncapped, 16/16 | 3 | 85.40 ± 0.331 | 28.97 ± 0.341 | 30.59 ± 0.312 | | Fixed-coefficient DASE, 16/16 | 3 | 86.85 ± 0.562 | 30.68 ± 0.274 | 30.02 ± 0.245 | | Linear mapping, 16/16 | 3 | 85.34 ± 0.459 | 34.94 ± 0.186 | 21.50 ± 0.028 | | ALP, published coefficient | 3 | 85.58 ± 0.084 | 41.64 ± 0.068 | 15.51 ± 0.138 | | ALP, scale-matched coefficient | 3 | 85.97 ± 0.245 | 34.05 ± 0.482 | 25.16 ± 0.187 | | LASER-D | 3 | 85.08 ± 0.269 | 28.33 ± 0.250 | 30.36 ± 0.369 | | GFPO | 3 | 84.33 ± 0.333 | 26.91 ± 0.196 | 30.54 ± 0.224 | | DLER, reachable 32,768-token threshold | 3 | 79.45 ± 0.243 | 23.63 ± 0.307 | 28.11 ± 0.291 | | DASE, first 16 of 24 candidates to return | 3 | 87.22 ± 0.485 | 46.76 ± 0.413 | 16.89 ± 0.172 | | Fixed coefficient, first 16 of 24 candidates to return | 3 | 86.84 ± 0.633 | 47.67 ± 0.386 | 14.20 ± 0.154 |

Relative to the specified linear mapping, DASE improves accuracy by 1.27 percentage points, reduces response length from 34.94k to 26.61k, a 23.8% reduction, and increases OAA from 21.50 to 33.22, a gain of 11.72 points. Relative to the fixed coefficient, responses are 13.3% shorter and OAA is 3.20 points higher, with accuracy 0.24 percentage points lower. Both controlled comparisons show that DASE’s mapping improves correct-response efficiency while maintaining high accuracy.

Minerva was excluded from the current round of parameter selection. Before evaluation, we audited the inputs and answers of the original 272 questions under a fixed protocol and checked for exact and approximate textual overlap with the current RL data mixture and the files used for parameter selection and result reporting, retaining 271 questions. Evaluation directly uses the final checkpoints from 3 independent training runs with the established parameters and decoding settings. Each evaluation generates 4 responses per question, totaling 1,084 responses, with statistics organized around the 271 original questions. Results are as follows:

Table S3. Additional Minerva Evaluation Without Parameter Reselection

ModelMethodAccuracy (%)Response tokensOAA × 100
Qwen3.5-122B-A10BDASE44.27 ± 0.5878,646 ± 10123.11 ± 0.137
Fixed coefficient42.85 ± 0.32612,535 ± 90016.22 ± 0.581
Linear mapping43.17 ± 0.13011,476 ± 6416.48 ± 0.170
ALP, scale-matched41.61 ± 1.04410,688 ± 2216.43 ± 0.365
LASER-D43.27 ± 0.9139,648 ± 10918.71 ± 0.899
Qwen3-14BDASE44.05 ± 0.1964,183 ± 822.17 ± 0.011
Unpenalized43.45 ± 0.3915,702 ± 3917.90 ± 0.180
Fixed coefficient44.19 ± 0.6525,390 ± 1018.85 ± 0.299
GFPO44.23 ± 0.5874,318 ± 4321.81 ± 0.241
DLER, optimized configuration43.59 ± 0.5875,413 ± 4618.48 ± 0.177
Linear mapping43.82 ± 0.0006,045 ± 4317.12 ± 0.151
ALP, scale-matched43.77 ± 0.0655,437 ± 1518.26 ± 0.002
LASER-D43.68 ± 0.0655,293 ± 7518.60 ± 0.017

Accuracy, response length, and OAA are reported as the mean ± sample standard deviation across 3 independent training runs with different seeds. Accuracy standard deviations are in percentage points; response-length means and standard deviations are in tokens; OAA standard deviations use the OAA × 100 scale.

On this additional benchmark with Qwen3.5-122B-A10B, DASE improves accuracy by 1.10 percentage points, shortens responses by 24.7%, and increases OAA by 6.63 points relative to the linear mapping. Relative to the fixed coefficient, the corresponding improvements are 1.42 percentage points, 31.0%, and 6.89 points. These results show that the combination of the group-conditioned mapping and group-centered updates also achieves higher accuracy and OAA with shorter responses on Minerva, which was excluded from parameter selection.

Q2. Baseline validity: Since DLER’s truncation effectively occurs at the 65,536-token generation limit, how was this budget chosen? Are results available for DLER with other budgets, and for the closest difficulty-aware baselines under comparable training settings?

The original 65,536-token generation limit came from an existing fixed-budget DLER configuration, while its reward-zeroing threshold was 81,919 tokens, so training responses always remained below that threshold. The additional Qwen3.5-122B-A10B configuration uses a 131,072-token training generation limit and a 32,768-token reward-zeroing threshold, with nonzero truncation-rate logs during training confirming activation. After generation returns and before group filtering, task rewards are zeroed for responses strictly above the threshold and retained for responses exactly at it. Length excludes prompt and masked tokens, and the original correctness labels remain unchanged. Final evaluation uses a common generation limit across methods.

Relative to DLER with a reachable threshold, DASE improves accuracy by 7.16 percentage points and OAA by 5.11 points, with longer responses (26.61k vs. 23.63k). Relative to ALP with its published coefficient, DASE improves accuracy by 1.03 percentage points, shortens responses by 36.1%, and increases OAA by 17.71 points. Relative to scale-matched ALP, the corresponding improvements are 0.64 percentage points, 21.9%, and 8.06 points; relative to LASER-D, they are 1.53 percentage points, 6.1%, and 2.86 points; and relative to GFPO, they are 2.28 percentage points, 1.1%, and 2.68 points. DASE achieves higher OAA in all these comparisons, reflecting higher correct-response efficiency (see Table S1 in Q1).

ALP charges a cost for all response tokens; DASE uses reasoning length and a cap. LASER-D retains the dynamic, difficulty-specific reward from its original configuration. Its monitoring mechanism samples 8 responses for each of 500 prompts and refreshes at updates 0, 20, and 40. The deduplicated additional generation totals are 644,705,227 tokens for Qwen3.5-122B-A10B and 224,159,236 tokens for the smaller model, with corresponding refresh wall times of 5,608.5 and 6,706.2 seconds.

Table S2. Qwen3-14B Results at Checkpoint 50

Each configuration reports the mean ± sample standard deviation across 3 independent training runs with different seeds. Accuracy standard deviations are in percentage points; response-length means and standard deviations are in thousands of tokens; OAA standard deviations use the OAA × 100 scale.

MethodAccuracy (%)Response tokens (k)OAA × 100
DASE61.46 ± 0.00212.62 ± 0.19922.53 ± 0.281
Unpenalized60.90 ± 0.45113.32 ± 0.01220.49 ± 0.005
Fixed coefficient62.21 ± 0.19612.82 ± 0.08322.10 ± 0.156
GFPO58.54 ± 0.13612.03 ± 0.02621.29 ± 0.011
DLER57.69 ± 0.36311.57 ± 0.04121.61 ± 0.114
Linear mapping61.41 ± 0.09513.32 ± 0.03819.73 ± 0.273
ALP, scale-matched coefficient61.29 ± 0.72112.78 ± 0.05821.90 ± 0.121
LASER-D60.85 ± 0.14212.52 ± 0.01222.05 ± 0.071

Relative to Unpenalized, DASE increases accuracy from 60.90% to 61.46%, a gain of 0.56 percentage points; reduces response length from 13.32k to 12.62k, a 5.3% reduction; and improves OAA from 20.49 to 22.53, a gain of 2.04 points.

The smaller-model DLER configuration uses a 16,384-token reward-zeroing threshold and 16 prompts per PPO batch; the other smaller-model configurations use a training batch size of 128. DASE achieves 61.46% accuracy, 3.77 percentage points above DLER’s 57.69%, and 0.92 points higher OAA, with response lengths of 12.62k and 11.57k, respectively. DASE achieves higher accuracy and correct-response efficiency with a moderate increase in response length.

Q3. Evidence for the proposed components: Are there controlled ablations that isolate the exponential mapping, the penalty cap, and first-K selection while holding filtering and optimization settings fixed? Such evidence would materially affect my assessment of whether the combination provides a meaningful methodological improvement.

The reward controls under the common 16/16 setting fix initialization, training data, actor settings, and sampling. The candidate-selection controls preserve the matched reward settings while changing sampling to generate 24 candidates and retain the first 16 to return.

Reviewer_vHRpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a reinforcement learning method for reducing reasoning length. It uses the success rate of sampled answers to set a capped length penalty, with stronger penalties on questions with high success rates and weaker penalties on those with low success rates. The penalty strength changes exponentially with success rate, and the cap ensures that correct responses still receive higher rewards than incorrect ones. The method is tested on Qwen across five benchmarks and compared with GFPO and DLER. The method produces much shorter responses than the unpenalized model, with slightly lower average accuracy, but higher average accuracy than GFPO and DLER.

Critical strengths
  1. The method needs no extra samples to estimate the success rate and no separate difficulty model or controller at test time. The success rate is calculated from the responses retained for the policy update, so the method can reuse feedback already available during training.

  2. Across the five main benchmarks, it reduces average response length by 37.9% compared with the unpenalized model. This comes with some loss in average accuracy, but the results suggest that much shorter responses can still retain most of the model's accuracy.

  3. The chosen penalty cap keeps the reward for a correct answer above that for an incorrect answer. This avoids rewarding a wrong answer more highly simply because it is shorter.

Critical weaknesses
  1. The main ideas overlap with existing work, and I do not see other clear new contributions. ALP (https://arxiv.org/abs/2506.05256) already adjusts length penalties using success rate, and the difficulty based version of LASER (https://proceedings.iclr.cc/paper_files/paper/2026/file/47795c4ae2f7d07ea2fb0d11fa2c3c90-Paper-Conference.pdf) also adjusts length rewards based on difficulty. The capped exponential penalty builds on DeepSeek (2026; https://arxiv.org/html/2609.19969v1). Without direct comparisons with ALP or LASER, it is unclear what DASE adds.

  2. DLER's rule for setting the reward to zero never takes effect because its threshold is above the maximum training response length (Appendix B). The reported result therefore does not show an advantage over DLER when that rule is active.

  3. The paper does not compare adaptive and fixed penalty coefficients under the same training setup. This comparison is needed to show whether adjusting the penalty by success rate helps, beyond the other changes to training.

  4. The writing and figures would benefit from further polishing. The introduction could be less repetitive, and the main contributions should be summarized more clearly.

  5. The penalty parameters are chosen using the same benchmarks on which the main results are reported. This makes the main comparison less convincing as evidence that the gains will hold on new problems.

Questions for authors
  1. A high success rate may depend on long reasoning. What evidence shows that stronger length penalties on these questions remove unnecessary reasoning rather than necessary steps?

  2. Does adjusting the penalty based on success rate outperform a fixed penalty under the same training setup?

  3. In Section 4.6, could you report reasoning lengths for correct and incorrect responses separately within each group defined by success rate?

  4. How does DASE compare with ALP and LASER-D when trained from the same model on the same data, with similar training budgets? Can you compare their accuracy at similar response lengths?

  5. Does DASE still outperform DLER when DLER uses a threshold that allows its length penalty to take effect?

5 comments
AuthorsOpenReview ↗

Response to Reviewer vHRp

Summary of the Reviewer’s Concerns

Thank you for closely examining the reasoning-compression mechanism and experimental comparisons, and for your constructive suggestions on the positioning and presentation of the method.

Your main concerns are whether reasoning compression removes necessary steps, the controlled effect of an adaptive penalty relative to a fixed coefficient, and the reasoning lengths of correct and incorrect responses within different solve-rate groups. You also request comparisons with ALP and LASER-D using the same model and data, with comparable budgets or response lengths, and a DLER comparison with an active threshold. You further raise questions about novelty, presentation, and parameter selection. We first address the five experimental questions, then discuss the method’s positioning and revisions to the presentation.

Q1. A high success rate may depend on long reasoning. What evidence shows that stronger length penalties on these questions remove unnecessary reasoning rather than necessary steps?

We assess compression through both overall performance and reasoning excerpts for the same problems. Table S2 shows that, relative to Unpenalized, DASE reduces mean response length by 5.3% while increasing mean accuracy by 0.56 percentage points and OAA by 2.04 points. The mathematical example below further illustrates how a shorter response preserves the key derivation. We also include longer correct responses and failure cases to illustrate behavior on individual problems.

Table S2. Qwen3-14B Results at Checkpoint 50

Each configuration reports the mean ± sample standard deviation across 3 independent training runs with different seeds. Accuracy standard deviations are in percentage points; response-length means and standard deviations are in thousands of tokens; OAA standard deviations use the OAA × 100 scale.

MethodAccuracy (%)Response tokens (k)OAA × 100
DASE61.46 ± 0.00212.62 ± 0.19922.53 ± 0.281
Unpenalized60.90 ± 0.45113.32 ± 0.01220.49 ± 0.005
Fixed coefficient62.21 ± 0.19612.82 ± 0.08322.10 ± 0.156
GFPO58.54 ± 0.13612.03 ± 0.02621.29 ± 0.011
DLER57.69 ± 0.36311.57 ± 0.04121.61 ± 0.114
Linear mapping61.41 ± 0.09513.32 ± 0.03819.73 ± 0.273
ALP, scale-matched coefficient61.29 ± 0.72112.78 ± 0.05821.90 ± 0.121
LASER-D60.85 ± 0.14212.52 ± 0.01222.05 ± 0.071

Relative to Unpenalized, DASE increases accuracy from 60.90% to 61.46%, a gain of 0.56 percentage points; reduces response length from 13.32k to 12.62k, a 5.3% reduction; and improves OAA from 20.49 to 22.53, a gain of 2.04 points.

We collected paired responses to the same task/prompt inputs from the second decoding evaluation of Qwen3-14B’s final checkpoint. Case selection follows a predetermined rule: for each prompt, we take the first response in a fixed order by source file and line number, using only one normalized prompt variant per original question. Among pairs where both responses are correct, we select cases at the 25th, 50th, and 75th percentiles of the DASE/reference response-length ratio. We also select the case with the median length ratio among pairs where the reference is correct and DASE is incorrect. This rule covers different response-length ratios, illustrating both compression and variation in reasoning length.

Table S5. Same-Problem Case Studies

CaseSaved verifier judgmentDASE tokensReference tokensDASE/reference
1: Intersections of a line with y = x² + 2/xBoth correct3,9895,2610.758
2: Organic synthesis problemBoth accepted by the verifier9,72810,7670.904
3: Symmetric square-root identityBoth correct14,28613,3471.070
4: Shortest wildcard-matching substringReference correct; DASE incorrect19,20718,7671.023

Correct mathematical example: Case 1. For the line–curve intersection problem, both responses derive x³ − mx² − bx + 2 = 0 and obtain the third x-coordinate, −1/21. DASE’s final explanation uses the product of the roots:

“6 · 7 · k = −2 ⇒ 42k = −2 ⇒ k = −1/21.”

The full-response breakdown identifies where the tokens are saved:

Response regionUnpenalizedDASETokens saved
Reasoning4,5013,2201,281
Other tokens, including final answer and boundary markers760769−9
Total5,2613,9891,272

Reasoning is 28.5% shorter; the other region is nine tokens longer, giving 1,272 tokens saved overall (24.2%). The reduction occurs within intermediate reasoning, with the correct answer and sufficient final derivation retained. Case 3 complements this example with a longer, correct DASE response.

Shorter-but-incorrect example: GPQA IUPAC naming. Among 32 reference-correct/DASE-incorrect pairs in the same Qwen3-14B case-study sample, 13 have shorter DASE responses. We select their median DASE/reference length ratio (rank 7; ties broken by canonical-question hash), using the same question and option order:

MethodJudgmentReasoning tokensOther tokensTotal tokens
UnpenalizedCorrect: C5,2618166,077
DASEIncorrect: B4,3345084,842

The accepted option C is 3-cyano-6-(dimethylamino)-5-formyl-2-hydroxy-4-methoxybenzoic acid. DASE selects B, 5-cyano-2-(dimethylamino)-3-formyl-6-hydroxy-4-methoxybenzoic acid, reversing the ring-numbering direction. Its trace states:

“But in this case, since the carboxylic acid is fixed at position 1, the numbering is fixed.”

Position 1 fixes the starting point; the numbering direction still needs to be determined. DASE saves 1,235 tokens (20.3%): 927 reasoning + 308 other tokens. This pairs shorter reasoning with a specific naming error.

Original Case 4 illustrates an empty-prefix handling error in a response 440 tokens longer (19,207 vs. 18,767). The GPQA pair directly illustrates shorter-but-incorrect behavior. Table S2 captures the aggregate accuracy–efficiency gains; Q3 examines reasoning lengths by difficulty and correctness.

Q2. Does adjusting the penalty based on success rate outperform a fixed penalty under the same training setup?

The following comparisons share initialization, training data, and actor update settings. The 16/16 setting generates 16 candidates and retains all of them; first-16/24 generates 24 candidates and retains the first 16 to return. The fixed coefficient matches DASE’s per-token cost in the unsaturated regime at a group solve rate of p = 0.5.

Table S1. Additional Controls on Qwen3.5-122B-A10B at Checkpoint 50

MethodTraining runsAccuracy (%)Response tokens (k)OAA × 100
DASE, 16/16386.61 ± 0.24326.61 ± 0.21733.22 ± 0.286
Uncapped, 16/16385.40 ± 0.33128.97 ± 0.34130.59 ± 0.312
Fixed-coefficient DASE, 16/16386.85 ± 0.56230.68 ± 0.27430.02 ± 0.245
Linear mapping, 16/16385.34 ± 0.45934.94 ± 0.18621.50 ± 0.028
ALP, published coefficient385.58 ± 0.08441.64 ± 0.06815.51 ± 0.138
ALP, scale-matched coefficient385.97 ± 0.24534.05 ± 0.48225.16 ± 0.187
LASER-D385.08 ± 0.26928.33 ± 0.25030.36 ± 0.369
GFPO384.33 ± 0.33326.91 ± 0.19630.54 ± 0.224
DLER, reachable 32,768-token threshold379.45 ± 0.24323.63 ± 0.30728.11 ± 0.291
DASE, first 16 of 24 candidates to return387.22 ± 0.48546.76 ± 0.41316.89 ± 0.172
Fixed coefficient, first 16 of 24 candidates to return386.84 ± 0.63347.67 ± 0.38614.20 ± 0.154

Under 16/16 sampling, DASE uses 26.61k response tokens versus 30.68k for the fixed coefficient, a 13.3% reduction. Their OAA values are 33.22 and 30.02, a gain of 3.20 points, and their accuracies are 86.61% and 86.85%, a decrease of 0.24 percentage points. This controlled comparison shows that DASE reduces response length by 13.3% while maintaining similar mean accuracy and achieving higher correct-response efficiency.

Under first-16/24 sampling, DASE improves accuracy by 0.38 percentage points over the matched fixed-coefficient control (87.22% vs. 86.84%), shortens responses by 1.9% (46.76k vs. 47.67k), and improves OAA by 2.69 points (16.89 vs. 14.20). Relative to DASE’s own 16/16 setting, first-16/24 increases response length by 75.7%, decreases OAA by 16.33 points, and improves accuracy by 0.61 percentage points, while also incurring additional candidate-generation costs.

Minerva was excluded from the current round of parameter selection. Before evaluation, we audited the inputs and answers of the original 272 questions under a fixed protocol and checked for exact and approximate textual overlap with the current RL data mixture and the files used for parameter selection and result reporting, retaining 271 questions. Evaluation directly uses the final checkpoints from 3 independent training runs with the established parameters and decoding settings. Each evaluation generates 4 responses per question, totaling 1,084 responses, with statistics organized around the 271 original questions. On Minerva, the OAA budget is calibrated separately for each model and fixed for comparisons among methods on the same model and benchmark. Results are as follows:

Table S3. Additional Minerva Evaluation Without Parameter Reselection

ModelMethodAccuracy (%)Response tokensOAA × 100
Qwen3.5-122B-A10BDASE44.27 ± 0.5878,646 ± 10123.11 ± 0.137
Fixed coefficient42.85 ± 0.32612,535 ± 90016.22 ± 0.581
Linear mapping43.17 ± 0.13011,476 ± 6416.48 ± 0.170
ALP, scale-matched41.61 ± 1.04410,688 ± 2216.43 ± 0.365
LASER-D43.27 ± 0.9139,648 ± 10918.71 ± 0.899
Qwen3-14BDASE44.05 ± 0.1964,183 ± 822.17 ± 0.011
Unpenalized43.45 ± 0.3915,702 ± 3917.90 ± 0.180
Fixed coefficient44.19 ± 0.6525,390 ± 1018.85 ± 0.299
GFPO44.23 ± 0.5874,318 ± 4321.81 ± 0.241
DLER, optimized configuration43.59 ± 0.5875,413 ± 4618.48 ± 0.177
Linear mapping43.82 ± 0.0006,045 ± 4317.12 ± 0.151
ALP, scale-matched43.77 ± 0.0655,437 ± 1518.26 ± 0.002
LASER-D43.68 ± 0.0655,293 ± 7518.60 ± 0.017

Accuracy, response length, and OAA are reported as the mean ± sample standard deviation across 3 independent training runs with different seeds. Accuracy standard deviations are in percentage points; response-length means and standard deviations are in tokens; OAA standard deviations use the OAA × 100 scale.

On Qwen3.5-122B-A10B, DASE achieves 44.27% accuracy, 8,646 response tokens, and 23.11 OAA. Relative to the fixed coefficient, accuracy is 1.42 percentage points higher, responses are 31.0% shorter, and OAA is 6.89 points higher. Relative to the linear mapping, the corresponding improvements are 1.10 percentage points, 24.7%, and 6.63 points; relative to scale-matched ALP, they are 2.66 percentage points, 19.1%, and 6.68 points. DASE also achieves higher mean accuracy than LASER-D (44.27% vs. 43.27%), shorter responses (8,646 vs. 9,648), and higher OAA (23.11 vs. 18.71).

Q3. In Section 4.6, could you report reasoning lengths for correct and incorrect responses separately within each group defined by success rate?

We performed this breakdown using 6,144 training trajectories from updates 90, 100, and 110 in the original paper. This table reports reasoning length in tokens, consistent with the length used in the training reward; the other performance tables report total response tokens.

Table S6. Breakdown by Difficulty and Correctness

Group solve-rate rangeCorrect trajectoriesCorrect meanCorrect medianIncorrect trajectoriesIncorrect meanIncorrect median
Easy: p ≥ 0.751,74937,09335,41829942,18138,970
Medium: 0.25 ≤ p < 0.751,30342,38339,7831,54544,86242,354
Hard: p < 0.2514544,40742,9141,10344,43641,855

The mean reasoning length of correct trajectories increases as the group solve rate decreases: from 37,093 tokens in the Easy group to 42,383 in the Medium group and 44,407 in the Hard group. This pattern is consistent with applying weaker length penalties to lower-success groups to preserve more room for reasoning.

Reviewer_BcRaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies excessively long outputs in reasoning models and proposes DASE, a difficulty-adaptive reward shaping method. For each problem, the method generates multiple candidate responses, retains the first K to return, and uses the retained group's original correctness rate to set a length-penalty coefficient that varies exponentially with the solve rate. It then applies a capped reward deduction based on each response's reasoning length for reinforcement learning training. Experiments are conducted on a single model, Qwen3.5-122B-A10B. Compared with the unpenalized reference, DASE reduces mean response length by approximately 37.9%, with a slight decrease in mean accuracy; compared with the evaluated GFPO and DLER configurations, it achieves shorter mean responses, higher mean accuracy, and a higher OAA area.

Critical strengths
  1. DASE uses the current rollout group’s solve rate to adjust the length penalty. It requires neither a separate difficulty predictor nor an inference-time controller. Among the evaluated length-control methods, the selected configuration achieves the shortest mean responses and the highest mean accuracy. Reporting accuracy, response length, and OAA together gives a clearer picture of the accuracy–length trade-off.

  2. The paper clearly explains the reward design and training settings. It distinguishes reasoning length used for the training penalty from total response length used for evaluation. Training curves and parameter analyses provide additional information about the model’s behavior during training. Under the stated assumptions, the capped penalty ensures that correct responses receive higher shaped rewards than incorrect responses.

Critical weaknesses
  1. The DLER comparison does not establish an advantage over an active length penalty. sec 4.2 states that DLER’s reward-zeroing threshold exceeds its training response limit, so the penalty cannot activate. Differences in other training settings also make it difficult to attribute the gains to DASE’s reward design. a valid DLER configuration and controlled comparisons would provide stronger evidence.
  2. The practical value of the exponential coefficient and cap remains unclear. sec4.5 changes alpha and beta together, while only approximately 0.005%–0.023% of trajectories reach the cap. These results do not show whether either design improves the accuracy–length trade-off over simpler alternatives. Separate parameter ablations and comparisons with fixed-coefficient and uncapped penalties would clarify their contributions.
  3. The evaluation does not sufficiently establish how well the gains generalize or how stable they are. Parameters are selected using the same five benchmarks used for reporting, which may favor the selected configuration. Single training runs without uncertainty estimates also leave the small accuracy gains difficult to assess. Evaluation on benchmarks excluded from selection and repeated runs would strengthen the conclusions.
  4. The experiments are limited to only one model( Qwen3.5-122B-A10B). It remains unclear whether the reported accuracy–length trade-off holds across other models and model sizes. Validation on an additional model would strengthen the evidence that DASE’s benefits extend beyond the evaluated model.
Questions for authors
  1. Does DASE retain its advantage when DLER uses a reachable reward-zeroing threshold? Can you provide a comparison that controls other training differences to clarify the effect of DASE’s reward design?
  2. How does DASE compare with a fixed penalty coefficient, a linear solve-rate-dependent coefficient, and an uncapped penalty? Given the low cap activation rate, is there evidence that the cap improves performance or training stability?
  3. Do the gains persist on benchmarks excluded from parameter selection? How stable are the accuracy differences across independent training runs?
  4. Have you evaluated on another model or model size? If so, does the accuracy–length trade-off remain consistent with the results on Qwen3.5-122B-A10B?
  5. It may be helpful to include a small case study comparing key reasoning excerpts from DASE and the unpenalized reference on the same problems. Examples where both methods answer correctly could show what DASE removes or shortens. A case where DASE loses correctness could also illustrate the accuracy–length trade-off.
7 comments

Summary of the Reviewer’s Concerns

Thank you for carefully reviewing our work and for your specific suggestions regarding the fairness of the comparisons, the generalizability of the findings, and the statistical evidence.

Your concerns focus on five aspects: whether the DLER comparison is fair when its threshold is actually reached; whether fixed-coefficient, linear, uncapped, and single-parameter controls support the reward design and the role of the cap, particularly in training stability; whether the findings extend to benchmarks excluded from parameter selection and are supported by adequate uncertainty estimates; whether there is validation on other models or model sizes; and whether same-problem case studies illustrate reasoning compression and cases where correctness is lost. We address these five aspects below.

Q1. Does DASE retain its advantage when DLER uses a reachable reward-zeroing threshold? Can you provide a comparison that controls other training differences to clarify the effect of DASE’s reward design?

Thank you for pointing out the limitation of the original DLER baseline. Its reward-zeroing threshold was 81,919 tokens, above the 65,536-token training response limit, so training responses always remained below that threshold. The additional DLER experiment on Qwen3.5-122B-A10B uses a 131,072-token training generation limit and a 32,768-token reward-zeroing threshold. Nonzero truncation-rate logs during training confirm that this branch is activated.

Specifically, after generation returns and before group filtering, trajectories whose response length is strictly greater than 32,768 tokens have their task reward set to zero; trajectories exactly at the threshold retain their reward. Length excludes prompt and masked tokens, and the original correctness labels remain unchanged. This is a training reward rule; final evaluation uses a common generation limit across methods. The table below reports the mean ± sample standard deviation across 3 independent training runs with different seeds.

Table S1. Additional Controls on Qwen3.5-122B-A10B at Checkpoint 50

MethodTraining runsAccuracy (%)Response tokens (k)OAA × 100
DASE, 16/16386.61 ± 0.24326.61 ± 0.21733.22 ± 0.286
Uncapped, 16/16385.40 ± 0.33128.97 ± 0.34130.59 ± 0.312
Fixed-coefficient DASE, 16/16386.85 ± 0.56230.68 ± 0.27430.02 ± 0.245
Linear mapping, 16/16385.34 ± 0.45934.94 ± 0.18621.50 ± 0.028
ALP, published coefficient385.58 ± 0.08441.64 ± 0.06815.51 ± 0.138
ALP, scale-matched coefficient385.97 ± 0.24534.05 ± 0.48225.16 ± 0.187
LASER-D385.08 ± 0.26928.33 ± 0.25030.36 ± 0.369
GFPO384.33 ± 0.33326.91 ± 0.19630.54 ± 0.224
DLER, reachable 32,768-token threshold379.45 ± 0.24323.63 ± 0.30728.11 ± 0.291

In Table S1, DASE exceeds DLER by 7.16 percentage points in accuracy and 5.11 OAA points. DLER produces 23.63k tokens, 2.98k fewer than DASE’s 26.61k, or 11.2% shorter responses using DASE as the denominator. DASE achieves higher accuracy and correct-response efficiency with a moderate increase in response length. The fixed-coefficient, linear, and uncapped controls share the actor and sampling settings to assess the reward design.

Q2. How does DASE compare with a fixed penalty coefficient, a linear solve-rate-dependent coefficient, and an uncapped penalty? Given the low cap activation rate, is there evidence that the cap improves performance or training stability?

DASE, the fixed coefficient, the linear mapping, and the uncapped variant share the same initialization, training data, actor update settings, and sampling setup of generating 16 candidates and retaining all 16. The fixed and linear coefficients match DASE’s per-token cost in the unsaturated regime at a group solve rate of p = 0.5.

Relative to the fixed coefficient, DASE reduces response length from 30.68k to 26.61k, a 13.3% reduction, and increases OAA from 30.02 to 33.22, a gain of 3.20 points; accuracy decreases from 86.85% to 86.61%, a difference of 0.24 percentage points. Relative to the linear mapping, DASE reduces response length by 23.8%, improves OAA by 11.72 points, and increases accuracy by 1.27 percentage points. These controlled comparisons show that DASE’s adaptive mapping effectively improves correct-response efficiency (see Table S1 in Q1).

Capped DASE achieves 86.61% accuracy, 26.61k response tokens, and 33.22 OAA, compared with 85.40%, 28.97k, and 30.59 for the uncapped variant. Retaining the cap improves accuracy by 1.21 percentage points, shortens responses by 8.1%, and increases OAA by 2.63 points, showing improvements in accuracy, response length, and OAA together (see Table S1 in Q1). Formally, with binary task rewards and a penalty cap δ < 1, the shaped reward of a correct response is at least 1 − δ > 0, higher than that of an incorrect response.

We further provide controlled comparisons that change only one parameter at a time to distinguish the roles of α and β. Aggregate results are in Table S7, and per-benchmark results for the α-only change are in Table S7a.

Table S7. Separate Parameter Ablations

These controls all use 16/16 sampling, are evaluated at the final checkpoint 50, and fix δ = 0.5 and L_ref = 131,072. Each variant changes only α or β relative to the original configuration. Results are the mean ± sample standard deviation across 3 independent training runs with different seeds.

ConfigurationαβTraining runsMean accuracy over five benchmarks (%)Response tokens (k)OAA × 100
Original configuration0.800.30386.61 ± 0.24326.61 ± 0.21733.22 ± 0.286
Change α only1.000.30386.81 ± 0.33628.06 ± 0.03931.80 ± 0.030
Change β only0.800.20387.40 ± 0.05137.96 ± 0.41120.82 ± 0.129

Relative to the original configuration, changing only α increases accuracy by 0.20 percentage points and response length from 26.61k to 28.06k, while decreasing OAA by 1.42 points. Changing only β increases accuracy by 0.79 percentage points and response length to 37.96k, while decreasing OAA by 12.40 points. The original configuration has the highest OAA and shortest responses among the three settings, reflecting a better accuracy–efficiency balance.

Table S7a. Per-Benchmark Results at the Final Checkpoint for the α-Only Change

BenchmarkMean accuracy (%, mean ± SD)Response tokens (mean ± SD)OAA × 100 (mean ± SD)
GPQA86.44 ± 0.03619,052 ± 437.98 ± 0.128
AIME 202592.60 ± 0.14727,856 ± 13428.79 ± 0.347
HMMT February 202682.58 ± 2.14335,171 ± 28229.23 ± 0.190
AIME 202691.25 ± 0.00028,347 ± 58029.72 ± 0.148
LiveCodeBench-v6 Python81.18 ± 0.35429,867 ± 2633.30 ± 0.111
Equal-weight mean over five benchmarks86.81 ± 0.33628,058 ± 3931.80 ± 0.030
HMMT February 2025, additional86.88 ± 0.88433,474 ± 13328.51 ± 0.331
HMMT November 2025, additional88.96 ± 2.06231,792 ± 1,48430.76 ± 0.922

Q3. Do the gains persist on benchmarks excluded from parameter selection? How stable are the accuracy differences across independent training runs?

Minerva was excluded from the current round of parameter selection. Before evaluation, we audited the inputs and answers of the original 272 questions under a fixed protocol and checked for exact and approximate textual overlap with the current RL data mixture and the files used for parameter selection and result reporting, retaining 271 questions. Evaluation directly uses the final checkpoints from 3 independent training runs with the established parameters and decoding settings. Each evaluation generates 4 responses per question, totaling 1,084 responses, with statistics organized around the 271 original questions. Results are as follows:

Table S3. Additional Minerva Evaluation Without Parameter Reselection

ModelMethodAccuracy (%)Response tokensOAA × 100
Qwen3.5-122B-A10BDASE44.27 ± 0.5878,646 ± 10123.11 ± 0.137
Fixed coefficient42.85 ± 0.32612,535 ± 90016.22 ± 0.581
Linear mapping43.17 ± 0.13011,476 ± 6416.48 ± 0.170
ALP, scale-matched41.61 ± 1.04410,688 ± 2216.43 ± 0.365
LASER-D43.27 ± 0.9139,648 ± 10918.71 ± 0.899
Qwen3-14BDASE44.05 ± 0.1964,183 ± 822.17 ± 0.011
Unpenalized43.45 ± 0.3915,702 ± 3917.90 ± 0.180
Fixed coefficient44.19 ± 0.6525,390 ± 1018.85 ± 0.299
GFPO44.23 ± 0.5874,318 ± 4321.81 ± 0.241
DLER, optimized configuration43.59 ± 0.5875,413 ± 4618.48 ± 0.177
Linear mapping43.82 ± 0.0006,045 ± 4317.12 ± 0.151
ALP, scale-matched43.77 ± 0.0655,437 ± 1518.26 ± 0.002
LASER-D43.68 ± 0.0655,293 ± 7518.60 ± 0.017

Accuracy, response length, and OAA are reported as the mean ± sample standard deviation across 3 independent training runs with different seeds. Accuracy standard deviations are in percentage points; response-length means and standard deviations are in tokens; OAA standard deviations use the OAA × 100 scale.

On Qwen3.5-122B-A10B, DASE achieves 44.27% accuracy, 8,646 response tokens, and 23.11 OAA. Relative to the fixed coefficient, accuracy is 1.42 percentage points higher, responses are 31.0% shorter, and OAA is 6.89 points higher. Relative to the linear mapping, the corresponding improvements are 1.10 percentage points, 24.7%, and 6.63 points; relative to scale-matched ALP, they are 2.66 percentage points, 19.1%, and 6.68 points. DASE also achieves higher mean accuracy than LASER-D (44.27% vs. 43.27%), shorter responses (8,646 vs. 9,648), and higher OAA (23.11 vs. 18.71).

We estimate uncertainty using 5,000 paired bootstrap replicates clustered by original question. Within each benchmark, the methods being compared share the same resampled original questions, with all same-question responses, formatting variants, and formatting weights retained together. We then take an equal-weight average across the five benchmarks. This procedure preserves the dependence among repeated responses to the same question. Results are as follows:

Table S4. Uncertainty Estimates Accounting for Repeated Questions

ModelComparison: DASE minus controlMetricMean difference95% confidence interval
Qwen3.5-122B-A10BFixed coefficient, 16/16OAA points+3.20[+2.55, +3.80]
Linear mapping, 16/16OAA points+11.72[+10.76, +12.66]
Linear mapping, 16/16Accuracy, percentage points+1.27[−0.27, +2.89]
ALP, scale-matchedOAA points+8.06[+7.48, +8.65]
Qwen3-14BUnpenalizedOAA points+2.04[+1.65, +2.44]
UnpenalizedAccuracy, percentage points+0.56[−0.51, +1.70]

Table S4 shows that DASE’s OAA advantage remains statistically supported after accounting for dependence among repeated responses to the same question. On Qwen3.5-122B-A10B, DASE improves OAA by 3.20, 11.72, and 8.06 points over the fixed coefficient, linear mapping, and scale-matched ALP, respectively, with corresponding 95% confidence intervals of [+2.55, +3.80], [+10.76, +12.66], and [+7.48, +8.65]. On Qwen3-14B, the OAA gain over Unpenalized is 2.04 points, with a 95% confidence interval of [+1.65, +2.44]. The confidence intervals for all four OAA differences lie entirely above zero, demonstrating a consistent advantage in correct-response efficiency across both models and the listed comparisons.

Together with the main results, DASE reduces mean response length by 23.8% and increases mean accuracy by 1.27 percentage points relative to the linear mapping on Qwen3.5-122B-A10B. On Qwen3-14B, it reduces mean response length by 5.3% and increases mean accuracy by 0.56 percentage points relative to Unpenalized. These results jointly show that DASE compresses responses while maintaining high accuracy and improves correct-response efficiency as measured by OAA.

Q4. Have you evaluated on another model or model size? If so, does the accuracy–length trade-off remain consistent with the results on Qwen3.5-122B-A10B?

We provide complete checkpoint-50 results for eight configurations initialized from the publicly available Qwen3-14B model.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response(1/2)

We thank the reviewers for their careful assessment and constructive suggestions. We have completed additional experiments and analyses addressing the methodological contribution, the roles of individual components, baseline validity, generalization, and evaluation reliability, with point-by-point responses below. Together, the new results support the paper’s central conclusion: DASE effectively compresses responses while maintaining high accuracy, achieving a better accuracy–efficiency balance. DASE attains the highest OAA among the listed methods in the main evaluations on both Qwen3.5-122B-A10B and Qwen3-14B, and this efficiency advantage extends to Minerva, which was excluded from parameter selection. We summarize the additional work and its findings below.

Controlled ablations establish the empirical roles of the reward mapping and cap. To address questions about the methodological contribution and simpler alternatives, we added fixed-coefficient, linear-mapping, and uncapped controls with the same initialization, training data, actor update settings, and 16/16 sampling. We also conducted separate ablations of α and β and compared candidate selection under 16/16 and first-16/24. Relative to the linear mapping, DASE reduces response length by 23.8% while improving accuracy by 1.27 percentage points and OAA by 11.72 points. Relative to the fixed coefficient, it achieves shorter responses and higher OAA with similar mean accuracy. Retaining the cap also improves accuracy, response length, and OAA together; under both candidate-selection settings, the adaptive reward achieves higher efficiency than the corresponding fixed coefficient. These controls distinguish the effects of the mapping, cap, parameters, and candidate selection, supporting the practical benefits of the solve-rate-dependent exponential mapping and bounded reward deduction within group-centered updates (Tables S1, S7, and S7a).

A valid DLER comparison and evaluations against related methods further demonstrate DASE’s advantages. To address the inactive reward-zeroing threshold in the original DLER setup, we added a configuration with a reachable threshold and confirmed activation in the training logs. We also added ALP with both its published and scale-matched coefficients, as well as LASER-D. On Qwen3.5-122B-A10B, DASE achieves higher accuracy, shorter responses, and higher OAA than both ALP configurations and LASER-D. Relative to DLER with an active threshold, DASE improves accuracy by 7.16 percentage points and OAA by 5.11 points. The additional Qwen3-14B comparisons likewise show higher accuracy and OAA than ALP, LASER-D, and DLER. These results directly address concerns about baseline validity and coverage of related methods, highlighting DASE’s favorable balance between accuracy and generation cost (Tables S1 and S2).

AuthorsOpenReview ↗

Overall Response(2/2)

Evaluation on a second model and an independent benchmark supports the transferability of the efficiency gains. We added eight configurations on the publicly available Qwen3-14B model and evaluated both models on Minerva, which was excluded from the current round of parameter selection, directly using the existing final checkpoints and established parameters. In the main Qwen3-14B evaluation, DASE improves accuracy, shortens responses, and increases OAA relative to Unpenalized. On Minerva, DASE achieves the highest OAA and shortest mean responses among the listed methods on both models, as well as the highest mean accuracy on Qwen3.5-122B-A10B. These evaluations validate the efficiency gains on a second model and beyond the benchmarks used for parameter selection, addressing concerns about single-model validation and overlap between parameter selection and result reporting (Tables S2 and S3).

Independent training runs and question-clustered statistical analysis strengthen the reliability of the results. The additional experiments report evaluation means and sample standard deviations across 3 independent training runs with different seeds. Uncertainty analysis further uses 5,000 paired bootstrap replicates clustered by question, retaining repeated responses, formatting variants, and their weights together. All four OAA differences in Table S4 have 95% confidence intervals entirely above zero, covering the fixed-coefficient, linear-mapping, and scale-matched ALP controls on Qwen3.5-122B-A10B and the Unpenalized control on Qwen3-14B. Thus, DASE’s efficiency advantage remains statistically supported after accounting for dependence among repeated responses to the same question (Table S4).

Same-problem case studies and difficulty-group analyses further explain reasoning behavior after compression. We selected paired responses using a predetermined rule to show how shorter correct responses preserve key derivations, alongside longer correct responses and failure cases that illustrate behavior on individual problems. Breaking down training trajectories by group solve rate and correctness shows that correct responses contain longer reasoning in lower-success groups, consistent with DASE’s design of applying weaker length penalties to these groups to preserve more room for reasoning. These analyses complement the aggregate performance results with evidence from specific reasoning steps and training trajectories, addressing questions about what is compressed and how the method adapts to difficulty (Tables S5 and S6).

Common Evaluation Protocol

All policies in the additional experiments are evaluated at the final checkpoint after 50 optimizer updates. Unless otherwise stated, results are reported as the mean ± sample standard deviation across 3 independent training runs with different seeds. Accuracy, response length, and OAA are first aggregated using the sample weights within each benchmark, then averaged with equal weight across GPQA, AIME 2025, HMMT February 2026, AIME 2026, and LiveCodeBench-v6 Python.

Response length counts all output tokens, including the final answer; the training penalty counts only reasoning tokens. OAA is computed by exact integration over the empirical distribution of the saved binary correctness labels and per-response lengths. Methods evaluated on the same model and benchmark share a fixed budget calibrated at initialization; the smaller model is calibrated separately.

In the tables, “k” denotes thousands of tokens. All OAA values in the text use the OAA × 100 scale, with differences reported in points; accuracy differences are reported in percentage points. The relative reduction in response length is calculated as “1 − DASE length / control length.”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.