- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies DASE, a reinforcement learning approach for reducing excessive reasoning. DASE combines first-K candidate selection with a capped length penalty whose coefficient depends exponentially on the retained rollout group's solve rate. The intended behavior is to encourage stronger compression on reliably solved prompts while allowing more reasoning on lower-success prompts. Experiments use Qwen3.5-122B-A10B across five science, mathematics, and coding benchmarks. Relative to an unpenalized training reference, the selected configuration reduces average response length by 37.9%, while average accuracy decreases from 87.12% to 85.35%. It also achieves a higher mean Overthinking-Adjusted Accuracy area than GFPO and the reported DLER configuration.
Critical strengths
The paper demonstrates a substantial reduction in generated tokens on a large reasoning model, with evaluation covering several task domains. Within the reported configurations, DASE achieves both shorter average responses and higher average accuracy than the two length-control baselines. These results suggest potential practical value, although the comparison and evaluation limitations below make the extent of the improvement uncertain.
Critical weaknesses
- The methodological contribution beyond closely related work is limited and insufficiently established.
-
The central motivation and mechanism substantially overlap with existing approaches. ALP [1] already uses online rollout success rates to adapt length penalties, applying stronger compression to easier prompts. LASER-D [2] likewise performs difficulty-aware length reward shaping based on rollout correctness. The manuscript also explicitly attributes the capped exponential penalty to prior work in Sections 2.3 and 3.2. The remaining contribution therefore appears primarily to be the combination of these ideas with first-K selection and particular optimization settings.
-
The paper does not establish why this combination constitutes a substantive advance. In particular, it lacks controlled comparisons against a fixed penalty coefficient, a coefficient that varies linearly with solve rate, an uncapped penalty, and the same training procedure without first-K selection. Table 4 changes both penalty parameters simultaneously, demonstrating different operating points rather than isolating the contribution of individual design choices. Moreover, Section 4.5 reports that only approximately 0.005%–0.023% of training trajectories reach the penalty cap, leaving the practical importance of the “bounded” component unclear. Without these comparisons, the observed gains cannot be attributed specifically to the proposed design.
[1] Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning
[2] Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
- The experimental coverage and evaluation protocol are insufficient to establish generality or reliable superiority over existing methods.
-
All experiments use a single model, Qwen3.5-122B-A10B, with one training run per configuration. Evaluation across several tasks is useful, but it does not establish whether the method transfers across model families, model sizes, or different initial reasoning capabilities.
-
The baseline comparison has further limitations. The closest difficulty-aware methods, particularly ALP and LASER-D, are not evaluated. In addition, for DLER, the explicit reward-zeroing threshold (81,919 tokens) exceeds the training response limit (65,536 tokens), so in effect the truncation penalty is applied only through the generation limit at a single, untuned budget. Together with differences in sampling, filtering, and advantage normalization, this prevents the experiments from isolating the effect of DASE's reward design.
-
Finally, Appendix D selects the penalty parameters using accuracy on the same five benchmarks at checkpoints 100 and 110, then reports checkpoint-100 results on those benchmarks. Although this overlap is disclosed, it creates a risk of optimistic selection bias and does not provide an independent test of the selected configuration. The 0.64-percentage-point average accuracy improvement over GFPO is not accompanied by uncertainty estimates that account for repeated responses to the same questions or variability across training runs. The lower accuracies on both additional HMMT 2025 benchmarks, which were excluded from parameter selection, also indicate that the main-table advantage is not consistent across the available evaluations. Together, these limitations make the evidence insufficient to support a robust comparative advantage.
Questions for authors
-
Clarification of novelty: What specific methodological advance distinguishes DASE from ALP, LASER-D, and the previously introduced capped exponential penalty? Which reported result demonstrates the benefit of this distinction, rather than the benefit of length-aware training more generally?
-
Baseline validity: Since DLER's truncation effectively occurs at the 65,536-token generation limit, how was this budget chosen? Are results available for DLER with other budgets, and for the closest difficulty-aware baselines under comparable training settings?
-
Evidence for the proposed components: Are there controlled ablations that isolate the exponential mapping, the penalty cap, and first-K selection while holding filtering and optimization settings fixed? Such evidence would materially affect my assessment of whether the combination provides a meaningful methodological improvement.
-
Generalization and evaluation robustness: Are there results on additional base models, independent evaluation benchmarks not used for parameter selection, or repeated training runs? Please distinguish any existing evidence from proposed future experiments, and clarify how uncertainty from repeated responses to the same questions is handled.
7 comments
Relative to the specified linear mapping, DASE improves accuracy by 1.27 percentage points, reduces response length from 34.94k to 26.61k, a 23.8% reduction, and increases OAA from 21.50 to 33.22, a gain of 11.72 points. Relative to the fixed coefficient, responses are 13.3% shorter and OAA is 3.20 points higher, with accuracy 0.24 percentage points lower. Both controlled comparisons show that DASE’s mapping improves correct-response efficiency while maintaining high accuracy.
Minerva was excluded from the current round of parameter selection. Before evaluation, we audited the inputs and answers of the original 272 questions under a fixed protocol and checked for exact and approximate textual overlap with the current RL data mixture and the files used for parameter selection and result reporting, retaining 271 questions. Evaluation directly uses the final checkpoints from 3 independent training runs with the established parameters and decoding settings. Each evaluation generates 4 responses per question, totaling 1,084 responses, with statistics organized around the 271 original questions. Results are as follows:
Table S3. Additional Minerva Evaluation Without Parameter Reselection
| Model | Method | Accuracy (%) | Response tokens | OAA × 100 |
|---|---|---|---|---|
| Qwen3.5-122B-A10B | DASE | 44.27 ± 0.587 | 8,646 ± 101 | 23.11 ± 0.137 |
| Fixed coefficient | 42.85 ± 0.326 | 12,535 ± 900 | 16.22 ± 0.581 | |
| Linear mapping | 43.17 ± 0.130 | 11,476 ± 64 | 16.48 ± 0.170 | |
| ALP, scale-matched | 41.61 ± 1.044 | 10,688 ± 22 | 16.43 ± 0.365 | |
| LASER-D | 43.27 ± 0.913 | 9,648 ± 109 | 18.71 ± 0.899 | |
| Qwen3-14B | DASE | 44.05 ± 0.196 | 4,183 ± 8 | 22.17 ± 0.011 |
| Unpenalized | 43.45 ± 0.391 | 5,702 ± 39 | 17.90 ± 0.180 | |
| Fixed coefficient | 44.19 ± 0.652 | 5,390 ± 10 | 18.85 ± 0.299 | |
| GFPO | 44.23 ± 0.587 | 4,318 ± 43 | 21.81 ± 0.241 | |
| DLER, optimized configuration | 43.59 ± 0.587 | 5,413 ± 46 | 18.48 ± 0.177 | |
| Linear mapping | 43.82 ± 0.000 | 6,045 ± 43 | 17.12 ± 0.151 | |
| ALP, scale-matched | 43.77 ± 0.065 | 5,437 ± 15 | 18.26 ± 0.002 | |
| LASER-D | 43.68 ± 0.065 | 5,293 ± 75 | 18.60 ± 0.017 |
Accuracy, response length, and OAA are reported as the mean ± sample standard deviation across 3 independent training runs with different seeds. Accuracy standard deviations are in percentage points; response-length means and standard deviations are in tokens; OAA standard deviations use the OAA × 100 scale.
On this additional benchmark with Qwen3.5-122B-A10B, DASE improves accuracy by 1.10 percentage points, shortens responses by 24.7%, and increases OAA by 6.63 points relative to the linear mapping. Relative to the fixed coefficient, the corresponding improvements are 1.42 percentage points, 31.0%, and 6.89 points. These results show that the combination of the group-conditioned mapping and group-centered updates also achieves higher accuracy and OAA with shorter responses on Minerva, which was excluded from parameter selection.
Q2. Baseline validity: Since DLER’s truncation effectively occurs at the 65,536-token generation limit, how was this budget chosen? Are results available for DLER with other budgets, and for the closest difficulty-aware baselines under comparable training settings?
The original 65,536-token generation limit came from an existing fixed-budget DLER configuration, while its reward-zeroing threshold was 81,919 tokens, so training responses always remained below that threshold. The additional Qwen3.5-122B-A10B configuration uses a 131,072-token training generation limit and a 32,768-token reward-zeroing threshold, with nonzero truncation-rate logs during training confirming activation. After generation returns and before group filtering, task rewards are zeroed for responses strictly above the threshold and retained for responses exactly at it. Length excludes prompt and masked tokens, and the original correctness labels remain unchanged. Final evaluation uses a common generation limit across methods.
Relative to DLER with a reachable threshold, DASE improves accuracy by 7.16 percentage points and OAA by 5.11 points, with longer responses (26.61k vs. 23.63k). Relative to ALP with its published coefficient, DASE improves accuracy by 1.03 percentage points, shortens responses by 36.1%, and increases OAA by 17.71 points. Relative to scale-matched ALP, the corresponding improvements are 0.64 percentage points, 21.9%, and 8.06 points; relative to LASER-D, they are 1.53 percentage points, 6.1%, and 2.86 points; and relative to GFPO, they are 2.28 percentage points, 1.1%, and 2.68 points. DASE achieves higher OAA in all these comparisons, reflecting higher correct-response efficiency (see Table S1 in Q1).
ALP charges a cost for all response tokens; DASE uses reasoning length and a cap. LASER-D retains the dynamic, difficulty-specific reward from its original configuration. Its monitoring mechanism samples 8 responses for each of 500 prompts and refreshes at updates 0, 20, and 40. The deduplicated additional generation totals are 644,705,227 tokens for Qwen3.5-122B-A10B and 224,159,236 tokens for the smaller model, with corresponding refresh wall times of 5,608.5 and 6,706.2 seconds.
Table S2. Qwen3-14B Results at Checkpoint 50
Each configuration reports the mean ± sample standard deviation across 3 independent training runs with different seeds. Accuracy standard deviations are in percentage points; response-length means and standard deviations are in thousands of tokens; OAA standard deviations use the OAA × 100 scale.
| Method | Accuracy (%) | Response tokens (k) | OAA × 100 |
|---|---|---|---|
| DASE | 61.46 ± 0.002 | 12.62 ± 0.199 | 22.53 ± 0.281 |
| Unpenalized | 60.90 ± 0.451 | 13.32 ± 0.012 | 20.49 ± 0.005 |
| Fixed coefficient | 62.21 ± 0.196 | 12.82 ± 0.083 | 22.10 ± 0.156 |
| GFPO | 58.54 ± 0.136 | 12.03 ± 0.026 | 21.29 ± 0.011 |
| DLER | 57.69 ± 0.363 | 11.57 ± 0.041 | 21.61 ± 0.114 |
| Linear mapping | 61.41 ± 0.095 | 13.32 ± 0.038 | 19.73 ± 0.273 |
| ALP, scale-matched coefficient | 61.29 ± 0.721 | 12.78 ± 0.058 | 21.90 ± 0.121 |
| LASER-D | 60.85 ± 0.142 | 12.52 ± 0.012 | 22.05 ± 0.071 |
Relative to Unpenalized, DASE increases accuracy from 60.90% to 61.46%, a gain of 0.56 percentage points; reduces response length from 13.32k to 12.62k, a 5.3% reduction; and improves OAA from 20.49 to 22.53, a gain of 2.04 points.
The smaller-model DLER configuration uses a 16,384-token reward-zeroing threshold and 16 prompts per PPO batch; the other smaller-model configurations use a training batch size of 128. DASE achieves 61.46% accuracy, 3.77 percentage points above DLER’s 57.69%, and 0.92 points higher OAA, with response lengths of 12.62k and 11.57k, respectively. DASE achieves higher accuracy and correct-response efficiency with a moderate increase in response length.
Q3. Evidence for the proposed components: Are there controlled ablations that isolate the exponential mapping, the penalty cap, and first-K selection while holding filtering and optimization settings fixed? Such evidence would materially affect my assessment of whether the combination provides a meaningful methodological improvement.
The reward controls under the common 16/16 setting fix initialization, training data, actor settings, and sampling. The candidate-selection controls preserve the matched reward settings while changing sampling to generate 24 candidates and retain the first 16 to return.
Response to Reviewer FgyF
Summary of the Reviewer’s Concerns
Thank you for carefully reviewing the paper and for your specific and helpful suggestions on how the method differs from prior work, the ablation design, and the reliability of the evaluation.
Your concerns focus on four aspects: the specific distinction between DASE and prior methods and the evidence for it; whether the DLER generation budget and reward threshold are well chosen and whether valid controls have been added; whether controlled ablations isolate the mapping, cap, and first-K selection; and whether the evidence from a second model, independent benchmarks, and repeated runs is sufficient, including how statistical dependence among repeated responses to the same question is handled. We address these four aspects below.
Q1. Clarification of novelty: What specific methodological advance distinguishes DASE from ALP, LASER-D, and the previously introduced capped exponential penalty? Which reported result demonstrates the benefit of this distinction, rather than the benefit of length-aware training more generally?
DASE combines an exponential mapping from the retained group’s solve rate to a shared per-token coefficient with a bounded reward deduction and group-centered updates without division by the within-group reward standard deviation. When the penalty is below the cap, the per-token coefficient and its corresponding group-centered length signal are:
\qquad a_{\mathrm{length}}^{(k)}=-w(p)\bigl(\ell^{(k)}-\overline\ell\bigr).$$ This explicitly shows how the observed solve rate scales the relative-length signal. The controlled linear alternative is: $$w_{\mathrm{linear}}(p)=\frac{2\alpha}{L_{\mathrm{ref}}} \exp\!\left(-\frac{0.5}{\beta}\right)p,$$ This linear mapping matches the exponential coefficient at p = 0.5. The DASE, fixed-coefficient, and linear-mapping configurations in Table S1 share initialization, training data, actor settings, the length definition, cap, filtering, and 16/16 sampling, differing in how the coefficient depends on the observed solve rate. The table reports the mean ± sample standard deviation across 3 independent training runs with different seeds: ### Table S1. Additional Controls on Qwen3.5-122B-A10B at Checkpoint 50 | Method | Training runs | Accuracy (%) | Response tokens (k) | OAA × 100 | | --- | ---: | ---: | ---: | ---: | | DASE, 16/16 | 3 | 86.61 ± 0.243 | 26.61 ± 0.217 | **33.22 ± 0.286** | | Uncapped, 16/16 | 3 | 85.40 ± 0.331 | 28.97 ± 0.341 | 30.59 ± 0.312 | | Fixed-coefficient DASE, 16/16 | 3 | 86.85 ± 0.562 | 30.68 ± 0.274 | 30.02 ± 0.245 | | Linear mapping, 16/16 | 3 | 85.34 ± 0.459 | 34.94 ± 0.186 | 21.50 ± 0.028 | | ALP, published coefficient | 3 | 85.58 ± 0.084 | 41.64 ± 0.068 | 15.51 ± 0.138 | | ALP, scale-matched coefficient | 3 | 85.97 ± 0.245 | 34.05 ± 0.482 | 25.16 ± 0.187 | | LASER-D | 3 | 85.08 ± 0.269 | 28.33 ± 0.250 | 30.36 ± 0.369 | | GFPO | 3 | 84.33 ± 0.333 | 26.91 ± 0.196 | 30.54 ± 0.224 | | DLER, reachable 32,768-token threshold | 3 | 79.45 ± 0.243 | 23.63 ± 0.307 | 28.11 ± 0.291 | | DASE, first 16 of 24 candidates to return | 3 | 87.22 ± 0.485 | 46.76 ± 0.413 | 16.89 ± 0.172 | | Fixed coefficient, first 16 of 24 candidates to return | 3 | 86.84 ± 0.633 | 47.67 ± 0.386 | 14.20 ± 0.154 |