acceptodds
Under review as a conference paper at ICLR 2027

Fewer Tokens, Adaptive Thinking: Dynamic Routing Framework with Soft budgets

Abstract

Chain-of-Thought (CoT) improves large language model reasoning but requires costly token-by-token decoding. Latent reasoning reduces explicit decoding but can be unstable and opaque. Hybrid reasoning combines sampled tokens with continuous context. However, using Hybrid inputs at every step may introduce noise and leave the token-efficiency potential of latent reasoning underused. To address these limitations, we propose DASH, a Dynamic routing framework with Adaptive Soft budgets for Hybrid reasoning. At each step, DASH selects between Token-only and Hybrid candidates using the current reasoning state and predictive uncertainty. This makes continuous-context use state-dependent while retaining token anchoring. An adaptive soft budget guides the reasoning scale without truncating generation. DASH learns routing and budget allocation through outcome-based reinforcement learning (RL). A three-stage curriculum establishes reliable routing, rewards shorter correct trajectories, and adapts the soft budget. No CoT annotations or intermediate labels are required. This avoids costly supervision and lets DASH discover reasoning paths that better balance correctness and length efficiency. Across five mathematical benchmarks and three model scales, DASH achieves higher accuracy with shorter trajectories than supervised fine-tuning with CoT (SFT-CoT) and outperforms representative latent-reasoning baselines. Compared with Hybrid Reasoning Policy Optimization (HRPO), it produces substantially shorter trajectories at an accuracy cost, achieving higher accuracy gain per reasoning position on four arithmetic benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent reasoning, hybrid reasoning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_USfiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds on HRPO, which feeds both the sampled token embedding and a projection of the hidden state into the next reasoning step. DASH introduces a learned router that selects between token-only and hybrid inputs at each reasoning step, and a predictor that assigns a soft per-question budget. The budget is embedded into the reasoning inputs and used in the reward, but never truncates generation. Training uses questions and final-answer feedback, without CoT annotations, and proceeds through three RL stages: correctness, length bonuses for correct traces, and budget-aware optimization with predictor training. Qwen3-1.7B/4B/8B are trained on GSM8K and evaluated on GSM8K, GSM-Hard, SVAMP, MultiArith and MATH500. The authors report higher accuracy and shorter reasoning than SFT-CoT, and much shorter reasoning than HRPO at an accuracy cost. Validation includes end-to-end latency, routing and training-stage ablations, and fixed-versus-adaptive budget comparisons.

Critical strengths
  1. The full training framework gives a useful accuracy–length trade-off. On Qwen3-4B, the four arithmetic benchmarks move from 72.96% accuracy with 99.00 reasoning tokens (SFT-CoT) to 78.19% with 36.72 tokens (DASH), and the reported latency includes the additional modules. The fixed B=96 variant in Appendix E.4 is particularly promising: 82.88% accuracy with 57.37 tokens, compared with 83.09% and 178.82 tokens for HRPO. This provides evidence that the budget-aware training framework can retain nearly all of HRPO's arithmetic accuracy at about a third of the reasoning length.

  2. The supplementary evaluations strengthen the empirical evidence: fixed-routing and fixed-budget variants, stage removal, a question-level paired analysis, and routing comparisons across three training seeds. The authors also acknowledge important limitations, including that adaptive budgets do not establish a uniformly better frontier and that the straight-through routing gradient is biased. These controls and qualifications make the reported trade-offs easier to assess.

Critical weaknesses
  1. The incremental benefits of the dynamic and adaptive mechanisms are not sufficiently compelling to establish them as substantial advances over HRPO. In Table 3, Fixed Hybrid retains the full curriculum and budget optimization but achieves 72.90% average accuracy, only 0.44 percentage points below DASH's 73.34%; Fixed Token-only reaches 72.71%. DASH also uses longer traces than both controls (50.19 versus 49.33 and 48.22 tokens). Appendix D.10 supports the repeatability of the small gains, but does not establish an improved trade-off at matched reasoning length or latency. Appendix F shows probability modulation around a mean Hybrid probability of about 0.6, but there is no state-independent stochastic mixture with a matched Hybrid rate to establish the value of conditioning the route on the reasoning state.

    The adaptive-budget results raise a related concern about the value of input-dependent allocation. In Tables 16–17, fixed B=64 achieves 74.74% accuracy with 57.75 tokens, close to adaptive Bmax=96 at 74.53% with 55.20 tokens. This does not establish that fixed budgets dominate, and I acknowledge the favorable MATH500 result for adaptive Bmax=96. Nevertheless, the reported evidence mainly demonstrates alternative compression levels, rather than a clear and repeatable advantage from pairing budgets with individual questions. A uniformly superior frontier is not required, but a meaningful benefit at comparable cost is important to substantiate the paper's emphasis on dynamic routing and adaptive allocation.

  2. The overall efficiency gains have not been evaluated against a competitive standard token-only RL baseline with a designed length penalty. The comparison with the original HRPO changes representation selection, rewards, budget conditioning and the curriculum simultaneously. Separate ablations retain much of the performance when routing or learned budget prediction is individually removed, but there is no simple token-only GRPO baseline whose length-penalty strength is tuned to trace out an accuracy–length curve. Fixed Token-only and Fixed Hybrid both retain the learned budget predictor and the full curriculum, so neither tests whether a simpler length-aware reward is sufficient. The supervision-based latent baselines also do not address this question. Without the comparison, it remains unclear how much of the practical improvement requires the proposed mechanisms rather than ordinary length-aware RL. This limits the significance of the contribution beyond the combined training framework.

Questions for authors
  1. Routing: What evidence shows that dynamic routing improves over Fixed Hybrid or Fixed Token-only at matched reasoning length or latency? Would a state-independent stochastic router with the same Hybrid usage rate perform similarly? Latency measurements for the fixed-routing controls would help establish whether the small accuracy gains justify the additional mechanism.

  2. Budget allocation: At comparable realized reasoning cost, what advantage does the learned budget predictor provide over tuned fixed budgets, and how stable is it across training seeds? Would shuffling predicted budgets across questions while preserving their distribution materially reduce performance?

  3. Assessment-relevant comparison: How does DASH compare with token-only GRPO trained with a tuned length penalty, using the same backbone initialization and data with comparable training resources? Please report an accuracy–length curve. Applying the same penalty to HRPO would further clarify the gains beyond reward design.

  4. Evaluation clarification: Could the authors report unadapted Qwen3 results under the same prompt, decoding and answer format, and separate formatting failures from answer errors? The similar SFT-w/o CoT GSM8K scores at 4B and 8B (27.07% and 27.35%) warrant clarification, although they do not establish a flawed baseline. Please also clarify how Bmax=64 was selected and whether test-set results informed that choice.

1 comment

Thank you for your thoughtful review. We have addressed your concerns in our rebuttal and would appreciate any further feedback or clarification you may have. As the discussion period is nearing its end, we would be grateful if you could take a look at our responses when time permits.

Reviewer_Q3FiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DASH, a dynamic routing framework with adaptive soft budgets for efficient mathematical reasoning. DASH dynamically selects between Token-only and Hybrid representations during reasoning and introduces an adaptive budget predictor to guide reasoning length without explicitly truncating generation.

The framework adopts a three-stage reinforcement learning procedure: Stage 1 focuses on correctness, Stage 2 introduces length-based reward shaping, and Stage 3 jointly optimizes the reasoning policy and adaptive budget predictor.

Experiments are conducted on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across five mathematical reasoning benchmarks. DASH achieves higher accuracy than several supervised and latent-reasoning baselines while generating substantially shorter reasoning trajectories. Compared with HRPO, DASH significantly reduces reasoning length and inference latency, but at the cost of lower reasoning accuracy.

Critical strengths

S1. Interesting approach to improving reasoning efficiency.

The paper explores dynamic routing between Token-only and Hybrid representations, combined with adaptive reasoning budgets. This provides an interesting approach to balancing reasoning accuracy and computational efficiency.

S2. Extensive comparisons with existing methods.

The experiments compare DASH against multiple baselines, including SFT-CoT, HRPO, iCoT, Coconut, CoLaR, and RoT, across three model scales. The paper also includes ablation studies on routing strategies, training stages, and budget configurations.

S3. Significant reduction in reasoning length and inference latency.

DASH substantially reduces reasoning length compared with HRPO and provides end-to-end latency measurements. These results demonstrate the potential efficiency benefits of the proposed framework, although they come with noticeable accuracy degradation.

Critical weaknesses

W1. Significant accuracy degradation compared with the strongest baseline.

My primary concern is that DASH suffers from substantial accuracy degradation compared with HRPO, which is the strongest and most relevant baseline.

According to Table 1, DASH underperforms HRPO by approximately 4.90–8.80 percentage points in average Pass@1 across the three model scales. The degradation becomes even more pronounced on MATH500 (Table 2), where the accuracy gaps reach 17.36, 9.88, and 14.76 percentage points for Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, respectively.

Although DASH achieves substantial reductions in reasoning tokens and inference latency, I am not convinced that the resulting accuracy–efficiency trade-off is sufficiently compelling. In particular, sacrificing approximately 10–17 percentage points of accuracy on MATH500 is a considerable cost.

The paper reports inference latency comparisons in Figure 5, which demonstrate meaningful speedups. However, these improvements do not necessarily justify such substantial performance degradation, especially for reasoning tasks where correctness is critical.

Overall, I believe the paper should provide stronger justification for the practical value of this accuracy–efficiency trade-off.

W2. Insufficient evaluation on challenging mathematical reasoning benchmarks.

The evaluation benchmarks are relatively simple for the Qwen3 model family.

Four of the five benchmarks (GSM8K, GSM-Hard, SVAMP, and MultiArith) primarily evaluate arithmetic and mathematical word-problem solving. Some of these benchmarks are already close to saturation. Although MATH500 provides a more challenging evaluation, it is insufficient to establish the effectiveness of DASH on complex mathematical reasoning tasks.

I believe that benchmarks such as AIME 2024, AIME 2025, and AIME 2026 would be more appropriate for evaluating the reasoning capabilities of Qwen3-1.7B, Qwen3-4B, and Qwen3-8B.

This is particularly important because the performance degradation relative to HRPO becomes more pronounced on MATH500. It remains unclear whether DASH can maintain acceptable reasoning accuracy on substantially more challenging problems.

W3. Limited maximum generation length and insufficient analysis of truncation.

The maximum completion length is set to only 1,024 tokens.

This relatively short generation limit may constrain the reasoning capabilities of the evaluated models, particularly on challenging mathematical problems requiring longer reasoning trajectories.

The paper does not provide sufficient analysis of how frequently generated trajectories reach this limit or how the maximum generation length affects the performance of different baselines, including SFT-CoT and HRPO.

I believe the authors should report truncation statistics and investigate whether the conclusions remain consistent under larger generation limits.

W4. Insufficient evidence for the necessity and effectiveness of adaptive budgets.

Figure 3 raises concerns about whether the proposed adaptive budget mechanism provides meaningful advantages over simpler fixed-budget approaches.

In particular, a fixed budget of 96 achieves higher accuracy than an adaptive budget with a maximum of 256, although the former produces somewhat longer reasoning trajectories.

According to Appendix E.4, the fixed-budget configuration with B = 96 achieves 77.09% average Pass@1, compared with 75.37% for the adaptive configuration with B_max = 256.

These results raise questions about the necessity of learning an input-dependent budget predictor. While adaptive budgets can encourage shorter reasoning, the experiments do not convincingly establish that they provide a better accuracy–efficiency trade-off than appropriately selected fixed budgets.

Furthermore, Figure 6 shows substantial discrepancies between the predicted budgets and actual reasoning lengths. For example, on MATH500, the mean predicted budget is approximately 59.9 tokens, while the mean reasoning length reaches 104.1 tokens.

Although the budget is designed to be soft rather than a strict generation limit, these observations make it difficult to assess how effectively the predictor controls reasoning length.

W5. Potential issues with Stage 2 reward design and insufficient justification for the three-stage training procedure.

Table 3 raises serious concerns about the effectiveness of Stage 2.

Removing Stage 2 results in only a marginal performance decrease (73.34% to 72.59% average Pass@1). In contrast, removing Stage 3 and extending Stage 2 training results in a substantial accuracy drop to 58.07%.

This suggests that prolonged Stage 2 optimization may severely degrade reasoning accuracy.

More importantly, Equation (8) introduces a potential issue with the length-based reward:

Ri(2)=Ci[1+ηLexp⁡(−Li/κL)R_i^{(2)} = C_i [1 + η_L \exp(-L_i / κ_L).

When all rollouts within a group are correct, shorter trajectories receive higher rewards than longer trajectories. After group-relative advantage normalization in Equation (5), longer correct trajectories may receive negative advantages and consequently be penalized despite producing correct answers.

This mechanism may encourage excessive reasoning compression and suppress useful reasoning behaviors. The substantial performance degradation observed when Stage 2 is extended is consistent with this concern, although the ablation does not isolate this mechanism from other changes.

The paper does not sufficiently investigate the consequences of this reward design or explain whether all-correct groups are masked during training.

Additionally, the three-stage training schedule is not adequately justified. Stage 1 and Stage 2 each receive only one epoch, whereas Stage 3 receives ten epochs. Given that Stage 3 accounts for the vast majority of the training budget, it is unclear whether the proposed three-stage curriculum is necessary.

W6. Missing training dynamics and several presentation issues.

The paper does not provide training curves showing how reward, accuracy, and reasoning length evolve throughout the three training stages.

Such curves are particularly important for understanding whether Stage 2 introduces excessive compression pressure and how Stage 3 subsequently affects reasoning accuracy and length.

Several methodological details also require clarification:

  1. Equation (3): The hidden states used by the Budget Predictor are not sufficiently specified. The authors should explicitly clarify whether these correspond to the final Transformer layer.
  2. Equation (5): The group-relative advantage formulation should be more explicitly connected to the original GRPO formulation, with an appropriate citation near the equation.
  3. Equation (6): The aggregation of the policy-gradient loss is not sufficiently clear. In particular, it is unclear whether the loss uses token-level averaging across all generated tokens or sequence-level averaging followed by averaging across trajectories.

These details are important for understanding and reproducing the proposed method.

Questions for authors

see weakness

1 comment

Thank you for your thoughtful review. We have addressed your concerns in our rebuttal and would appreciate any further feedback or clarification you may have. As the discussion period is nearing its end, we would be grateful if you could take a look at our responses when time permits.

Reviewer_zjepOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose DASH, an RL framework for length control in hybrid reasoning. It builds on HRPO with a router to choose at each reasoning step between either the embedding of the sampled token or a hybrid representation which incorporates hidden state as well as an expected top-k embedding. A predictor, assigns a soft budget conditioned on the prompt, with reward shaping for shorter trajectories as well as learning budget-conditioned generation. Experiments with the Qwen3 model family show improved performance compared to SFT on verbal CoT traces and several latent reasoning methods, while relative to HRPO, accuracy is traded off for clearly shorter generation length (token-efficiency).

Critical strengths
  1. Overall, it introduces a useful recipe for inducing shorter reasoning while achieving clear performance gains over popular latent reasoning methods like Coconut and iCoT. On select benchmarks, the performance is competitive (MultiArith for Qwen3-4/8B) with HRPO while using 3-4x fewer tokens.

  2. Thorough experimentation in the appendix is appreciated, such as the Fixed Token-only and Fixed Hybrid ablations, which separate the role of the training recipe from the router and help to better motivate the elements of the core technical contribution.

Critical weaknesses
  1. If the method were as performant as HRPO overall, it would be quite clear that the token efficiency is worth switching to DASH, but the performance hit induces a tradeoff. It would be helpful if this tradeoff could be discussed a bit more. Also, evaluating an experiment of RL training on top of the SFT-CoT (or directly doing RL on the base Qwen3 model) seems like it might be a more fair comparison.

  2. The role of routing in performance seems to be pretty limited, as most benchmarks are within 1 point of one another for DASH, Fixed Token-only, and Fixed Hybrid (at least based on Table 15). It seems plausible then that a fixed-route model that is allowed to reason for slightly longer could reduce (or eliminate) the accuracy-cost tradeoff here.

Questions for authors
  1. Could you discuss how other purely token-space recipes (e.g. Abstract-CoT, Pause tokens) relate to DASH and the Token-only variant, and whether trained length control and hybrid reasoning are actually necessary if the goal is token-efficient reasoning? 


  2. Are there any specific failure modes that you observe in DASH that do not appear in HRPO which could explain the performance gap? Given the claim of preserving readability, the traces should technically be inspectable (assuming faithfulness) to perform such an analysis, or at least determine whether the gap is due to truncation (early stopping of the CoT due to budget constraints).

2 comments

Thank you for your thoughtful review. We have addressed your concerns in our rebuttal and would appreciate any further feedback or clarification you may have. As the discussion period is nearing its end, we would be grateful if you could take a look at our responses when time permits.

Thank you for the clarifications and for providing the additional analysis. The fixed B=96 experiment, compared against HRPO, does help to make the practical value of the training recipe a bit clearer. I do still think that having a verbalized CoT RL baseline is important, both directly and following SFT, for a fair comparison, and the fixed-token version still has the curriculum and budget-awareness/prediction, which makes it hard to isolate the role of each element. It is also unclear whether a fixed-route model that would be allowed to reason for longer would perform similarly, given that the routing gains are small while using slightly more tokens. Overall, I see some useful evidence for the recipe in terms of length control, but it is insufficiently established as to whether simpler RL alternatives could achieve similar gains, so I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.