acceptodds
Under review as a conference paper at ICLR 2027

CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment

Abstract

Reasoning-capable language models often produce long chains of thought when direct answers suffice, wasting inference compute. Dual-mode models offer both thinking and direct-answer modes, but typically leave mode selection to users. Automating this selection while improving responses under both modes is challenging because routing targets evolve with the policy, strong initial mode preferences can destabilize exploration, and sequence-level objectives entangle routing with response learning. We introduce CounterRoute, an online reinforcement-learning framework that jointly learns routing and mode-conditioned responses in one shared policy directly from a native dual-mode checkpoint, without method-specific SFT warm-up. Paired current-policy counterfactual rollouts estimate cross-mode routing credit, which is assigned only to the routing token, while within-mode GRPO trains response tokens. A paired-to-self-routed curriculum stabilizes early training with forced coverage of both modes before progressively transferring training to autonomous routing. Across nine benchmarks, CounterRoute better balances accuracy and efficiency than heuristic and learned adaptive-routing methods, improving macro-average accuracy at both Qwen3 scales while cutting mean generated tokens by 51% and 41%, respectively, versus always-thinking checkpoints. On instruction-following and commonsense benchmarks where direct answering is strong, think rates fall as low as 1% while response quality improves. Despite training only on math and synthetic instruction-following data, its routing and response quality generalize to held-out tasks in coding, science, knowledge, and commonsense.

Primary area:
reinforcement learning
Keywords:
self-routed reasoning, adaptive test-time compute, hierarchical credit assignment, counterfactual reinforcement learning, reasoning efficiency, GRPO

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3inUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CounterRoute, an online RL framework for a single dual-mode model to learn both when to think and how to answer under thinking and direct-answer modes. It uses paired current-policy rollouts to estimate the relative value of the two modes, assigns this cross-mode credit only to the routing token, and uses within-mode GRPO for response tokens. A paired-to-self-routed curriculum gradually transfers training from forced mode coverage to autonomous routing. Experiments on Qwen3-8B and Qwen3-14B show a better accuracy–efficiency trade-off than fixed-mode, heuristic, and learned routing baselines across nine benchmarks. Same-checkpoint forced-mode evaluations and component ablations further support the routing and credit-assignment design.

Critical strengths
  1. The routing formulation is well motivated and experimentally supported. CounterRoute separates routing credit from response credit, and the ablations show that counterfactual pairing, routing-token credit, and self-routed updates are all important for avoiding near-always-thinking behavior.
  2. The empirical evaluation is strong. The method is evaluated at two model scales and across nine benchmarks, including domains not seen during RL training. The same-checkpoint forced-think/no-think analysis is particularly useful because it directly checks whether learned routing tracks the actual benefit of thinking.
Critical weaknesses
  1. The accuracy improvement is not broad across benchmarks. The headline macro-average gain over Think-Only GRPO is largely driven by IFBench. On Qwen3-8B, CounterRoute is worse than Think-Only on MATH500, GPQA, MMLU-Pro, MMLU, and IFEval; on Qwen3-14B, it is worse on six of nine benchmarks. If IFBench is excluded, the average accuracy is actually slightly lower than Think-Only at both scales. The efficiency gain is clear, but the claim that joint routing also broadly improves response quality is less convincing.

  2. The paper does not clearly show prompt-level adaptive routing. The central claim is that the model decides per prompt whether thinking is worthwhile. However, the main routing analysis compares benchmark-level forced-think gains with benchmark-level think rates. Several datasets are routed almost entirely to one mode: 100% thinking on MATH500, GSM8K, and HumanEval, versus below 1% on IFEval and IFBench. This leaves open whether CounterRoute learns genuine within-task difficulty-sensitive routing or mostly recognizes the task/domain.

  3. Training compute is not matched to the fixed-mode GRPO baselines. CounterRoute uses paired rollouts for part of training, which increases the rollout count by up to 2× early in training and about 1.3× at steady state. Therefore, some of the response-quality gain may come from additional online samples and training compute rather than the routing objective itself. A compute-matched GRPO baseline would make the comparison much cleaner.

Questions for authors
  1. Does CounterRoute actually learn prompt-level routing within a benchmark? For example, within MATH500 or GPQA, are prompts with a larger forced-Think advantage also more likely to be routed to Think? The current analysis mainly compares benchmark-level Think gains with benchmark-level think rates, which does not fully distinguish prompt-level routing from task-level routing.

  2. Can you provide a training-compute-matched GRPO baseline? CounterRoute uses paired Think/No-Think rollouts for part of training, resulting in up to 2× the rollout groups early in training and about 1.3× at steady state. Matching optimizer steps therefore does not fully match training compute.

  3. How robust is the reported macro-average accuracy gain to benchmark aggregation? The improvement over Think-Only GRPO appears to be driven largely by a small number of benchmarks, especially IFBench. Excluding IFBench, CounterRoute is slightly worse on average at both model scales.

5 comments
AuthorsOpenReview ↗

Prompt-level routing within a benchmark (W2, Q1)

We thank the reviewer for recognizing the value of the paper's technical contribution and we are providing more analysis and new experiments to address the questions on aggregation robustness, prompt-level routing, and compute matching.

Summary. For the main model, Figure 3 establishes cross-domain route–value transfer showing generalization to unseen tasks/prompts. As discussed under "Routing alignment and residual misrouting" in the paper, the main model over-thinks on mathematics because of its training data distribution, so within-MATH routing cannot be read from it. To answer Q1 directly, we ran a new single-benchmark MATH experiment, which shows that CounterRoute routes with strong within-benchmark discrimination while maintaining accuracy. Starting from a policy that thinks on every MATH-500 problem, the think rate falls to 60.4% at unchanged accuracy (92.5% versus 92.4%, within 0.5 points of forced thinking). Routing rises from 16% thinking on level 1 to 90% on level 5, and within each level it follows each problem's own forced-thinking advantage; where the two modes differ, the router selects the better mode 84.9% of the time.

Existing evidence. Unseen benchmarks with larger forced-thinking gains generally receive higher think rates (Figure 3); for example, HellaSwag, where forced thinking lowers accuracy, is routed predominantly to direct answering. GPQA's high think rate is consistent with its positive overall forced-thinking gain; we treat this as cross-benchmark alignment and do not claim within-GPQA calibration.

Limitation. The original training mixture contrasts difficult DAPO mathematics with simple instruction following and contains little evidence about mathematics-like prompts that can be answered directly. The main policy therefore retains an overly strong thinking prior on some math-like benchmarks, even though its cross-domain ordering is sensible.

New experiment. We continued CR-8B within a single category, removing benchmark identity as a routing cue:

  • training uses level-balanced MATH problems spanning levels 1–5, with every MATH-500 evaluation problem removed by exact-text matching;
  • the initial CR-8B policy routes every MATH-500 problem to thinking;
  • evaluation uses eight forced-thinking and eight forced-direct samples per problem; and
  • the continuation uses K = 8, scheduled b: 0.05 → 0.01, and a direct-response length cap that prevents unmarked long-form reasoning.
MATH-500 slicenThink rate
Level 14316%
Level 29031%
Level 310549%
Level 412874%
Level 513490%
Forced-thinking gap ≤ 032745%
Gap in (0, 0.25]8185%
Gap > 0.259293.5%

The overall think rate falls from 100% to 60.4% at unchanged accuracy (92.5% self-routed versus 93.0% forced thinking). It rises from 16% on level-1 problems to 90% on level 5, and problems where direct answering already matches thinking are routed to thinking 45% of the time, compared with 85–93.5% for problems where thinking helps.

TestResult
Level-demeaned gap–route correlation+0.248
Accuracy gain over routes shuffled within each level+2.70 [1.80, 3.70] points
Selected route no worse than forced alternative94.4%
Better route selected when outcomes differ84.9%
Counterfactual gaps shuffled across prompts during trainingRemains at 100% thinking

The two shuffle controls separate prompt-level routing from coarser explanations. The within-level shuffle keeps each level's think rate fixed and randomly reassigns which problems think; if routing only followed difficulty level, this would not lower accuracy. The training-time control permutes the counterfactual gaps across the prompts of each batch, so the overall push toward or away from thinking is unchanged but no longer matches the individual prompt; this run never leaves uniform thinking. The over-routing of the main model is therefore a correctable data-calibration issue, not a restriction to task-level routing.

AuthorsOpenReview ↗

Compute-matched fixed-mode GRPO (W3, Q2)

Summary. Under both measures of training compute, Think-Only GRPO with an equal or larger budget does not exceed CounterRoute. The reported GRPO policy already uses more generated tokens, and extending it to a matched rollout count lowers its accuracy. The extended runs retain the same initialization, data mixture, optimizer, rollout configuration, and evaluation protocol; only the training budget changes.

Training runCumulative rolloutsCumulative generated rollout tokensNine-task accuracy
Think-Only GRPO, reported operating point12,80061.0M78.3%
Think-Only GRPO, 95% of CR rollout count20,48078.7M76.7%
Think-Only GRPO, 107% of CR rollout count23,04086.0M76.2%
CounterRoute, reported operating point21,504approximately 50M79.1%

Generated-token matching.

  • The reported Think-Only GRPO policy already consumes more training tokens than CounterRoute (61.0M versus approximately 50M) without exceeding its accuracy.
  • CounterRoute's rollout count is exact. Its token count is estimated from the stable tokens-per-rollout ratio of a fully logged controlled run because the main run did not retain rollout dumps.

Rollout-count matching.

  • Extending the same GRPO run to 95% and 107% of CounterRoute's rollout count lowers average accuracy to 76.7% and 76.2%.
  • MATH-500 falls from 92.8% to 82.5%, and GPQA from 57.9% to 49.0%, as the extended fixed-mode run progressively shortens and degrades its reasoning.

Thus, additional online trajectories do not recover CounterRoute's result. Under equal or larger training budgets, fixed-mode GRPO does not exceed CounterRoute. The main advantage remains lower inference generation cost at comparable or better aggregate accuracy, rather than uniform per-benchmark score improvement.

Uniform profiler telemetry was not retained, so we report normalized hardware estimates relative to fixed-mode GRPO: at most 1.3× rollout workload per update after the curriculum transition (at least 0.77× update throughput), since the additional direct-answer rollouts are short, and comparable peak memory under identical model parallelism and per-device microbatching.

AuthorsOpenReview ↗

Breadth of the accuracy gains and aggregation robustness (W1, Q3)

Summary. The macro-average gain over Think-Only GRPO depends on IFBench. Removing IFBench changes the difference to −0.6 points at 8B and −0.4 at 14B, while removing any other single benchmark leaves it positive. Most per-benchmark differences are within evaluation uncertainty, and CounterRoute generates significantly fewer tokens. We therefore present the main result as a better accuracy–efficiency balance, with IFBench as the clearest held-out score improvement, rather than as uniform per-benchmark improvement.

CR minus Think-Only GRPOQwen3-8BQwen3-14B
Nine-benchmark macro-average+0.8 [−0.04, +1.53]+1.7 [+0.95, +2.53]
Macro-average after removing IFBench−0.6−0.4
Macro-average after removing any other single benchmarkPositive in every casePositive in every case
Median per-benchmark difference−0.4−0.2
Significantly better / worse / not significant1 / 2 / 61 / 1 / 7

Of the benchmarks with lower point estimates (five at 8B, six at 14B), only two and one differences, respectively, are significant under paired bootstrap tests; the others are within evaluation uncertainty.

Why IFBench is informative.

  • Qwen3 was released on April 29, 2025; IFBench v1 was released on July 3, 2025 and contains 58 new, verifiable instruction constraints.
  • IFBench is absent from CounterRoute's RL mixture and is the only benchmark in the suite released after Qwen3, making it a useful temporal and constraint-level generalization test.
  • There is substantial headroom: Always Think scores 34.7%/37.5%, Think-Only GRPO scores 34.9%/37.8%, and CounterRoute scores 46.7%/56.9% for Qwen3-8B/Qwen3-14B.
  • For Qwen3-8B, the CounterRoute policy scores 46.2% when forced to think and 46.1% when forced to answer directly, close to 46.7% under self-routing. The increase therefore persists under either route rather than arising only from route selection.

Conclusion. The IFBench improvement is a meaningful result in its own right, as a held-out test released after the backbone. Even without it, our main claim holds: excluding IFBench, CounterRoute stays within one point of Think-Only GRPO, with most per-benchmark differences within evaluation uncertainty, while generating significantly fewer tokens, 51.4% and 41.4% fewer than always thinking. CounterRoute therefore achieves a better accuracy–efficiency balance with or without IFBench.

Reviewer_XF6HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that automating the selection between thinking and no-thinking modes in large language models is important yet challenging. CounterRoute is introduced to address these challenges jointly. The method uses paired current-policy rollouts under thinking and no-thinking modes to estimate their relative utility. It assigns cross-mode credit to the routing token while applying within-mode GRPO to response tokens. A paired-to-self-routed curriculum is further introduced to mitigate early routing collapse and progressively train the model under its own routing decisions. Experiments on Qwen3-8B and Qwen3-14B across nine benchmarks demonstrate improved accuracy–efficiency trade-offs compared with fixed-mode GRPO and several adaptive-routing baselines. Additional ablations examine the contributions of hierarchical credit assignment, paired counterfactual supervision, and the training curriculum.

Critical strengths
  1. The paper studies the routing between thinking and non-thinking modes in practical applications of large language models. This problem is critical, as its effective resolution can optimize inference costs and achieve a better cost-performance trade-off. The paper reasonably points out three issues in existing research and offers targeted solutions in the proposed method.

  2. The paper provides extensive evaluations across nine benchmarks on Qwen3-8B and Qwen3-14B. The results demonstrate that the proposed method achieves a better trade-off between accuracy and inference efficiency compared with several existing methods, significantly reducing the number of generated tokens while maintaining or improving response accuracy.

  3. The ablation studies investigate the effectiveness of the proposed components, including counterfactual pairing, hierarchical credit assignment, and the paired-to-self-routed curriculum, providing further support for the method's design.

Critical weaknesses
  1. Previous works such as Thinkless and HGPO have already explored the separate optimization of routing and response generation, while HGPO also employs paired rollouts and inter-group advantages for routing tokens. The main methodological contributions of CounterRoute are the routing-token advantage overwrite and the paired-to-self-routed curriculum. Although these designs appear reasonable and the paper reports promising results, the analysis of why they work remains limited.

  2. It remains unclear why including intra-group advantages in routing-token optimization would lead to ineffective routing decisions, and whether the benefit of advantage overwriting indeed comes from eliminating interference between routing and response optimization.

  3. While the paired-to-self-routed curriculum is intended to address self-reinforcing exploration, the paper provides limited analysis of how this problem develops during training and how the proposed curriculum mitigates it.

  4. The reliability of the counterfactual routing supervision on difficult problems remains unclear. CounterRoute estimates the utility difference between thinking and non-thinking modes using only a small number of sampled responses. When both modes fail to produce correct answers, the estimated utility gap may be dominated by the efficiency margin, encouraging the model to select non-thinking even if thinking has a higher underlying probability of success.

Questions for authors
  1. The paper aims to determine whether thinking is beneficial for each prompt, but the reported thinking rates appear strongly associated with benchmark categories. Could the authors further investigate whether the learned policy can distinguish problems with different reasoning requirements within the same benchmark, rather than primarily relying on task-level patterns?

  2. Could the authors further explain why including intra-group advantages in routing-token optimization may interfere with effective routing learning? What is the underlying mechanism through which the proposed advantage overwrite addresses this issue?

6 comments
AuthorsOpenReview ↗

Contribution and mechanism (Response to W1, W2, Q2, W3)

We thank the reviewer for their detailed questions regarding the underlying mechanisms. To address these concerns, we have conducted additional analyses and experiments and provided more detailed explanations.

The two prior designs each leave one failure mode open.

  • Thinkless separates control-token and response objectives, but learns only from modes the policy itself samples. When one mode dominates initially, the other is rarely sampled, so it neither improves nor gathers routing evidence, and exploration collapses onto the dominant mode. Starting from a thinking-preferring model, direct answering cannot emerge (self-routed-only arm, Table 2).
  • HGPO avoids this collapse with paired forced-mode rollouts, but keeps paired sampling throughout its RL stage, so the router is never trained under its own routing distribution. It also adds the inter-group routing credit to the intra-group response advantage at the routing token; we analyze the problem this causes in (a).

CounterRoute addresses both. It overwrites the routing-token advantage with counterfactual credit, and its paired-to-self-routed curriculum supplies cross-mode evidence early and then trains under the model's own routing decisions (b). It starts directly from a native dual-mode checkpoint without method-specific SFT or hybrid-fine-tuning warm-up.

(a) Why intra-group advantages interfere with routing, and what overwrite fixes (W2, Q2). In the ideal GRPO setting, additive and overwrite credit produce the same expected routing gradient. The difference comes from three properties of actual training:

  • Intra-group response noise. Under additive credit, each row's routing-token advantage is the routing credit plus that response's within-group advantage, so the sign of an individual row depends on whether its response happened to beat its group, not only on whether thinking helps.
  • PPO clipping. The sign of each row's advantage selects its clipping branch, so rows whose sign was flipped by the response term are clipped differently and no longer cancel.
  • Minibatching. The rows of one group are spread across minibatches, so the zero-sum response term is not cancelled within a single update.

These effects matter most where routing credit is weak, which is exactly where the model learns that thinking is unnecessary. Overwrite removes them by giving every row the sign of its prompt's counterfactual credit. Our claim is therefore narrow: overwrite does not change the ideal expected gradient or uniformly improve the endpoint; it makes the emergence of selective routing reliable.

Ideal case. Cross-mode credit ama_m, a bounded function of RˉTH(x)−RˉNT(x)−b\bar R_{\text{TH}}(x) - \bar R_{\text{NT}}(x) - b, asks whether thinking improves expected utility for prompt xx. Within-mode credit cm,kc_{m,k}, based on Rm,k(x)−Rˉm(x)R_{m,k}(x) - \bar R_m(x), asks whether a response beats its group after the route is fixed. All K rows of a forced mode share the routing-token gradient hm=∇θlog⁡πθ(m∣x)h_m = \nabla_\theta \log \pi_\theta(m \mid x), and ∑kcm,k=0\sum_k c_{m,k} = 0, so for a complete, equally weighted group under an unclipped objective the response term cancels: ∑k(am+cm,k),hm=Kamhm\sum_k (a_m + c_{m,k})\\, h_m = K a_m h_m.

Simulation. To locate where the three effects matter, we simulated 3,000 training steps with the MATH-continuation settings (K = 8, routing scale 4, b = 0.05, 32 prompts per step), using the starting checkpoint's observed MATH-500 success probabilities.

Table 1. Routing-advantage sign changes under additive credit. Rows are restricted to groups with reward variance, since additive and overwrite differ only on these rows (25% of simulated rows).

Routing-credit magnitudeShare among response-variable rowsSign changed by additive credit
<0.25, ties or near-ties1.3%50.0%
0.25–1.033.4%43.1%
1.0–2.553.0%1.1%
>2.512.3%0.0%

As the cancellation identity predicts, the aggregate update is nearly unchanged: the additive minibatch update correlates 0.99 with overwrite and adds only about 2% extra variance. The damage is per-row and local: 43–50% of weak or moderate rows change sign, while strong "think" evidence is almost unaffected.

Training evidence. Our controlled position-0 ablation holds the later-token objective and all other settings fixed.

Position-0 creditAccuracyThink rateMean words
Counterfactual overwrite77.4%69.0%1,015
Additive, HGPO-style77.7%99.9%1,149
Response advantage only77.5%100%1,002

Accuracy is similar, but only overwrite moves the router off uniform thinking. Under the same ablation settings, we find more generally that selective routing emerges less reliably with additive credit than with overwrite.

AuthorsOpenReview ↗

Contribution and mechanism (Response to W1, W2, Q2, W3) Part2

(b) Why the curriculum helps (W3, W1). With self-routing alone, a strong initial thinking preference removes support for the alternative mode: no direct-answer trajectories are sampled, that mode cannot improve, and the router receives no evidence that could reverse its preference. Pairing restores coverage, while later self-routed groups optimize under the distribution induced by the model's own decisions.

Table 2. Paired-fraction ablation (b = 0.05). The correlation column measures whether the router assigns higher thinking probability to prompts where thinking helps more: in each training step, it is the correlation, across the batch's paired prompts, between each prompt's utility gap and the router's p_think on that prompt. We report the range of its 20-step means over steps 41–120.

ArmPaired fractionRouting behaviorgap–p_think correlationNo-think training samples
Self-routed only0Remains at 100% thinkingUndefinedNone
Always paired1Leaves saturation only late0.02–0.09Forced only
Curriculum (ours)1 → 0.3Leaves saturation during the transition0.15–0.24Forced, then self-selected

At the reviewer's request, we added the complete trajectories as Appendix Figure 5:

  • self-routing alone never samples a direct-answer response, so neither that mode nor its routing evidence can improve;
  • always-paired training provides forced evidence but develops little prompt-level alignment; and
  • the curriculum first improves both modes under forced coverage, then optimizes under the policy's own routing distribution.

The always-paired arm mirrors HGPO's Stage-II rollout schedule, which samples forced Thinking and No-Thinking groups for every prompt, while retaining CounterRoute's remaining objective and optimization choices. It isolates the effect of never transferring from forced comparison to autonomous self-routing.

AuthorsOpenReview ↗

Prompt-level routing and reliability (Q1, W4)

Summary. The submission already shows that routing generalizes to held-out domains rather than reproducing the two training categories. As discussed under "Routing alignment and residual misrouting", in the paper, we acknowledge that the main model over-thinks on mathematics due to training data distribution. To test finer-grained routing within one category, we ran a new single-benchmark MATH experiment. Starting from a policy that thinks on every MATH-500 problem, routing becomes graded by difficulty and tracks each problem's own thinking benefit beyond its level, while accuracy is maintained. On the hardest problems, including those both modes fail, the policy keeps thinking rather than following the efficiency margin.

Existing evidence.

  • The RL training mixture contains only two task families: relatively difficult DAPO-Math-17k problems and simpler instruction following, with mean thinking-minus-direct utility gaps of +15.4 and −2.0 points.
  • Routing nevertheless transfers to held-out coding, science, knowledge, and commonsense benchmarks in a value-dependent way (Figure 3). For example, HellaSwag, where forced thinking lowers accuracy, is routed predominantly to direct answering.

Limitation. As discussed in under "Routing alignment and residual misrouting" in the experiment section, the main model over-thinks on mathematics: it routes every GSM8K and MATH-500 prompt to thinking, although direct answering suffices for some of them. We attribute this to the training mixture, which provides little within-math evidence that direct answering can be enough.

New experiment. To test whether CounterRoute can route at a finer granularity within a single category, we ran a new MATH experiment that removes benchmark identity as a routing cue:

  • training uses level-balanced MATH problems spanning levels 1–5, with every MATH-500 evaluation problem removed by exact-text matching;
  • the initial CR-8B policy routes every MATH-500 problem to thinking;
  • evaluation uses eight forced-thinking and eight forced-direct samples per problem;
  • the continuation uses K = 8, scheduled b: 0.05 → 0.01, and a direct-response length cap that prevents unmarked long-form reasoning.

Prompt-sensitive routing within MATH. The direct answer to Q1 is yes: prompts with a larger same-checkpoint forced-thinking advantage are more likely to be routed to thinking, beyond what difficulty-level identity explains.

MATH-500 slicenThink rate
Level 14316%
Level 29031%
Level 310549%
Level 412874%
Level 513490%
Forced-thinking gap ≤ 032745%
Gap in (0, 0.25]8185%
Gap > 0.259293.5%

The think rate rises from 16% on level-1 problems to 90% on level 5. Across problems, those where direct answering already matches thinking are routed to thinking 45% of the time, compared with 85–93.5% for problems where thinking helps.

TestResult
Level-demeaned prompt gap–route correlation+0.248
Accuracy gain over routes shuffled within each level+2.70 [1.80, 3.70] points
Selected route no worse than forced alternative94.4%
Better route selected when outcomes differ84.9%
Counterfactual gaps shuffled across promptsRemains at 100% thinking

The two shuffle controls separate prompt-level routing from coarser explanations. The within-level shuffle keeps each level's think rate fixed and randomly reassigns which problems think; if routing only followed difficulty level, this would not lower accuracy. The training-time control permutes the counterfactual gaps across the prompts of each batch, so the overall push toward or away from thinking is unchanged but no longer matches the individual prompt; this run never leaves uniform thinking. Routing therefore contains prompt-specific information beyond benchmark and difficulty-level identity, and learning it requires credit tied to the individual prompt.

Reliability on difficult prompts.

  • If every sampled rollout fails, the margin contributes only 4·tanh(0.05) ≈ 0.20, falling to 4·tanh(0.01) ≈ 0.04 after decay.
  • Observed evidence that thinking helps commonly produces routing-credit magnitudes of 1–2.5, five to twelve times larger.
Sampled outcome groupnThink rate
Thinking helps, but forced-think pass rate is at most 0.58100%
Both modes fail in all 8 + 8 forced samples1491.1%
Both modes succeed in all 8 + 8 forced samples29440.1%

Thus, the learned policy remains strongly biased toward thinking on empirically difficult prompts, rather than following the small efficiency margin toward direct answering. Finite-K sampling can still create false ties: a mode with success probability p produces no success with probability (1 − p)^K. We therefore describe the signal as sampled current-policy evidence, not a ground-truth label.

Reviewer_uvymOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CounterRoute trains a dual-mode LM to choose between reasoning and direct answering without a separate router. The first token (<think> or </think>) acts as the routing decision. Paired rollouts compare both modes: cross-mode reward differences train the routing token, while within-mode GRPO trains response tokens. A curriculum shifts from forced paired exploration to self-routing. On Qwen3-8B/14B across nine benchmarks, it improves average accuracy while reducing generated tokens by 51%/41% versus always thinking.

Critical strengths
  1. The paper is well motivated. Automatically deciding when explicit reasoning is needed addresses an important and practical trade-off between accuracy and inference efficiency.
  2. CounterRoute separates routing credit from response credit at the token level, enabling joint training without a separate router or additional SFT warm-up. The technical design is clean and intuitive.
  3. The empirical validation is strong. Experiments on two model scales and nine benchmarks show improved average accuracy with 41–51% fewer generated tokens, supported by relevant baselines and component ablations.
Critical weaknesses
  1. The routing analysis is mainly conducted across benchmarks, so it remains unclear whether the model makes fine-grained decisions based on individual sample difficulty or primarily learns broad domain-level routing patterns.
  2. The efficiency evaluation focuses on generated tokens. Since paired rollouts introduce additional training overhead, reporting GPU-hours, throughput, memory usage, and wall-clock inference latency would provide a more complete view of the end-to-end cost.
Questions for authors
  1. Within a single benchmark, does the routing probability correlate with sample difficulty or the empirical performance gap between thinking and no-thinking?
  2. What are the total training costs of CounterRoute and the baselines in terms of generated tokens, GPU-hours, throughput, and peak memory?
  3. Are the main results based on multiple training and evaluation seeds? If so, could you report variance or confidence intervals for the key comparisons?
  4. For the reported 95.08% prompt-level routing alignment, how much of this result comes from ties where both modes are correct or both are incorrect? Among examples where the two modes produce different outcomes, how often does the router select the better mode?
6 comments
AuthorsOpenReview ↗

Prompt-level routing and interpretation of the alignment statistic (W1, Q1, Q4)

We thank the reviewer for the positive assessment of the motivation, technical design, and empirical validation. We further provided detailed analysis and additional experiments to address the concerns and questions.

Summary. The submission already shows that routing generalizes to held-out domains rather than reproducing the two training categories. As discussed under "Routing alignment and residual misrouting" in the paper, we acknowledge that the main model over-thinks on mathematics. To test finer-grained routing within one category, we ran a new single-benchmark MATH experiment. Starting from a policy that thinks on every MATH-500 problem, routing becomes graded by difficulty and tracks each problem's own thinking benefit beyond its level, while the think rate falls to 60.4% at unchanged accuracy. The 95.08% statistic is a tie-inclusive non-degradation rate; on prompts where the two modes differ, the router selects the better mode 75.9% of the time, and 84.9% in the MATH experiment.

Existing evidence.

  • The RL mixture contains only two task families: relatively difficult DAPO-Math-17k problems and simpler instruction following, with mean thinking-minus-direct utility gaps of +15.4 and −2.0 points.
  • Routing nevertheless transfers to held-out coding, science, knowledge, and commonsense benchmarks in a value-dependent way (Figure 3). For example, HellaSwag, where forced thinking lowers accuracy, is routed predominantly to direct answering.

Limitation. As discussed under "Routing alignment and residual misrouting," the main model over-thinks on mathematics: it routes every GSM8K and MATH-500 prompt to thinking, although direct answering suffices for many of them. We attribute this to the training mixture, which provides little within-math evidence that direct answering can be enough.

New experiment. To test whether CounterRoute can route at a finer granularity within a single category, we ran a new MATH experiment that removes benchmark identity as a routing cue:

  • training uses level-balanced MATH problems spanning levels 1–5, with every MATH-500 evaluation problem removed by exact-text matching;
  • the initial CR-8B policy routes every MATH-500 problem to thinking;
  • evaluation uses eight forced-thinking and eight forced-direct samples per problem; and
  • the continuation uses K = 8, scheduled b: 0.05 → 0.01, and a direct-response length cap that prevents unmarked long-form reasoning.
MetricBefore continuationAfter continuation
Self-routed accuracy92.4%92.5%
Think rate100%60.4%
Mean response length983 words779 words
Forced-thinking accuracy92.2%93.0%
Forced-direct accuracy56.3%79.7%

The policy now answers 40% of problems directly at unchanged accuracy, and mean response length falls by 21%. Forced-direct accuracy rises from 56.3% to 79.7%, so direct answering has become a viable choice rather than a shortcut.

MATH-500 slicenThink rate
Level 14316%
Level 29031%
Level 310549%
Level 412874%
Level 513490%
Forced-thinking gap ≤ 032745%
Gap in (0, 0.25]8185%
Gap > 0.259293.5%

The think rate rises from 16% on level-1 problems to 90% on level 5. Problems where direct answering already matches thinking are routed to thinking 45% of the time, compared with 85–93.5% for problems where thinking helps.

TestResult
Level-demeaned prompt gap–route correlation+0.248
Accuracy gain over routes shuffled within each level+2.70 [1.80, 3.70] points
Counterfactual gaps shuffled across prompts during trainingRemains at 100% thinking

The two shuffle controls separate prompt-level routing from coarser explanations. The within-level shuffle keeps each level's think rate fixed and randomly reassigns which problems think; if routing only followed difficulty level, this would not lower accuracy. The training-time control permutes the counterfactual gaps across the prompts of each batch, so the overall push toward or away from thinking is unchanged but no longer matches the individual prompt; this run never leaves uniform thinking.

Interpreting the 95.08% statistic (Q4).

EvaluationSelected route no worseBetter route selected when outcomes differ
Main CR-8B, nine benchmarks95.08%75.9%
New MATH experiment94.4%84.9%

Ties make up 79.6% of the main evaluation (62.9% both correct, 16.6% both wrong), so 95.08% is a non-degradation rate rather than routing accuracy; the second column measures better-mode selection directly. In the MATH experiment, the router answers 59.9% of both-correct ties directly but only 8.9% of both-wrong ties, saving computation on easy problems while continuing to reason on hard ones.

AuthorsOpenReview ↗

End-to-end training and inference cost (W2, Q2)

Summary. Measured by generated tokens, CounterRoute's training budget is below that of the reported Think-Only GRPO policy; measured by rollout count, it is 1.68 times larger, and giving GRPO a matched rollout budget lowers its accuracy. At inference, CounterRoute uses one model with no external router and generates 51.4% and 41.4% fewer tokens than always thinking.

Training workload.

  • All controlled runs use the same 32-GPU stack, model parallelism, and per-device microbatching.
  • With paired fraction ρ, CounterRoute processes 1+ρ rollout groups per update. This is 2.0× during initial forced coverage and 1.3× under the final 70/30 self-routed/paired mixture.
  • Direct-answer rollouts are much shorter than thinking rollouts. Consequently, rollout count and generated-token count give different, complementary views of training cost.
Training runCumulative rolloutsGenerated rollout tokensNine-benchmark accuracy
CounterRoute, reported policy21,504approximately 50M79.1%
Think-Only GRPO, reported policy12,80061.0M78.3%
Think-Only GRPO, 95% of CR rollout count20,48078.7M76.7%
Think-Only GRPO, 107% of CR rollout count23,04086.0M76.2%

The reported Think-Only GRPO policy already consumes more generated training tokens than CounterRoute. Increasing GRPO to matched or greater rollout counts lowers rather than improves its average accuracy. The result is therefore not explained by CounterRoute observing more online trajectories.

Hardware-normalized estimate.

QuantityCounterRoute relative to fixed-mode GRPO
Rollout workload per update after the curriculum transitionat most 1.3×
Optimizer-update throughput after the curriculum transitionat least 0.77×
Peak memorycomparable

These hardware values are normalized estimates because uniform profiler telemetry was not retained. The per-update figures are bounds, since the additional direct-answer rollouts are much shorter than thinking rollouts. Peak memory remains similar because pairing changes the number of rollout groups rather than model size or per-device microbatching.

Inference cost. Mean generated tokens fall by 51.4% and 41.4% relative to the always-thinking Qwen3-8B and Qwen3-14B policies. We report generated tokens rather than claim an unmeasured wall-clock latency speedup.

AuthorsOpenReview ↗

Evaluation uncertainty (Q3)

Summary. The main results use one training seed per learned method, because each full RL run occupies 32 GPUs for several days; we will state this limitation explicitly. We now report evaluation uncertainty for every score in the main table. These intervals are conditional on the trained policies and do not measure variation across independent training runs. We will also add these new CI to our paper draft.

Each interval comes from 2,000 bootstrap replicates over prompts. GPQA is resampled by question across its answer-order presentations, and multi-sample benchmarks are resampled by per-prompt mean score.

BenchmarkCR-8BThink-Only 8BAlways-Think 8BCR-14BThink-Only 14BAlways-Think 14B
GSM8K95.8 ± 1.195.6 ± 1.195.8 ± 1.196.1 ± 1.196.3 ± 1.096.1 ± 1.1
MATH50092.5 ± 2.092.8 ± 1.992.0 ± 2.094.0 ± 1.894.4 ± 1.893.1 ± 1.9
GPQA56.9 ± 5.157.9 ± 5.357.6 ± 5.461.9 ± 5.561.2 ± 5.562.7 ± 5.5
HumanEval93.3 ± 3.992.7 ± 3.990.2 ± 4.693.9 ± 3.796.3 ± 2.993.3 ± 3.9
MMLU-Pro73.3 ± 0.8†73.8 ± 0.865.8 ± 0.976.7 ± 0.776.6 ± 0.874.9 ± 0.8
MMLU81.2 ± 0.682.4 ± 0.682.7 ± 0.684.4 ± 0.684.9 ± 0.684.5 ± 0.6
HellaSwag84.0 ± 0.783.9 ± 0.784.0 ± 0.787.4 ± 0.687.5 ± 0.687.8 ± 0.6
IFEval88.0 ± 2.0†90.9 ± 1.690.2 ± 1.890.5 ± 1.691.4 ± 1.691.1 ± 1.7
IFBench46.7 ± 4.734.9 ± 4.434.7 ± 4.456.8 ± 4.537.8 ± 4.537.5 ± 4.6
Macro average79.1 ± 1.0†78.3 ± 0.977.0 ± 1.082.4 ± 1.080.7 ± 0.980.1 ± 1.0

Here, ± is the 95% CI half-width. Because all policies are evaluated on the same prompts, significance statements use paired bootstrap differences rather than overlap between the displayed intervals.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: New Experiments and Revised Paper

We thank the reviewers for recognizing the technical contribution of the paper. To address the questions raised, we conducted additional experiments and analyses.

In particular, we added a new single-benchmark CounterRoute training run on MATH. It further shows that the method routes according to problem difficulty and each problem's own thinking-versus-direct gap, rather than benchmark-level signals: starting from a policy that thinks on every MATH-500 problem, the think rate falls to 60.4% at unchanged accuracy, rises from 16% on level 1 to 90% on level 5, and tracks the per-problem gap within each level (responses below; new Appendix D.4 in the revised paper). We also added a training-dynamics analysis of the paired-to-self-routed curriculum, showing how it avoids the routing collapse of self-routed-only and always-paired training (new Figure 5).

Further additions include confidence intervals for all main results, compute-matched GRPO comparisons, a decomposition of the prompt-level routing statistic, an aggregation analysis of the held-out IFBench gain, and a clarified analysis of the routing-token overwrite mechanism.

We thank the reviewers again for their insightful comments and look forward to further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.