- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CounterRoute, an online RL framework for a single dual-mode model to learn both when to think and how to answer under thinking and direct-answer modes. It uses paired current-policy rollouts to estimate the relative value of the two modes, assigns this cross-mode credit only to the routing token, and uses within-mode GRPO for response tokens. A paired-to-self-routed curriculum gradually transfers training from forced mode coverage to autonomous routing. Experiments on Qwen3-8B and Qwen3-14B show a better accuracy–efficiency trade-off than fixed-mode, heuristic, and learned routing baselines across nine benchmarks. Same-checkpoint forced-mode evaluations and component ablations further support the routing and credit-assignment design.
Critical strengths
- The routing formulation is well motivated and experimentally supported. CounterRoute separates routing credit from response credit, and the ablations show that counterfactual pairing, routing-token credit, and self-routed updates are all important for avoiding near-always-thinking behavior.
- The empirical evaluation is strong. The method is evaluated at two model scales and across nine benchmarks, including domains not seen during RL training. The same-checkpoint forced-think/no-think analysis is particularly useful because it directly checks whether learned routing tracks the actual benefit of thinking.
Critical weaknesses
-
The accuracy improvement is not broad across benchmarks. The headline macro-average gain over Think-Only GRPO is largely driven by IFBench. On Qwen3-8B, CounterRoute is worse than Think-Only on MATH500, GPQA, MMLU-Pro, MMLU, and IFEval; on Qwen3-14B, it is worse on six of nine benchmarks. If IFBench is excluded, the average accuracy is actually slightly lower than Think-Only at both scales. The efficiency gain is clear, but the claim that joint routing also broadly improves response quality is less convincing.
-
The paper does not clearly show prompt-level adaptive routing. The central claim is that the model decides per prompt whether thinking is worthwhile. However, the main routing analysis compares benchmark-level forced-think gains with benchmark-level think rates. Several datasets are routed almost entirely to one mode: 100% thinking on MATH500, GSM8K, and HumanEval, versus below 1% on IFEval and IFBench. This leaves open whether CounterRoute learns genuine within-task difficulty-sensitive routing or mostly recognizes the task/domain.
-
Training compute is not matched to the fixed-mode GRPO baselines. CounterRoute uses paired rollouts for part of training, which increases the rollout count by up to 2× early in training and about 1.3× at steady state. Therefore, some of the response-quality gain may come from additional online samples and training compute rather than the routing objective itself. A compute-matched GRPO baseline would make the comparison much cleaner.
Questions for authors
-
Does CounterRoute actually learn prompt-level routing within a benchmark? For example, within MATH500 or GPQA, are prompts with a larger forced-Think advantage also more likely to be routed to Think? The current analysis mainly compares benchmark-level Think gains with benchmark-level think rates, which does not fully distinguish prompt-level routing from task-level routing.
-
Can you provide a training-compute-matched GRPO baseline? CounterRoute uses paired Think/No-Think rollouts for part of training, resulting in up to 2× the rollout groups early in training and about 1.3× at steady state. Matching optimizer steps therefore does not fully match training compute.
-
How robust is the reported macro-average accuracy gain to benchmark aggregation? The improvement over Think-Only GRPO appears to be driven largely by a small number of benchmarks, especially IFBench. Excluding IFBench, CounterRoute is slightly worse on average at both model scales.
5 comments
Compute-matched fixed-mode GRPO (W3, Q2)
Summary. Under both measures of training compute, Think-Only GRPO with an equal or larger budget does not exceed CounterRoute. The reported GRPO policy already uses more generated tokens, and extending it to a matched rollout count lowers its accuracy. The extended runs retain the same initialization, data mixture, optimizer, rollout configuration, and evaluation protocol; only the training budget changes.
| Training run | Cumulative rollouts | Cumulative generated rollout tokens | Nine-task accuracy |
|---|---|---|---|
| Think-Only GRPO, reported operating point | 12,800 | 61.0M | 78.3% |
| Think-Only GRPO, 95% of CR rollout count | 20,480 | 78.7M | 76.7% |
| Think-Only GRPO, 107% of CR rollout count | 23,040 | 86.0M | 76.2% |
| CounterRoute, reported operating point | 21,504 | approximately 50M | 79.1% |
Generated-token matching.
- The reported Think-Only GRPO policy already consumes more training tokens than CounterRoute (61.0M versus approximately 50M) without exceeding its accuracy.
- CounterRoute's rollout count is exact. Its token count is estimated from the stable tokens-per-rollout ratio of a fully logged controlled run because the main run did not retain rollout dumps.
Rollout-count matching.
- Extending the same GRPO run to 95% and 107% of CounterRoute's rollout count lowers average accuracy to 76.7% and 76.2%.
- MATH-500 falls from 92.8% to 82.5%, and GPQA from 57.9% to 49.0%, as the extended fixed-mode run progressively shortens and degrades its reasoning.
Thus, additional online trajectories do not recover CounterRoute's result. Under equal or larger training budgets, fixed-mode GRPO does not exceed CounterRoute. The main advantage remains lower inference generation cost at comparable or better aggregate accuracy, rather than uniform per-benchmark score improvement.
Uniform profiler telemetry was not retained, so we report normalized hardware estimates relative to fixed-mode GRPO: at most 1.3× rollout workload per update after the curriculum transition (at least 0.77× update throughput), since the additional direct-answer rollouts are short, and comparable peak memory under identical model parallelism and per-device microbatching.
Breadth of the accuracy gains and aggregation robustness (W1, Q3)
Summary. The macro-average gain over Think-Only GRPO depends on IFBench. Removing IFBench changes the difference to −0.6 points at 8B and −0.4 at 14B, while removing any other single benchmark leaves it positive. Most per-benchmark differences are within evaluation uncertainty, and CounterRoute generates significantly fewer tokens. We therefore present the main result as a better accuracy–efficiency balance, with IFBench as the clearest held-out score improvement, rather than as uniform per-benchmark improvement.
| CR minus Think-Only GRPO | Qwen3-8B | Qwen3-14B |
|---|---|---|
| Nine-benchmark macro-average | +0.8 [−0.04, +1.53] | +1.7 [+0.95, +2.53] |
| Macro-average after removing IFBench | −0.6 | −0.4 |
| Macro-average after removing any other single benchmark | Positive in every case | Positive in every case |
| Median per-benchmark difference | −0.4 | −0.2 |
| Significantly better / worse / not significant | 1 / 2 / 6 | 1 / 1 / 7 |
Of the benchmarks with lower point estimates (five at 8B, six at 14B), only two and one differences, respectively, are significant under paired bootstrap tests; the others are within evaluation uncertainty.
Why IFBench is informative.
- Qwen3 was released on April 29, 2025; IFBench v1 was released on July 3, 2025 and contains 58 new, verifiable instruction constraints.
- IFBench is absent from CounterRoute's RL mixture and is the only benchmark in the suite released after Qwen3, making it a useful temporal and constraint-level generalization test.
- There is substantial headroom: Always Think scores 34.7%/37.5%, Think-Only GRPO scores 34.9%/37.8%, and CounterRoute scores 46.7%/56.9% for Qwen3-8B/Qwen3-14B.
- For Qwen3-8B, the CounterRoute policy scores 46.2% when forced to think and 46.1% when forced to answer directly, close to 46.7% under self-routing. The increase therefore persists under either route rather than arising only from route selection.
Conclusion. The IFBench improvement is a meaningful result in its own right, as a held-out test released after the backbone. Even without it, our main claim holds: excluding IFBench, CounterRoute stays within one point of Think-Only GRPO, with most per-benchmark differences within evaluation uncertainty, while generating significantly fewer tokens, 51.4% and 41.4% fewer than always thinking. CounterRoute therefore achieves a better accuracy–efficiency balance with or without IFBench.
Prompt-level routing within a benchmark (W2, Q1)
We thank the reviewer for recognizing the value of the paper's technical contribution and we are providing more analysis and new experiments to address the questions on aggregation robustness, prompt-level routing, and compute matching.
Summary. For the main model, Figure 3 establishes cross-domain route–value transfer showing generalization to unseen tasks/prompts. As discussed under "Routing alignment and residual misrouting" in the paper, the main model over-thinks on mathematics because of its training data distribution, so within-MATH routing cannot be read from it. To answer Q1 directly, we ran a new single-benchmark MATH experiment, which shows that CounterRoute routes with strong within-benchmark discrimination while maintaining accuracy. Starting from a policy that thinks on every MATH-500 problem, the think rate falls to 60.4% at unchanged accuracy (92.5% versus 92.4%, within 0.5 points of forced thinking). Routing rises from 16% thinking on level 1 to 90% on level 5, and within each level it follows each problem's own forced-thinking advantage; where the two modes differ, the router selects the better mode 84.9% of the time.
Existing evidence. Unseen benchmarks with larger forced-thinking gains generally receive higher think rates (Figure 3); for example, HellaSwag, where forced thinking lowers accuracy, is routed predominantly to direct answering. GPQA's high think rate is consistent with its positive overall forced-thinking gain; we treat this as cross-benchmark alignment and do not claim within-GPQA calibration.
Limitation. The original training mixture contrasts difficult DAPO mathematics with simple instruction following and contains little evidence about mathematics-like prompts that can be answered directly. The main policy therefore retains an overly strong thinking prior on some math-like benchmarks, even though its cross-domain ordering is sensible.
New experiment. We continued CR-8B within a single category, removing benchmark identity as a routing cue:
The overall think rate falls from 100% to 60.4% at unchanged accuracy (92.5% self-routed versus 93.0% forced thinking). It rises from 16% on level-1 problems to 90% on level 5, and problems where direct answering already matches thinking are routed to thinking 45% of the time, compared with 85–93.5% for problems where thinking helps.
The two shuffle controls separate prompt-level routing from coarser explanations. The within-level shuffle keeps each level's think rate fixed and randomly reassigns which problems think; if routing only followed difficulty level, this would not lower accuracy. The training-time control permutes the counterfactual gaps across the prompts of each batch, so the overall push toward or away from thinking is unchanged but no longer matches the individual prompt; this run never leaves uniform thinking. The over-routing of the main model is therefore a correctable data-calibration issue, not a restriction to task-level routing.