acceptodds
Under review as a conference paper at ICLR 2027

When2Think: Learning When and How Much to Reason

Abstract

Large Reasoning Models (LRMs) often overthink easy problems and underthink hard ones, leading to inefficient computation allocation. Existing methods regulate generated computation or select between direct answering and explicit reasoning, but do not jointly control whether}to reason and how much computation to allocate within reasoning. We call the resulting difficulty-dependent loss in accuracy under computation reduction the efficiency tax. We propose When2Think, an RLVR-based post-training framework for instance-adaptive computation allocation. Its core mechanism, Instance-level Difficulty-Aware Control (IDAC), uses cached reference statistics of success and token cost to modulate a correctness-gated efficiency bonus based on generated token count. Importance sampling supports exploration of Think and NoThink, while Batch-Wise Standardization constructs standardized advantages for critic-free optimization. The framework requires neither a learned reward model nor a learned critic, and offline reference caching avoids online reference-model queries during policy updates. On AIME24, When2Think improves Pass@3 by 10.0 percentage points while reducing token usage by 27.9% relative to the backbone.

Primary area:
optimization
Keywords:
large reasoning models, language models, reinforcement fine tuning, hybrid reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PL3rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

When2Think is an RL post-training framework that learns whether to use an explicit reasoning block and how much computation to spend within it. Its central mechanism, Instance-level Difficulty-Aware Control (IDAC), uses cached reference-model success rates and response lengths to shape a correctness-gated efficiency bonus. It combines this with batch-wise advantage standardization and importance-sampled exploration of THINK/NOTHINK modes.

The main contribution is difficulty-conditioned reward shaping for computation allocation, supported by analyses separating mode selection from within-THINK length control. Importance sampling and advantage standardization build on existing methods. NOTHINK skips the dedicated reasoning block but can still contain substantial reasoning, and its reward does not directly penalize response length.

Validation covers mathematical reasoning benchmarks at 1.5B and 7B scales, difficulty-stratified evaluation on MATH-500, and cross-domain transfer on MMLU-Pro Stratified. The experiments compare accuracy and token usage, examine THINK/NOTHINK behavior, and ablate the main training components and reward coefficients.

Critical strengths
  • IDAC provides a simple, explicit mechanism for conditioning efficiency incentives on reference-estimated difficulty and cost. This addresses a consequential limitation of uniform compression: reducing unnecessary reasoning without equally discouraging useful computation on harder problems. It is the paper’s strongest methodological contribution.
Critical weaknesses
  • Limited evidence of improvement over the strongest relevant baseline. The headline gains are measured against R1-Distill, whereas DeepScaleR uses the same backbone and training dataset. On AIME24, DeepScaleR achieves higher Pass@3 with fewer tokens (58.0%, 8,473 versus 56.0%, 10,236); on AIME25, When2Think gains only 0.7 percentage points while using more tokens. These results leave the incremental benefit over conventional RL post-training unclear, weakening the central accuracy–efficiency claim.
  • The efficiency objective does not consistently track actual computation. Correct NOTHINK responses receive the full efficiency bonus regardless of length, although they can contain lengthy step-by-step reasoning. Moreover, mode-specific comparisons evaluate different model-selected subsets rather than forcing both modes on the same questions. Consequently, the paper neither guarantees nor cleanly isolates the computational benefit of NOTHINK, limiting support for its joint “whether and how much to reason” formulation.
Questions for authors

Please refer to the Weaknesses.

6 comments

We thank Reviewer PL3r for recognizing IDAC as our strongest methodological contribution and for asking us to distinguish checkpoint-level dominance from mechanism-level contribution.

W1: Baseline Competitiveness

We agree with the empirical comparison. On AIME24, DeepScaleR-Preview achieves higher Pass@3 with fewer tokens (58.0%58.0\% / 8,4738,473 vs. 56.0%56.0\% / 10,23610,236), and on AIME25, When2Think gains 0.70.7 pp (40.0%40.0\% vs. 39.3%39.3\%) while using more tokens (9,5499,549 vs. 8,0748,074) (Table 1, p. 7). We will state this trade-off directly in Section 6.1 and do not claim universal Pareto dominance.

Our question is distinct from finding a single dominant RL checkpoint:

Can a hybrid model learn an explicit, difficulty-conditioned computation policy that reduces redundant deliberation on easier problems without uniformly suppressing the depth required by harder instances?

Conventional accuracy-oriented RLVR does not specify how computation should scale across instances. As our reference-statistics analysis shows, substantial easy-long and hard-short populations exist, so length alone cannot proxy solvability.

DeepScaleR-PreviewWhen2Think
Primary objectiveScale RL reasoning capabilityLearn instance-adaptive computation allocation
RewardBinary correctness + formatCorrectness-gated IDAC reward (Section 4.2, Eqs. 5-6)
Computation controlGlobal 8K→16K→24K8\text{K}\to16\text{K}\to24\text{K} curriculum (~1,750 steps)Per-instance (αi,τi)(\alpha_i,\tau_i)-conditioned control (314 steps)
GranularityGlobal training stageIndividual instance and trajectory
Hybrid modeSingle deliberative mode (THINK)THINK/NOTHINK routing + within-THINK depth

DeepScaleR's own diagnostics show longer generations are not inherently productive: on AIME24, its incorrect responses are >3×3\times longer than its correct responses (20,34620,346 vs. 6,3956,395 tokens; DeepScaleR, Section 3.2, Table 1). When2Think addresses the complementary instance-level problem within a single fixed 16K stage (314 steps; App. A.1, Table 2) by encoding solvability (αi\alpha_i) and expected cost (τi\tau_i) into the reward.

While hybrid interfaces are now standard — e.g., Qwen3 integrates both modes with a user-specified budget (Qwen3 Tech Report, arXiv:2505.09388) — When2Think differs in that allocation is learned autonomously as an instance-conditioned policy, not manually tuned at inference.

This behavior appears across evaluations:

  • Monotonic mode scaling: On MATH-500, THINK ratio rises from ~0.20.2 at Level 1 to >0.70.7 at Level 5 (Section 6.2, Figure 4).
  • Easy-instance overthinking: At MATH-500 Level 1, tokens fall from 1,1991,199 (R1-Distill) and 1,2501,250 (DeepScaleR) to 619619, while Pass@1 improves to 95.8%95.8\% (vs. 92.1%92.1\%, 94.0%94.0\%) (App. C.3, Table 10).
  • Hard-instance deliberation: At MATH-500 Level 5, tokens fall by 33.3%33.3\% (6,830→4,5546,830 \to 4,554) while Pass@1 is preserved within overlapping std (71.0±2.3%71.0\pm2.3\% vs. 70.7±2.9%70.7\pm2.9\%) (Table 10).
  • Within-THINK control: Our THINK-only (IDAC+BWS) variant, labeled w/o IS in Table 1, achieves 57.3%57.3\% (10,04610,046 tokens) on AIME24 and 40.0%40.0\% (9,8469,846) on AIME25, closely matching DeepScaleR (58.0%58.0\%, 39.3%39.3\%) in a single 314-step stage (Table 1; Section 7.2).

In the revision we will distinguish: (1) unconditional checkpoint dominance over multi-stage curriculum models, which we do not claim; and (2) mechanism-level contribution, which lies in formulating and isolating instance-adaptive computation control.

W2: NOTHINK Definition and Adaptive Validity

We agree on both points: (1) NOTHINK does not mean zero reasoning, and (2) its bonus is length-invariant within its cap.

NOTHINK denotes direct answer-channel generation that bypasses the dedicated ⟨BOT⟩...⟨EOT⟩ deliberation block under a bounded budget (4,096 tokens vs. 16,384 for THINK; App. A.1, Table 2). This is substantive, not terminological: NoThinking (Ma et al., arXiv:2504.09858) shows bypassing the explicit thinking block can outperform explicit thinking at matched budgets because concise derivation can still occur in the answer channel.

For NOTHINK (λ=1\lambda=1), r=V(1+δ)−αir = \mathcal{V}(1+\delta)-\alpha_i (δ=0.05\delta=0.05) is length-independent within the budget. The reward therefore does not algebraically guarantee short NOTHINK responses. Brevity is an empirical property of the learned policy: bypassing the think block eliminates recursive self-reflection loops (e.g., "Wait, let me recheck..."; App. E.1-E.3), while the 4K cap prevents unbounded deliberation from spilling into the answer channel.

Illustrative NOTHINK Case (αi=0.811\alpha_i=0.811)V\mathcal{V}ri,kr_{i,k}
Easy, correct, 500 tokens (within 4K)111.05−0.811=+0.2391.05-0.811 = \mathbf{+0.239}
Easy, correct, 3,500 tokens (within 4K)111.05−0.811=+0.2391.05-0.811 = \mathbf{+0.239}
Easy, exceeds 4K (truncated, no valid \boxed{})000−0.811=−0.8110-0.811 = \mathbf{-0.811}

All token counts measure total generated tokens, including answer-channel tokens. As shown in App. C.1 (Figure 5, KDE), correct NOTHINK trajectories peak sharply at <1k<1\text{k} tokens, while THINK trajectories consume substantially more even when correct and exhibit a heavy tail when incorrect.

Benchmark (App. C.2, Tables 4-8)NOTHINK Pass@3Mean NOTHINK TokensMean THINK TokensNOTHINK Rate
GSM-Plus93.0±0.5%93.0\pm0.5\%538±15538\pm151,401±271,401\pm2740.3±0.3%40.3\pm0.3\%
Minerva78.8±4.3%78.8\pm4.3\%811±192811\pm1923,282±823,282\pm829.0±0.6%9.0\pm0.6\%
OlympiadBench90.5±2.1%90.5\pm2.1\%1,086±931,086\pm936,635±576,635\pm5714.7±0.5%14.7\pm0.5\%
AIME24Not selectedN/A10,236±43810,236\pm4380.0±0.0%0.0\pm0.0\%
AIME25Not selectedN/A9,549±7199,549\pm7190.0±0.0%0.0\pm0.0\%

Selected NOTHINK is 2.6×2.6\times to 6.1×6.1\times shorter than THINK. We will move the KDE analysis (Figure 5) and this summary into the main text, framing NOTHINK brevity as an empirical outcome of bypassing the think block, not as a per-token length penalty.

Thank you for the clarification. Could you please clarify what empirical advantage IDAC offers over DeepScaleR in terms of accuracy and token usage? I do not see multi-stage training itself as a limitation, so a comparison of the resulting performance would help me better understand IDAC’s contribution. Are there specific metrics or comparisons that demonstrate IDAC’s advantages?

Reviewer_zwLZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes When2Think, an RLVR framework for instance-adaptive reasoning. It estimates each problem’s difficulty based on a success rate from a reference model and reasoning cost from the reference policy’s average length. These statistics drive a correctness-gated efficiency bonus that decays with generation length, and the decay is faster on easier problems. Direct answers receive the full bonus. On AIME24, it improves Pass@3 from 46.0% to 56.0% while cutting token usage by about 28%.

Critical strengths
  • Clear and intuitive method. The paper uses the reference model’s success rate and average response length to characterize instance difficulty and reasoning cost, then modulates the efficiency reward accordingly. The mechanism is easy to follow. The framework requires neither an additional reward model nor a critic, and the reference statistics are cached offline, keeping implementation simple and training overhead low.
  • Comprehensive experiments. The method is evaluated on multiple mathematical benchmarks spanning a wide range of difficulty. The paper also examines cross-domain transfer (MMLU-Pro) and different model scales (1.5B/7B).
  • Thorough analysis. The difficulty-stratified evaluation on MATH-500 clearly illustrates the intended allocation behavior: less reasoning on easy problems and more on hard ones. The ablation studies separately demonstrate the contributions of IDAC, BWS, and IS.
Critical weaknesses
  • The novelty and contribution of this paper are limited. It mainly combines AdaptThink-style THINK/NOTHINK routing with existing difficulty-aware length control methods such as DAST and LASER-D. The main new element appears to be the specific exponential reward formulation, whose advantage over prior reward designs is not clearly demonstrated.
  • The experimental results are not sufficiently convincing. In particular, When2Think does not consistently outperform strong baselines such as DeepScaleR-Preview, and the w/o IS variant achieves comparable or even better performance than the full model on several benchmarks, questioning the necessity of the IS component.
Questions for authors

N/A

3 comments

We thank Reviewer zwLZ for recognizing IDAC's intuitive structure, low overhead, and breadth of evaluation. We agree the original submission should have made the contribution boundary and the roles of individual components more explicit.

W1: Methodological Novelty

When2Think studies:

Given a hybrid THINK/NOTHINK interface, how can computation depth within THINK be adapted to empirical solvability and expected computation cost of each instance?

1. Solvability and cost are non-equivalent

Let si=1K∑kV(xi,oi,k′)s_i = \frac{1}{K}\sum_k \mathcal{V}(x_i, o'_{i,k}) be the cached reference success rate and αi=γsi+ϵ\alpha_i = \gamma s_i + \epsilon (γ=0.9\gamma=0.9). Partitioning by sis_i (Easy ≥2/3\ge 2/3, Hard <1/3<1/3) and by reference length τi\tau_i (short vs. long at 4,096 tokens), 53.6% of easy instances exceed 4,096 tokens, and 32.3% of all long instances are easy. Length alone cannot substitute for solvability.

Unlike DAST, which collapses solvability and correct-response length into a single Token Length Budget, IDAC retains two separately interpretable signals: αi\alpha_i for scaled solvability and τi\tau_i for reference computation cost (Section 4.1, Eq. 4). A single target length conflates easy-long (possibly redundant overthinking) with hard-long (possibly necessary deliberation).

2. Continuous, instance-conditioned reward

For THINK trajectory oi,ko_{i,k} (Section 4.2, Eqs. 5-6):

\exp\left( -\frac{\mathcal{T}(o_{i,k})\alpha_i}{\tau_i} \right), \quad r_{i,k} = \mathcal{V}(x_i,o_{i,k}) \left(1+\lambda_{i,k}\delta\right) -\alpha_i.$$ with $\lambda_{i,k}=\tilde{\lambda}_{i,k}$ for THINK ($\lambda=1$ for NOTHINK) and $\delta=0.05$ (App. A.1, Table 2). Correctness $\mathcal{V}\in\\{0,1\\}$ sets the primary gap, $-\alpha_i$ sets an instance-dependent baseline, and $\lambda_{i,k}\delta$ gives a bounded preference among correct trajectories. The ratio $\alpha_i/\tau_i$ governs decay: easy/low-cost instances get stronger length pressure, hard/high-cost instances weaker pressure. Comparison against LASER family (Liu et al., 2026): static **LASER** (fixed $L_T$), **LASER-D** (dynamic $L_A$ across three buckets), **LASER-DE** (`Laser-DE-L4096` in Table 1, adds $+\alpha$ bonus to incorrect responses exceeding $L_A$): | **Case** | **Description** | **LASER** | **LASER-D** | **LASER-DE** | **IDAC (Ours)** | | :--- | :--- | :--- | :--- | :--- | :--- | | **A** | Easy, short, correct | 1.5 | 1.5 | 1.5 | **0.216** | | **B** | Easy, long, correct | 1.0 | 1.0 | 1.0 | **0.196** | | **C** | Easy, long, incorrect | -0.5 | -0.5 | 0.0 | **-0.811** | | **D** | Hard, long, correct | 1.0 | 1.5 | 1.5 | **0.955** | | **E** | Hard, long, incorrect | -0.5 | -0.5 | -0.5 (or 0.0) | **-0.091** | Magnitudes are not directly comparable across columns due to different baselines; the comparison is **within-method discrimination**. In LASER-DE, $L_A$ is small on easy problems (\~1k-2k) and near the context limit on hard problems (\~16k), so an easy-long failure exceeding $L_{A,\text{easy}}$ gets the $+0.5$ bonus, offsetting $-0.5$ to $0.0$. - **Within-correct:** LASER treats easy-long (B) and hard-long (D) identically ($1.0$). LASER-D/DE separate them via discrete steps over three buckets, while IDAC separates them continuously via $(\alpha_i,\tau_i)$ decay. - **Within-incorrect:** LASER/LASER-D assign flat $-0.5$ to all incorrect responses. LASER-DE positively rewards long incorrect trajectories ($L(y)>L_A$) for exploration. IDAC distinguishes hard-long success ($0.955$) from hard-long failure ($-0.091$) and heavily penalizes easy-long failure ($-0.811$) via $-\alpha_i$, without positively rewarding long incorrect exploration. Exact representative $(s_i,\alpha_i,\tau_i)$ values will be reported in revised Appendix. This establishes a measurable difference in discrimination, not universal superiority under matched retraining. ### 3. BWS and THINK-only isolate the mechanism Per-prompt normalization (e.g., GRPO, used in LASER family) removes prompt-specific offsets such as $-\alpha_i$ (App. D.1, Figure 8). BWS (Section 4.3, Eqs. 7-8) computes advantages across instances for a fixed rollout index $k$, applying one shared affine transformation per batch, thereby preserving cross-instance ordering that IDAC encodes. Our claim is narrow: BWS retains a difficulty signal IDAC encodes; it does not create information absent from the reward. The variant labeled `w/o IS (IDAC+BWS)` in Table 1 samples **only THINK** trajectories. We will rename it **`THINK-only (IDAC+BWS)`** throughout. With hybrid exploration removed, it isolates depth control after THINK is selected. Importance Sampling (IS), adopted from AdaptThink (Section 3), is the exploration protocol to expose both modes — neither claimed as algorithmic novelty nor intended to improve THINK-only accuracy.

W2: Empirical Effectiveness

We agree with both observations: (1) When2Think does not uniformly outperform DeepScaleR-Preview at inference, and (2) IS does not consistently improve accuracy over w/o IS. Neither is our intended claim.

Our claim is that IDAC+BWS learns difficulty-sensitive computation control within THINK, while hybrid exploration makes an additional bounded NOTHINK regime available (Section 6.3, Section 7.2).

1. Full hybrid vs. THINK-only (IDAC+BWS)

Both use the same backbone (R1-Distill-Qwen-1.5B), DeepScaleR training set, IDAC+BWS objective, decoding, and evaluation protocol (Section 5). The only training difference is whether balanced hybrid-mode exploration via IS is enabled.

Benchmark (Table 1)Full When2ThinkTHINK-only (IDAC+BWS)Full minus THINK-only
GSM-Plus (Pass@3 / Tokens)85.7±0.3%85.7\pm0.3\% / 1,052±17.91,052\pm17.986.8±0.1%86.8\pm0.1\% / 1,652±13.21,652\pm13.2−1.1 pp-1.1\text{ pp}, −36.3%-36.3\% tokens
AIME24 (Pass@3 / Tokens)56.0±2.8%56.0\pm2.8\% / 10,236±438.010,236\pm438.057.3±8.3%57.3\pm8.3\% / 10,046±487.910,046\pm487.9−1.3 pp-1.3\text{ pp}, +1.9%+1.9\% tokens
AIME25 (Pass@3 / Tokens)40.0±0.0%40.0\pm0.0\% / 9,549±719.49,549\pm719.440.0±3.3%40.0\pm3.3\% / 9,846±269.29,846\pm269.20.0 pp0.0\text{ pp}, −3.0%-3.0\% tokens

On GSM-Plus, Full occupies a substantially lower-computation operating point (−36.3%-36.3\% tokens). On AIME24/25, Full selects THINK for every instance (App. C.2, Tables 7-8), so these benchmarks characterize how the two training procedures affect within-THINK behavior, not active switching.

THINK-only is therefore the cleaner evidence for depth control. Full adds access to a lower-budget mode on suitable instances — it does not claim higher hard-task accuracy (Section 7.2).

2. Conditional mode behavior on GSM-Plus

Selected Mode (App. C.2, Table 4)Pass@3Mean Total TokensSelection Rate
NOTHINK93.0±0.5%93.0\pm0.5\%538±15538\pm1540.3±0.3%40.3\pm0.3\%
THINK80.8±0.3%80.8\pm0.3\%1,401±271,401\pm2759.7±0.3%59.7\pm0.3\%

These are conditional behavioral statistics on model-selected subsets, not a same-instance causal comparison. They explain how the aggregate GSM-Plus reduction (1,652→1,0521,652 \to 1,052 tokens) is achieved relative to THINK-only, rather than claiming forcing NOTHINK would outperform THINK on the same questions.

3. DeepScaleR: inference vs. post-training structure

We do not claim inference-time Pareto dominance over DeepScaleR-Preview.

Model (Tables 1 & 10)AIME24 (Pass@3 / Tokens)AIME25 (Pass@3 / Tokens)GSM-Plus (Pass@3 / Tokens)MATH-500 L1 (Pass@1 / Tokens)
DeepScaleR-Preview58.0%58.0\% / 8,4738,47339.3%39.3\% / 8,0748,07485.4%85.4\% / 1,3581,35894.0%94.0\% / 1,2501,250
THINK-only (IDAC+BWS)57.3%57.3\% / 10,04610,04640.0%40.0\% / 9,8469,84686.8%86.8\% / 1,6521,652-
Full When2Think56.0%56.0\% / 10,23610,23640.0%40.0\% / 9,5499,54985.7%85.7\% / 1,0521,05295.8%95.8\% / 619619

The distinction lies in post-training procedure (App. A, Tables 2-3):

DimensionDeepScaleR-1.5B-PreviewWhen2Think 1.5B
Training schedule8K→16K→24K8\text{K}\to16\text{K}\to24\text{K} (3 stages), ~1,750 stepsSingle fixed 16K stage, 314 steps, 1 epoch
Reported policy-optimization cost~3,800 A100 GPU-hours~70 H100 GPU-hours
Control granularityGlobal training stageIndividual instance / trajectory

Because accelerator-hours are not hardware-normalized, we do not claim a direct multiplicative reduction. The cleaner comparison is structural: one fixed-context stage of 314 steps vs. three iterative stages totaling ~1,750 steps. When2Think additionally requires a one-time offline reference-sampling pass to cache (αi,τi)(\alpha_i,\tau_i) (Section 4.1), separate from inner-loop time and to be reported explicitly in App. A.2. Because DeepScaleR regulates length globally rather than per instance, it still overthinks easy instances (e.g., 1,2501,250 tokens on MATH-500 Level 1 vs. 619619 for When2Think; Table 10).

DeepScaleR and When2Think address complementary levels — staged global curriculum vs. instance-level reward shaping — and DeepScaleR's own diagnostic (incorrect AIME24 outputs average 20,34620,346 tokens vs. 6,3956,395 for correct ones) supports our shared motivation that longer generation alone does not identify useful computation.

Thank you for your initial review. As the discussion period is approaching its deadline, we would appreciate it if you could let us know whether our rebuttal addresses your concerns. We would be happy to provide any further clarification if needed. Thank you again for your time and consideration.

Reviewer_P238OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

When2Think learns whether to use explicit thinking and how long to reason through RL with verifiable rewards. IDAC uses cached reference success rates and response lengths to reward correct, concise answers while relaxing length pressure on harder questions. BWS standardizes rewards across questions to construct advantages, while IS balances THINK/NOTHINK exploration and corrects the sampling distribution during training. Evaluations cover mathematical reasoning and cross-domain transfer with 1.5B and 7B models. On AIME24, the 1.5B model improves Pass@3 from 46.0% to 56.0% over R1-Distill-Qwen-1.5B while reducing average response length by approximately 28%.

Critical strengths
  1. The paper addresses a meaningful practical problem: reducing unnecessary reasoning while preserving performance on difficult questions. It is clearly organized and easy to follow, with explicit descriptions of the reward and training procedure.

  2. The empirical evaluation is broad, covering multiple math benchmarks, two model scales, and cross-domain transfer. Difficulty-stratified results, component ablations, and reward-coefficient sensitivity analyses provide useful evidence beyond aggregate accuracy and token savings.

Critical weaknesses

W1. The specific benefit of combining IDAC with BWS is insufficiently justified beyond generic reward shaping and normalization. Preserving cross-question reward differences does not establish the necessity of BWS, and its sensitivity to batch composition remains unclear.

W2. The evaluation leaves key claims unresolved. Mode-conditioned results do not isolate the benefit of adaptive switching, while cross-domain savings often occur with THINK rates near 100%. AIME Pass@3 paired with per-response token costs does not establish single-answer efficiency, and relegating several benchmark comparisons to the appendix makes overall performance harder to assess.

Questions for authors

Q1. Using the same trained model, what happens when questions assigned to THINK are forced to NOTHINK, and vice versa? Please compare autonomous selection with always-THINK, always-NOTHINK, and random selection at a matched THINK rate, reporting accuracy and token cost, including out-of-domain results.

Q2. Can IDAC and a difficulty-independent length reward each be evaluated with BWS and per-prompt centering under matched settings? How often do correct responses receive negative BWS advantages, and how sensitive are updates and performance to batch size and difficulty composition?

Q3. Please report AIME24/25 Pass@1 for the method and baselines with uncertainty estimates, and clarify the total sampling cost for Pass@3. Please also consolidate Minerva, OlympiadBench, and MATH-500 comparisons, including aggregate MATH-500 accuracy and token usage, in the main table.

8 comments

We thank Reviewer P238 for recognizing the practical importance of the problem, the breadth of the evaluation, and the value of the difficulty-stratified and component-level analyses.

W1 & Q2: Methodological Justification

We agree that stating BWS "preserves cross-question reward differences" does not by itself establish necessity. Below we separate what the algebra shows, what independent evidence from SPO (Xu & Ding, arXiv:2509.13232) supports, and what our advantage-level and performance-level evidence shows.

SPO is a concurrent work on GRPO-style advantage construction that addresses the same issue: a per-group baseline can be noisy and loses all signal in degenerate groups. We cite it as independent evidence for the general principle that a prompt-specific reference should be estimated outside the current small group before shared normalization.

1. What Per-Prompt Centering Removes from IDAC

Under IDAC (Section 4.2, Eqs. 5–6):

ri,k=V(xi,oi,k)(1+λi,kδ)−αir_{i,k} = \mathcal{V}(x_i, o_{i,k})(1 + \lambda_{i,k}\delta) - \alpha_i

λi,k={exp⁡(−T(oi,k)αiτi)if Think1if NoThink\lambda_{i,k} = \begin{cases} \exp\left(-\dfrac{\mathcal{T}(o_{i,k})\alpha_i}{\tau_i}\right) & \text{if Think} \\\\ 1 & \text{if NoThink} \end{cases}

Here αi\alpha_i is the cached reference success rate, τi\tau_i is the cached reference mean length, T(oi,k)\mathcal{T}(o_{i,k}) is the rollout length, and δ=0.05\delta=0.05 bounds the efficiency bonus in [0,δ][0,\delta]. Reference statistics are computed once per epoch and cached, so they are fixed within each epoch and refreshed per epoch.

λ\lambda encodes difficulty-modulated length pressure for THINK: hard (αi\alpha_i low) →\to λ\lambda close to 11, full bonus, relaxed pressure; easy (αi\alpha_i high) →\to λ\lambda decays quickly with length, tightened pressure. For NoThink, λ=1\lambda=1 by definition.

Per-prompt centering cancels −αi-\alpha_i exactly:

ri,k−rˉi=Vi,k(1+λi,kδ)−1K∑jVi,j(1+λi,jδ)r_{i,k} - \bar{r}_i = \mathcal{V}_{i,k}(1+\lambda_{i,k}\delta) - \frac{1}{K}\sum_j \mathcal{V}_{i,j}(1+\lambda_{i,j}\delta)

What is removed is only the additive cross-instance level −αi-\alpha_i. Reference dependence is not removed entirely, because λ\lambda still contains αi,τi\alpha_i,\tau_i. What is lost is the ordering hard-correct > easy-correct encoded by the additive term; what is preserved is within-prompt correctness and λ\lambda-mediated length differences.

Per-prompt centering algebraically removes IDAC's additive cross-instance level (−αi-\alpha_i), while preserving λ\lambda-mediated differences. BWS is our critic-free construction for retaining the additive level as well. This algebra alone does not prove BWS is necessary for final performance.

2. Independent Evidence from SPO

SPO addresses a broader concern in GRPO-style advantage construction: an on-the-fly per-group baseline can be noisy and lose all signal when a group is degenerate. It identifies three sources of instability — noisy group mean, noisy group std, and complete loss in all-correct / all-incorrect groups — and reports that the fraction of samples in degenerate groups rises from ~60% to >80% during GRPO training.

SPO replaces the per-group baseline with a persistent, KL-adaptive Bayesian tracker:

A(x,y)=r(x,y)−v^−1(x),A~(x,y)=A(x,y)−μBσBA(x,y) = r(x,y) - \hat{v}_{-1}(x), \quad \tilde{A}(x,y) = \frac{A(x,y) - \mu_{\mathcal{B}}}{\sigma_{\mathcal{B}}}

where v^−1(x)\hat{v}_{-1}(x) is the pre-update estimate of the prompt's success probability. Because it is computed before the current action, SPO argues it preserves unbiasedness. The order matters: subtract a prompt-specific reference obtained outside the current small group, then apply a shared normalization.

SPO and BWS are not equivalent:

SPOIDAC + BWS
Prompt referencev^−1(x)\hat{v}_{-1}(x), current-policy value estimateαi\alpha_i, cached reference-policy solvability
ChangeUpdated via KL-adaptive trackerFixed within epoch, refreshed per epoch
UseSubtracted to form advantageAdditive baseline inside reward
NormalizationWhole batch, one response per promptRollout-index-wise across instances

SPO therefore supports only the general principle that a prompt-specific reference should be estimated outside the current small group before shared normalization. It does not evaluate IDAC or BWS, and does not replace the IDAC ×\times normalization factorial.

It also does not support replacing fixed αi\alpha_i with a raw online KK-sample mean. SPO's tracker is a smoothed history, not the current group's mean. If we set α^i=1K∑kVi,k\hat{\alpha}_i = \frac{1}{K}\sum_k \mathcal{V}_{i,k} from current rollouts, then πref=πθ\pi_{ref} = \pi_{\theta} by definition: the online policy becomes its own reference, with no external anchor, and KL(πθ∣∣πθ)=0KL(\pi_{\theta}||\pi_{\theta})=0 for the KL term. This is fatal for the additive term because its scale is 1.01.0 — with K=16,p=0.5K=16, p=0.5, std(α^)≈0.125\text{std}(\hat{\alpha}) \approx 0.125, larger than the entire bonus δ=0.05\delta=0.05 — while it is bounded for λ\lambda because its effect is scaled by δ\delta (a 0.0750.075 move in λ\lambda moves the bonus by ≈0.0038\approx 0.0038). We therefore keep a fixed reference for the additive level and leave an EMA-stabilized online λ\lambda as future work.

3. Advantage-Level and Performance-Level Evidence

The advantage-level questions depend only on the reward distribution at a fixed policy. The Monte Carlo landscape in Appendix D.1 (Fig. 8) resamples batches from cached (αi,τi)(\alpha_i,\tau_i). Because αi=0.9si+ϵ\alpha_i = 0.9 s_i + \epsilon (Section 4.1, Eq. 4), where sis_i is the reference success rate:

rcorrect=1−αi+δλi,k,rincorrect=−αir^{\text{correct}} = 1 - \alpha_i + \delta\lambda_{i,k}, \qquad r^{\text{incorrect}} = -\alpha_i

So rcorrect=0.10+δλr^{\text{correct}} = 0.10 + \delta\lambda when αi→0.9\alpha_i \to 0.9 (easy) and 1.0+δλ1.0 + \delta\lambda when αi→0\alpha_i \to 0 (hard), while the lowest tail comes from incorrect responses on easy prompts (r≈−0.90r \approx -0.90).

Because BWS estimates μk,σk\mu_k,\sigma_k from the current batch, its advantages are structurally dependent on batch size and difficulty composition. We will state this limitation explicitly. We do not claim a monotonic direction for correct-negative frequency across MM, composition, and δ\delta without completed simulation, because μk\mu_k and σk\sigma_k move together. Table 13 will be added only if the Monte Carlo resampling is completed with numerical results, and it conditions on fixed αi\alpha_i — if αi\alpha_i were also online, μk,σk\mu_k,\sigma_k and α^i\hat{\alpha}_i would move together, compounding batch dependence.

Performance-level evidence comes from existing ablations under the submitted configuration, not from the full factorial. Removing BWS (App. D.2, Fig. 9) destabilizes training: Think Ratio falls from ~0.9 to ~0.6 without BWS, with less stable length and accuracy. Removing IDAC while keeping BWS (App. D.3, Fig. 10) consistently underperforms on GSM8K and AIME24 with erratic Think Ratio. Together with the algebra, these results are consistent with BWS helping retain the reward structure introduced by IDAC, but the IDAC ×\times normalization interaction itself remains undetermined without the factorial.

W2, Q1, & Q3 — Evaluation Validity and Main-Text Consolidation

1. Mode-Conditioned Results Are Behavioral, Not Causal

We agree the mode-conditioned results in Appendix C.2 (Tables 4–8) compare subsets selected by the learned policy, not THINK vs. NOTHINK counterfactually on the same questions. Their purpose is behavioral characterization.

For example, on GSM-Plus the policy-selected NOTHINK subset averages 538 tokens and the THINK subset averages 1,401 tokens, but this is not a same-question causal comparison. We will state this bound explicitly in Section 6.2 and Appendix C.2.

2. What NOTHINK Means

NOTHINK (Ma et al., arXiv:2504.09858) does not mean zero reasoning or explanation-free output. It denotes answer-channel generation without a dedicated ⟨BOT⟩...⟨EOT⟩ deliberation block, under a bounded generation budget.

Operationally, it is bounded by a 4,096-token maximum, compared with 16,384 for THINK (Appendix A.1, Table 2). For a valid NOTHINK trajectory (λNOTHINK=1\lambda^{\text{NOTHINK}}=1), rNOTHINK=V(1+δ)−αir^{\text{NOTHINK}} = \mathcal{V}(1+\delta)-\alpha_i is length-invariant within the budget. Efficiency is evaluated using total generated tokens, including all answer-channel tokens (Section 5.2; Appendix C.1, Fig. 5).

Benchmark (App. C.2)NOTHINK Pass@3Mean NOTHINK TokensMean THINK TokensNOTHINK Rate
GSM-Plus93.0±0.5%93.0\pm0.5\%538±15538\pm151,401±271,401\pm2740.3±0.3%40.3\pm0.3\%
Minerva78.8±4.3%78.8\pm4.3\%811±192811\pm1923,282±823,282\pm829.0±0.6%9.0\pm0.6\%
OlympiadBench90.5±2.1%90.5\pm2.1\%1,086±931,086\pm936,635±576,635\pm5714.7±0.5%14.7\pm0.5\%
AIME24 / AIME25Not selectedN/A10,236±43810,236\pm438 / 9,549±7199,549\pm7190.0±0.0%0.0\pm0.0\%

Selected NOTHINK responses are 2.6×2.6\times to 6.1×6.1\times shorter than THINK responses. We will replace ambiguous phrasing such as "no reasoning" with: "NOTHINK denotes answer-channel generation without a dedicated explicit deliberation block under a bounded 4K generation budget," and move the 4K/16K limits and token-distribution summary into the main text.

3. Forced Mode: Asymmetry and What Forced NOTHINK Tests

Forced NOTHINK is one intervention: inserting ⟨EOT⟩\langle \text{EOT} \rangle immediately after ⟨BOT⟩\langle \text{BOT} \rangle.

There is no symmetric Forced THINK: suppressing ⟨EOT⟩\langle \text{EOT} \rangle, imposing a minimum length, or injecting a prefix would change the depth decision IDAC is meant to learn. A random matched-rate baseline would be random Forced-NOTHINK mixed with autonomous behavior, not a pure random selector.

Forced NOTHINK is a stress test, not the deployment policy. If autonomous performance is preserved where Forced NOTHINK degrades, that is consistent with the selector protecting deliberation-needing questions; if there is no difference, the selector adds little. Results will be reported in the revised manuscript with benchmark-level accuracy, total generated tokens, and uncertainty.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback.

  • P238 acknowledged the practical relevance of the problem, the breadth of evaluation, and the clarity of the reward and training procedure.
  • zwLZ highlighted IDAC's intuitive design, its implementation without an additional reward model or critic, and the use of cached offline statistics.
  • PL3r identified IDAC as the paper's strongest methodological contribution.

The reviewers differ in their assessments of novelty and empirical strength. This note clarifies the motivation, methodological contribution, and scope of our claims.

1. Motivation: Reasoning efficiency requires more than reducing tokens

Reasoning models face two inefficiencies:

  • Overthinking: Spending unnecessary computation on easily solvable instances.
  • Underthinking: Reducing computation on difficult instances at the expense of accuracy.

This creates an efficiency tax: reducing token usage can also reduce accuracy on difficult problems. For example, on AIME24, AdaptThink uses 5,806 tokens compared with DeepScaleR's 8,473, but achieves 44.7% rather than 58.0% Pass@3.

Our objective is therefore to allocate computation according to instance difficulty, rather than uniformly reducing reasoning length.

2. Baselines: Why DeepScaleR and efficiency-oriented methods

The other baselines — AdaptThink, Thinkless-RL, LC-R1, ThinkPrune, LASER-DE — are efficiency-oriented. They aim to reduce tokens, often at some accuracy cost.

DeepScaleR aims for the opposite. It uses the same backbone and training data as When2Think-1.5B, but its objective is to maximize accuracy, not to save tokens.

So the two comparisons answer different questions:

  • Against efficiency-oriented baselines: on easy tasks, how much accuracy is gained for a similar token budget?
  • Against DeepScaleR: on hard tasks, how much accuracy is preserved relative to the accuracy ceiling?

MATH-500 mixes difficulties, so the table below shows overall positioning.

ModelPass@1Tokens per response
AdaptThink81.4%213.1
When2Think (Hybrid)84.6%236.1
DeepScaleR87.8%346.6
When2Think - THINK-only (IDAC+BWS)88.0%313.8

THINK-only is a separately trained configuration to isolate IDAC+BWS within-THINK control. It is not Hybrid inference with forced THINK.

3. Core contribution: IDAC

IDAC is the central methodological contribution. It combines the two reference statistics to construct instance-conditioned, correctness-aware incentives for reasoning trajectories.

Its objective is to apply stronger length pressure to easily solvable instances and weaker pressure to harder ones. Unlike a uniform token budget or length penalty, IDAC conditions the computation incentive on both solvability and reference computation cost.

BWS preserves cross-instance reward-level information during advantage construction. IS is used as a balanced exploration setup for THINK and NOTHINK; it is not proposed as a novel contribution or an independent source of accuracy gains.

When2Think considers two computation decisions:

  • Whether to think: Select between explicit reasoning (THINK) and direct answering (NOTHINK).
  • How deeply to think: Regulate computation within explicit reasoning.

IDAC addresses the second decision. The hybrid reasoning setting additionally allows the model to select between the two modes. We evaluate these aspects using two separately trained configurations: THINK-only, which retains explicit reasoning, and a hybrid configuration that supports both modes.

4. Empirical evidence

The results provide evidence for within-THINK computation control and examine its integration into hybrid reasoning.

MATH-500:

  • THINK-only achieves 88.0% Pass@1 with 313.8 tokens, compared with DeepScaleR's 87.8% with 346.6 tokens. This corresponds to 9.5% fewer tokens.
  • The hybrid configuration achieves 84.6% Pass@1 with 236.1 tokens, using 31.9% fewer tokens than DeepScaleR, with a 3.2 percentage-point accuracy difference.
  • At Pass@3, the hybrid configuration achieves 93.3% with 6,944 total tokens, compared with DeepScaleR's 94.2% with 8,514 tokens. THINK-only achieves 94.3% with 8,725.6 tokens.

GSM-Plus:

  • The hybrid configuration achieves 85.7% Pass@3 with 1,052 tokens, compared with DeepScaleR's 85.4% with 1,358 tokens.

These results do not establish universal Pareto dominance. On AIME24, DeepScaleR achieves 58.0% Pass@3 with 8,473 tokens, compared with 56.0% with 10,236 tokens for the hybrid configuration.

In summary, IDAC is the central contribution: it translates offline estimates of instance solvability and computation cost into instance-conditioned learning incentives for reasoning trajectories. The experiments examine its effect on reasoning depth and its integration into a hybrid reasoning framework. We ask the AC to assess this mechanism-level contribution in light of the reported trade-offs and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.