acceptodds
Under review as a conference paper at ICLR 2027

Credit Where Reasoning is Due: Learning When to Reason via Treatment-Effect-Aware Credit Assignment

Abstract

Reinforcement learning has made language models increasingly capable of reasoning, yet when to reason remains largely unaddressed. We show that standard group-relative RL cannot learn this decision when every rollout in a group reasons, as happens on-policy: the per-input effect of reasoning is absent from the policy gradient, regardless of the outcome reward. Forcing both modes into each group makes this counterfactual identifiable, but is not sufficient: the resulting signal must credit reasoning for its effect rather than for the output's quality, and it must remain reliable under noisy per-input comparisons. We address both by formulating selective reasoning as a treatment-effect learning problem and propose Treatment-Effect-Aware Credit Assignment (TECA), which combines forced 50/50 mode mixing with importance correction, a reward that scores both modes on the same scale, and a persistent empirical-Bayes estimate of whether reasoning beats direct generation on each input. That estimate gates reasoning-token credit and sets the target for the mode decision. On machine translation with Qwen3-8B, where reasoning barely moves mean quality yet widens its distribution, TECA matches or beats always-reasoning baselines while reasoning on selective inputs, and significantly outperforms the same objective without the gate. Learning when to reason therefore requires explicit treatment-effect credit, not only outcome-based RL or exposure to both modes.

Primary area:
reinforcement learning
Keywords:
GRPO, Machine Translation, Causal Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_d6A9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when language models should reason rather than always relying on reasoning. The authors propose TECA, a treatment-effect-aware credit assignment method that uses mixed think/no-think sampling and empirical-Bayes estimates to guide reasoning decisions. Experiments on machine translation demonstrate competitive translation quality with substantially reduced reasoning-token usage.

Critical strengths

Interesting motivation: The paper addresses an important problem of when reasoning is beneficial, rather than assuming that reasoning always improves performance.

Well-motivated method: The proposed TECA framework provides a clear analysis of the limitations of standard GRPO and introduces treatment-effect-aware credit assignment to address them.

Critical weaknesses

The experiments are limited to small models, most reasoning failures are caused by reaching the token budget limit, and nearly all invalid reasoning outputs result from hitting the token budget limit. This suggests that the observed degradation may partly reflect insufficient token budgets rather than an inherent drawback of reasoning, raising concerns about generalizability to larger models.

Questions for authors

Does this phenomenon persist in larger models, such as those from OpenAI, Anthropic, or DeepSeek, when comparing thinking and non-thinking modes with sufficiently large token budgets?

5 comments
AuthorsOpenReview ↗

Response to reviewer d6A9

Thank you for the review.

1. Does the cost of reasoning come from the token budget?

TECA does not assume that reasoning hurts on average. It assumes only that reasoning helps on some inputs and not on others, and it learns which. That remains true when the token budget is large enough that reasoning almost never runs out.

We repeated the Section 2.2 measurement with a 3,200-token budget for reasoning (same 1,350 sources, 12 samples per mode, base Qwen3-8B). The table gives the share of reasoning samples that produce no translation, and reasoning's effect on COMET-22 relative to direct translation, with failed samples scored 0 as in the paper.

Tier1,280 tokens: failedEffect2,048 tokens: failedEffect3,200 tokens: failedEffect
Figurative17.7%−0.0703.1%+0.0050.5%+0.018
OpenSubtitles8.1%−0.0501.6%−0.0090.1%−0.001
FLORES18.2%−0.0864.7%−0.0030.9%+0.017

The 1,280 column is the paper's (training cap). The 2,048 column, our evaluation cap, takes the 3,200-token samples and counts those whose reasoning was still open at 2,048 tokens as failures.

Two things hold at 3,200 tokens, where under 1% of reasoning samples fail.

  • Reasoning's average gain is small. It improves COMET-22 by at most 0.018, and on OpenSubtitles it gives no gain at all.
  • Reasoning is the better mode on only about half of the sources: 58% of figurative sources, 42% of OpenSubtitles sources and 54% of FLORES sources. Which sources those are is stable across draws (split-half reliability 0.82).

A larger budget removes the failures, but it does not make reasoning the better choice everywhere. Always reasoning would be the worse mode on roughly half of the inputs, and never reasoning would give up the gains on the other half. TECA decides per input, which is useful at any budget. We will add this table to Appendix I.

2. Does it hold for larger models?

Yes. Reasoning does not beat direct translation in any of these models:

Model (XCOMET-XL ×100)With reasoningWithout reasoning
Qwen3-32B7475
Claude-4-Opus80.180.4
DeepSeek-R174.978.7
Gemini-2.5-Flash78.679.9
Command-A-Reasoning (111B)77.378.3

Qwen3-32B is from our Table 3. The other four are from Rajaee et al. (2026) on WMT24++.

The budget is ample for Qwen3-32B: 98.5% of its outputs are valid, and direct translation also leads on COMET-22 and chrF++ (Table 3). Rajaee et al., discussed in Appendix A, test exactly the kinds of systems you mention. Zebaze et al. (2025) find the same for Qwen3 models up to 32B with a 3,500-token thinking budget. Across families and scales, reasoning's average effect on translation is near zero or negative even with generous budgets. TECA is built for this case: no fixed always-reason or never-reason policy is best, and the useful decision is made per input.

We cannot train TECA on closed models. It forces the first decision token into each mode and updates the model's weights, and closed APIs allow neither. For them the question can only be answered by evaluation, which the studies above provide.

We thank the Reviewer for their valuable time and efforts invested in our work, we believe that suggestions made will strengthen our work. We hope our rebuttal has clarified your concerns; please let us know if any questions remain, we would be happy to provide further clarification. If the rebuttal has resolved your concerns, we would greatly appreciate your reconsideration of the overall rating in light of the additional information provided.

Regards,

Authors

Dear Reviewer,

As the rebuttal period is nearing its end, we would be happy to provide any further clarifications that may help address your remaining concerns.

If our rebuttal has resolved your concerns, we would sincerely appreciate your reconsideration of the overall rating in light of the additional information provided.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_F9ZROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TECA learns whether to reason by comparing think and no-think outputs for the same input. A Bayesian win-rate estimate controls reasoning-token credit and provides a target for the mode decision.

Critical strengths
  1. The paper addresses a meaningful question: when does reasoning improve an answer enough to justify using it? The analysis clearly explains the missing comparison in mode-homogeneous training groups.

  2. The method is well motivated, and the translation experiments show a promising quality–cost trade-off. The ungated ablation provides useful evidence that credit assignment matters.

Critical weaknesses
  1. Simple baselines at matched budgets are missing. Comparisons with random mode selection, length-based heuristics, or a lightweight router at comparable reasoning-token budgets would help establish the value of the learned selection mechanism.

  2. The source of the gate’s benefit is not fully isolated. Removing the gate changes both input-dependent weighting and the overall strength of reasoning-token updates. A constant gate with matched average weight, or shuffled gates, would help determine whether the gains come from estimating reasoning benefit or simply attenuating updates.

  3. Generalization is demonstrated only within translation. Multiple language pairs, domains, and model sizes are useful checks, but they do not establish applicability to other task families. The broader claims about selective reasoning would benefit from such evaluation.

Questions for authors

How does it compare with constant or shuffled gates while keeping the remaining objective unchanged?

7 comments
AuthorsOpenReview ↗

Response to reviewer F9ZR

Thank you for the review and the concrete suggestions. All new numbers use the paper's test protocol (8 samples per input at T=0.7, 2,048-token cap).

1. Simple baselines at matched budgets

We compared TECA with routers that choose, for each input, between the base model's reasoning translation and its direct one. Each router reasons on the same share of inputs as TECA within every tier and language (45% overall).

  • Random: reason on a random subset.
  • Longest sources: reason on the longest inputs.
  • TF-IDF router: ridge regression on word and character n-grams plus source length, trained to predict the COMET-22 gain from reasoning.
  • Embedding routers: the same regression on Qwen3-8B's representation of the source, from its input embeddings or its last hidden layer.

The learned routers are cross-fitted on the test set itself (5 folds), which favours them.

Test set, average over tiers and languagesXCOMET-XLCOMET-22chrF++
TECA0.7140.73630.4
Random0.6880.70424.7
Longest sources0.6880.70524.5
TF-IDF router0.6870.70324.6
Embedding router, input embeddings0.6890.70524.6
Embedding router, last hidden layer0.6890.70525.0

TECA beats every router on XCOMET-XL and COMET-22 (paired bootstrap, p<0.001).

The learned routers do no better than the simple ones. They have to predict, from the source text alone, how much reasoning will help, and their predictions barely match the measured gain (correlation at most 0.05). TECA does not have to predict this. During training it generates both a reasoning and a direct translation for the same input and observes which one scores higher.

2. Is the gate's benefit isolated? (Question: constant or shuffled gates with the rest of the objective unchanged)

We trained both controls you suggested on Qwen3-8B model. Each is identical to TECA except for the weight on reasoning tokens.

  • Constant gate. Every reasoning token gets weight 0.645, matching TECA's average gate. The overall reduction is the same, but it no longer depends on the input.
  • Shuffled gate. In each batch, the gate values are randomly reassigned to other sources. The values are the same, but they go to the wrong sources.
Qwen3-8B, test setFigurative XCchrF++OpenSubtitles XCchrF++FLORES XCchrF++Think %
Mixed GRPO (ungated), paper0.63622.00.72032.50.67036.846
Constant gate (0.645)0.59815.40.71927.70.63428.128
Shuffled gate0.63620.50.75932.90.65636.441
TECA, paper0.65822.70.78133.90.70238.445

Lowering the weight on every reasoning token does not reproduce TECA's gain. Keeping the same weights but giving them to the wrong sources also loses. The gain comes from giving each source its own weight.

3. Generalisation beyond translation

We trained TECA on math in AdaptThink's setting. The model is DeepSeek-R1-Distill-Qwen-1.5B, trained on DeepScaleR, and evaluated with AdaptThink's grader and sampling (T=0.6, 16K tokens, 16 samples per AIME problem). The reward is answer correctness.

R1-Distill-Qwen-1.5BGSM8K accThink %MATH500 accThink %AIME 2024 accThink %
Base, reasoning79.210081.510026.9100
Base, no reasoning68.2064.7010.80
AdaptThink-1.5B (released)83.12782.23730.862
TECA84.03883.64131.163

TECA improves on the always-reasoning model by 4.8, 2.1 and 4.2 points while reasoning on 38%, 41% and 63% of problems. It matches or exceeds AdaptThink's accuracy on all three sets. Its think rate rises with difficulty, from 38% on GSM8K to 63% on AIME, so the decision tracks where reasoning pays off.

We thank the Reviewer for their valuable time and efforts invested in our work, we believe that suggestions made will strengthen our work. We hope our rebuttal has clarified your concerns; please let us know if any questions remain, we would be happy to provide further clarification. If the rebuttal has resolved your concerns, we would greatly appreciate your reconsideration of the overall rating in light of the additional information provided.

Regards,

Authors

Dear Reviewer,

As the rebuttal period is nearing its end, we would be happy to provide any further clarifications that may help address your remaining concerns.

If our rebuttal has resolved your concerns, we would sincerely appreciate your reconsideration of the overall rating in light of the additional information provided.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_b2zYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies an interesting question on how LLMs can learn when to reason rather than always reasoning. The authors show that standard group-relative RL cannot learn this decision when all rollouts use the same reasoning mode. To address this issue, they propose treatment-effect-aware credit assignment (TECA), which mixes reasoning and non-reasoning rollouts and uses an empirical-Bayes estimate of their relative performance to guide credit assignment. Experiments on machine translation with Qwen3-4B and 8B show that TECA achieves competitive or better translation quality than existing RL baselines while reducing reasoning costs.

Critical strengths

This paper has the following advantages that I appreciate very much:

  • I appreciate the theoretical analysis showing why standard group-relative RL fails to learn when to reason. The advantage decomposition is simple but useful, and it provides a good motivation for the proposed method in my opinion.

  • In addition, I think the empirical observation that reasoning increases the variance of translation quality without consistently improving its mean is interesting. It also provides useful evidence that reasoning should be applied selectively rather than uniformly.

  • Finally, the experimental results are encouraging. TECA achieves competitive performance with substantially fewer reasoning tokens.

Critical weaknesses

While I believe the paper has many advantages, the paper could also benefit from the following clarification/improvements:

  • The paper could benefit from a stronger ablation of the proposed gate. The current comparison with ungated Mixed GRPO does not fully establish whether the improvement comes from the treatment-effect estimates or simply from reducing the gradients on reasoning tokens. A constant-gate or shuffled-gate baseline would help clarify this.

  • In addition, the evidence for learning input-dependent reasoning decisions is could also be strengthened. The reported correlation is relatively weak, and the paper also observes that reasoning benefits are related to input length. I think it would be helpful to evaluate whether the learned decisions remain meaningful after controlling for input length.

  • Finally, I think the experimental evaluation could be strengthened with additional baselines and repeated runs. In particular, I noticed that the authors discussed Translation with Though as a closely related method, and I wonder if they could include it as an additional baseline.

Questions for authors

Please refer to my summarization of weaknesses.

5 comments
AuthorsOpenReview ↗

Response to reviewer b2zY

Thank you for the careful review. All new numbers use the paper's test protocol (8 samples per input at T=0.7, 2,048-token cap).

1. Is the gain from the gate's estimate, or just from smaller updates on reasoning tokens?

We trained both controls you suggested on Qwen3-8B model. Each is identical to TECA except for the weight on reasoning tokens.

  • Constant gate. Every reasoning token gets weight 0.645, matching TECA's average gate. The overall reduction is the same, but it no longer depends on the input.
  • Shuffled gate. In each batch, the gate values are randomly reassigned to other sources. The values are the same, but they go to the wrong sources.
Qwen3-8B, test setFigurative XCchrF++OpenSubtitles XCchrF++FLORES XCchrF++Think %
Mixed GRPO (ungated), paper0.63622.00.72032.50.67036.846
Constant gate (0.645)0.59815.40.71927.70.63428.128
Shuffled gate0.63620.50.75932.90.65636.441
TECA, paper0.65822.70.78133.90.70238.445

Lowering the weight on every reasoning token does not reproduce TECA's gain. Keeping the same weights but giving them to the wrong sources also loses. The gain comes from giving each source its own weight.

2. Do the learned decisions remain meaningful after controlling for input length?

Yes. TECA does not reason more on longer inputs. Within each tier and language pair, think rate is essentially uncorrelated with source length (r=0.01):

Source length (words)1–34–56–1011–2021–40>40All
Sources3211023370955791,650
TECA think %46.546.541.346.246.041.745.4

At matched length, TECA reasons less on social-media text (−8 points) and more on subtitles (+8 points) than on other domains. We will add these analyses to the appendix.

3. Additional baselines and Translation with Thought

Baselines. We added three routers that pick a mode per input with the same reasoning-token budget as TECA: random, reasoning on the longest sources, and a learned router. TECA beats all of them on all three metrics (p<0.001).

The full table is in our reply to Reviewer F9ZR ("Simple baselines at matched budgets"). In addition, we also conduct an experiment math task and compare our methods with an additional baseline AdaptThink (Zhang etal EMNLP 2025) in Reviewer F9ZR "Generalisation beyond translation".

Translation with Thought. TwT is built differently from TECA (Appendix A):

TwTTECA
Supervised stageSFT on DeepSeek-R1 traces, with input difficulty labelled by GPT-4o and traces rewritten by GPT-4o to matchNone
Learning signalOutside teacher models, then RLThe model's own paired samples
What it learnsHow much to reasonWhether reasoning wins on each input

A head-to-head comparison would mix the effect of stronger teachers with that of our credit assignment. TwT also has no public code, and a reimplementation would not be a fair baseline. The two approaches can be combined: a reasoning budget can be learned on top of the decision to reason at all. We will make this comparison explicit in the main text.

We thank the Reviewer for their valuable time and efforts invested in our work, we believe that suggestions made will strengthen our work. We hope our rebuttal has clarified your concerns; please let us know if any questions remain, we would be happy to provide further clarification. If the rebuttal has resolved your concerns, we would greatly appreciate your reconsideration of the overall rating in light of the additional information provided.

Regards,

Authors

Dear Reviewer,

As the rebuttal period is nearing its end, we would be happy to provide any further clarifications that may help address your remaining concerns.

If our rebuttal has resolved your concerns, we would sincerely appreciate your reconsideration of the overall rating in light of the additional information provided.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.