- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates when language models should reason rather than always relying on reasoning. The authors propose TECA, a treatment-effect-aware credit assignment method that uses mixed think/no-think sampling and empirical-Bayes estimates to guide reasoning decisions. Experiments on machine translation demonstrate competitive translation quality with substantially reduced reasoning-token usage.
Critical strengths
Interesting motivation: The paper addresses an important problem of when reasoning is beneficial, rather than assuming that reasoning always improves performance.
Well-motivated method: The proposed TECA framework provides a clear analysis of the limitations of standard GRPO and introduces treatment-effect-aware credit assignment to address them.
Critical weaknesses
The experiments are limited to small models, most reasoning failures are caused by reaching the token budget limit, and nearly all invalid reasoning outputs result from hitting the token budget limit. This suggests that the observed degradation may partly reflect insufficient token budgets rather than an inherent drawback of reasoning, raising concerns about generalizability to larger models.
Questions for authors
Does this phenomenon persist in larger models, such as those from OpenAI, Anthropic, or DeepSeek, when comparing thinking and non-thinking modes with sufficiently large token budgets?
5 comments
We thank the Reviewer for their valuable time and efforts invested in our work, we believe that suggestions made will strengthen our work. We hope our rebuttal has clarified your concerns; please let us know if any questions remain, we would be happy to provide further clarification. If the rebuttal has resolved your concerns, we would greatly appreciate your reconsideration of the overall rating in light of the additional information provided.
Regards,
Authors
Dear Reviewer,
As the rebuttal period is nearing its end, we would be happy to provide any further clarifications that may help address your remaining concerns.
If our rebuttal has resolved your concerns, we would sincerely appreciate your reconsideration of the overall rating in light of the additional information provided.
Thank you for your time and consideration.
Best regards,
The Authors
Response to reviewer d6A9
Thank you for the review.
1. Does the cost of reasoning come from the token budget?
TECA does not assume that reasoning hurts on average. It assumes only that reasoning helps on some inputs and not on others, and it learns which. That remains true when the token budget is large enough that reasoning almost never runs out.
We repeated the Section 2.2 measurement with a 3,200-token budget for reasoning (same 1,350 sources, 12 samples per mode, base Qwen3-8B). The table gives the share of reasoning samples that produce no translation, and reasoning's effect on COMET-22 relative to direct translation, with failed samples scored 0 as in the paper.
The 1,280 column is the paper's (training cap). The 2,048 column, our evaluation cap, takes the 3,200-token samples and counts those whose reasoning was still open at 2,048 tokens as failures.
Two things hold at 3,200 tokens, where under 1% of reasoning samples fail.
A larger budget removes the failures, but it does not make reasoning the better choice everywhere. Always reasoning would be the worse mode on roughly half of the inputs, and never reasoning would give up the gains on the other half. TECA decides per input, which is useful at any budget. We will add this table to Appendix I.
2. Does it hold for larger models?
Yes. Reasoning does not beat direct translation in any of these models:
Qwen3-32B is from our Table 3. The other four are from Rajaee et al. (2026) on WMT24++.
The budget is ample for Qwen3-32B: 98.5% of its outputs are valid, and direct translation also leads on COMET-22 and chrF++ (Table 3). Rajaee et al., discussed in Appendix A, test exactly the kinds of systems you mention. Zebaze et al. (2025) find the same for Qwen3 models up to 32B with a 3,500-token thinking budget. Across families and scales, reasoning's average effect on translation is near zero or negative even with generous budgets. TECA is built for this case: no fixed always-reason or never-reason policy is best, and the useful decision is made per input.
We cannot train TECA on closed models. It forces the first decision token into each mode and updates the model's weights, and closed APIs allow neither. For them the question can only be answered by evaluation, which the studies above provide.