- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how LLMs switch between "think" and "no-think" modes. The authors claim that prior work are unclear which strategy really works. So they build THINKSWITCH, a unified framework that crosses three switch strategies with four training regimes. They do experiments on 6 models, 5 benchmarks and 527 runs in total. They also conduct 12 existing methods in the same pipeline. The main findings are that PT gives the best accuracy-token trade-off, RT gives moderate token savings, SPEC raises accuracy but costs more tokens. Training is helpful for token cost but not accuracy. The best strategy changes with model scale and task domain.
Critical strengths
The unified setup is useful. Prior works don't do experiments under the same settings. This paper makes the comparison possible.
The scale and scope of the experiments are strong. 6 models, 5 benchmarks, 12 own configurations, and 12 external methods is a lot of work. The findings on scale and domain shifts are interesting and useful.
Critical weaknesses
Some baselines use modified implementations. These changes may affect the results. Although the paper explains these differences, it is still unclear how much they affect the main findings.
Token counts alone cannot fully reflect actual inference cost and efficiency.
Questions for authors
How much do the modified baseline implementations affect the main results?
Can the authors report inference latency alongside token cost?