- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether the benefit of turning on thinking transfers across model developers on particular questions, with the motivation of being able to predict per-question value of adaptive computation. Section 6 includes initial experiments demonstrating how the finding could be used to improve the efficiency of adaptive computation.
Critical strengths
- The core finding is interesting. It's good that they validate that there is cross-model predictability.
- The paper is easy to follow.
Critical weaknesses
- More work is needed to motivate the practicality of using these findings for adaptive computation. Section 6 is a good start, but the demonstrated allocation method requires donor evaluations of the target questions to be successful, while the attempts to predict value for unseen questions do not establish a practical alternative.
- The current analysis mostly considers turning thinking on vs off. It would be good to analyze different reasoning effort levels and the number of reasoning tokens as well.
- The core finding that the per-question value of thinking tokens transfers across models is not that surprising in retrospect. It basically validates the notion that some problems are intrinsically harder than others, which is good to validate quantitatively but also fairly obvious.
- Why is it framed as cross-developer? Surely cross-model for a given developer (or just for a large set of models without considering the developer) is similarly informative for the main thesis. I'm just not convinced that "cross-developer" should be the framing.
Questions for authors
See weaknesses