acceptodds
Withdrawn from ICLR 2027

Which Questions Are Worth Thinking About? Cross-Developer Transfer of Thinking Value

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Which questions are worth the additional cost of thinking? We test whether question-level accuracy gains from configured thinking modes transfer across model developers. We define thinking value as the paired accuracy change between configured thinking and non-thinking modes and evaluate 18 models from seven developers on 300 MMLU-Pro STEM questions and 198 GPQA-Diamond questions under three prompting strategies. Gains estimated from other developers predict a held-out developer's gains after IRT-style adjustment for measured difficulty and discrimination, with pooled correlations of 0.264 and 0.273. Transfer persists with an independent difficulty panel and under request-setting and scoring checks. In retrospective allocation on the evaluated questions, ranking transferred gains per predicted additional token recovers 91.8% and 79.1% of the always-thinking accuracy gain at half of the always-thinking additional-token budget, compared with 83.5% and 74.4% for a difficulty-based gain predictor per token. The advantage is statistically resolved on MMLU-Pro but not GPQA-Diamond. An ability-shift simulation produces comparable transfer, leaving the source of this predictability unresolved. The tested text-based predictors generalize weakly to unseen questions. These findings characterize cross-developer transfer of thinking value on previously evaluated questions and motivate learning to predict this value for adaptive computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM reasoning, test-time compute, adaptive reasoning, chain-of-thought, cost–accuracy trade-offs, cross-model transfer, compute allocation, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_FPwQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether the benefit of turning on thinking transfers across model developers on particular questions, with the motivation of being able to predict per-question value of adaptive computation. Section 6 includes initial experiments demonstrating how the finding could be used to improve the efficiency of adaptive computation.

Critical strengths
  • The core finding is interesting. It's good that they validate that there is cross-model predictability.
  • The paper is easy to follow.
Critical weaknesses
  • More work is needed to motivate the practicality of using these findings for adaptive computation. Section 6 is a good start, but the demonstrated allocation method requires donor evaluations of the target questions to be successful, while the attempts to predict value for unseen questions do not establish a practical alternative.
  • The current analysis mostly considers turning thinking on vs off. It would be good to analyze different reasoning effort levels and the number of reasoning tokens as well.
  • The core finding that the per-question value of thinking tokens transfers across models is not that surprising in retrospect. It basically validates the notion that some problems are intrinsically harder than others, which is good to validate quantitatively but also fairly obvious.
  • Why is it framed as cross-developer? Surely cross-model for a given developer (or just for a large set of models without considering the developer) is similarly informative for the main thesis. I'm just not convinced that "cross-developer" should be the framing.
Questions for authors

See weaknesses

Reviewer_cntrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether the accuracy gain from enabling a configured thinking mode transfers across model developers on the same questions. It estimates signed gains from other developers and uses them to predict a held-out developer's gains on MMLU-Pro and GPQA-Diamond. The paper tests several correctness adjustments and uses the transferred estimates for retrospective token-budget allocation. It also examines an ability-shift explanation and prediction on unseen questions.

Critical strengths

Holding out entire developers is a sensible design, and the equal weighting avoids letting a developer with many model variants dominate the result. The independent difficulty panel and protocol checks strengthen the empirical analysis. I also appreciate the authors' reporting of results that limit their interpretation, including the ability-shift simulation, weak generalization to unseen questions, and the unresolved cost-normalized advantage on GPQA.

Critical weaknesses
  1. The additional contribution beyond existing correctness structure remains unclear. Cross-model agreement on questions that benefit from more reasoning has already been examined in SciReas, Appendix C.3. Signed gains and developer-held-out prediction extend that observation, but the paper's own ability-shift simulation produces adjusted transfer of comparable magnitude without a thinking-specific item parameter. The paper's IRT-style adjustment, based on item response theory, therefore does not establish how much additional structure the empirical thinking-value estimates capture. A direct predictive comparison against a joint correctness model for both modes, using the same available information, would be more informative. I appreciate that the authors acknowledge the unresolved explanation; it still leaves an important question about the size of the contribution.

  2. The practical result is restricted and the advantage over the stronger baseline is modest. The successful rankings require prior evaluations of the target questions by other developers; the tested text predictors do not generalize consistently to new questions. At half the additional-token budget, with both scores normalized by predicted cost, the accuracy advantage over difficulty-fit is 1.1 percentage points on MMLU-Pro and 1.0 on GPQA, with the latter confidence interval including zero. This is a useful result for reusing a fixed question bank, but it does not yet establish a broadly useful allocation method. The retrospective cost axis uses realized target-model costs, and acquiring the donor evaluations is outside the reported budget. A prospective evaluation with an explicit reuse scenario would make the practical benefit clearer. The primary benchmarks also lack repeated calls within each model-question-prompt-mode cell, leaving the stability of individual gains insufficiently checked.

  3. I found the paper hard to follow, particularly when trying to identify the main contribution. The exposition moves between correlations of developer-group averages, prediction for one held-out developer, residual correlations under different controls, and fractions of recovered gain. These quantities answer different questions, but the main argument is difficult to track across them. I would lead with the actual predictor, the mean gain from six donor developers, and state exactly what information is available when making a decision. The developer-held-out prediction and its absolute allocation benefit could then form the main result, with the other analyses organized around the specific alternative explanations they test. A worked example would also help connect the definitions to the allocation experiment.

I recommend rejection. The study is careful, but I do not yet find the additional insight and demonstrated utility sufficient for acceptance. A stronger correctness-based comparison, more convincing allocation evidence, and a clearer organization around the contribution would substantially improve the paper.

Questions for authors
  1. How does the transferred-gain predictor compare with a joint response-curve model that predicts correctness in both modes? The comparison should use the same information constraints; any target-model calibration should use separate questions, rather than the target outcomes being evaluated.
  2. In a prospective allocation experiment using predicted costs, what accuracy and realized token usage are achieved? For what reuse scenario does the benefit justify acquiring the donor evaluations?
  3. On a prespecified subset of the primary benchmarks, do independent repeated calls preserve the gain rankings and allocation advantage?
Reviewer_vqsEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates what questions require thinking mode, by proposing a thinking value, which is defined as the change in accuracy obtained by enabling thinking mode on a given question. It evaluates 18 models from seven developers on two multiple-choice benchmarks and predicts a held-out developer’s question-level gains using gains measured for the remaining developers. Positive correlations persist after adjustment for difficulty and discrimination, and allocating tokens using this information achieves higher accuracy than a difficulty-based baseline on MMLU-Pro. However, the source of this transfer remains unresolved, and text-based predictors do not consistently generalize to unseen questions.

Critical strengths
  1. The experimental design systematically evaluates cross-developer transfer. Holding out entire developers rather than individual models reduces the risk that similarities among variants from the same developer inflate transfer performance. Equal weighting across developers, difficulty and discrimination adjustments, an independent difficulty panel, and sensitivity analyses covering prompts and request settings examine the observed relationship from multiple perspectives.

  2. The paper tries connecting observed transfer to the value of computational resource allocation. It uses signed gains that account for both accuracy improvement and degradation to evaluate selection under a constrained budget. Comparisons against difficulty-based prediction at matched budgets quantify the additional information provided by other developers’ question-level evaluations.

Critical weaknesses
  1. The additional conceptual contribution over closely related work is limited. Prior work has shown that models from different developers tend to select the same incorrect answers and that these correlated errors affect model evaluation and downstream decisions [2]. More directly related to the benefits of thinking, Li et al. [1] showed that questions changing from incorrect to correct under increased reasoning effort overlap across models. This paper extends these observations through question-level signed gains, developer-level separation, and difficulty and discrimination adjustments. These extensions improve experimental rigor, but the new understanding they provide beyond previously established cross-model commonalities is comparatively limited. In particular, the ability-shift simulation produces similar transfer, leaving the relationship between shared thinking gains and existing correctness patterns insufficiently explained.

[1] Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning. ICML 2026.

[2] Correlated Errors in Large Language Models. ICML 2025.

  1. The demonstrated practical benefits are under limited conditions. The successful allocation experiments require other developers’ thinking and non-thinking evaluations of the target questions, while excluding the cost of acquiring those evaluations from the allocation budget. The results therefore apply most directly to the reuse of previously evaluated question banks. Prediction on unseen questions is not guaranteed consistent. The practical contribution to general adaptive computation remains limited.

  2. The choice of developer-level aggregation could be better justified. Averaging within each developer seems one reasonable way to give developers equal weight. However, it requires to show whether models from the same developer tend to have similar question-level thinking gains, and how this compares with similarity across developers. This would provide additional support for treating developers as the unit of analysis.

Questions for authors
  1. Are question-level thinking gains more similar among models from the same developer than among models from different developers?

  2. How does transfer depend on model size? For example, do gains measured on smaller models predict gains on larger models as well as gains measured on similarly sized models?

Reviewer_zs3hOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigate the more thinking benefits, and whether they are transferable between model families.

Critical strengths
  1. The research question is interesting.

  2. The experiment is well-design.

Critical weaknesses
  1. The paper is not well written and hard to follow. It reads like a tech report rather than a paper. The problem is not well formulated and defined.

  2. Limited insights. Exposure does not lead to model exploitation (in LLM fine-tuning case). The benefits from clean validation is not clear/strong.

Questions for authors

Q1. why we need to average different models to define the thinking benefit? It should be both model-shared and model-dependent. It would be beneficial to analyze the two terms seperatly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.