acceptodds
Under review as a conference paper at ICLR 2027

Knowing When Thinking Is Not Enough: Teaching Small Reasoning Models to Reason Beyond Their Parametric Knowledge

Abstract

Scaling test-time computation is a powerful way to improve language-model reasoning, and is particularly appealing for small reasoning models (sRMs) that are cheap to serve. However, is additional thinking always the right operation? By intervening at intermediate reasoning states across two model families and multiple scales, we find that self-refinement largely consolidates probability mass onto solutions already reachable from the current state, rather than making new ones reachable. These interventions reveal two failure regimes: execution bottlenecks, where the correct path is reachable and reflection can recover it, and knowledge bottlenecks, where relevant external information makes it reachable. Motivated by this distinction, we introduce FlyBy, a selective querying framework, and train 4B and 8B variants to reason first, diagnose what remains unresolved, and, at a knowledge bottleneck, query stronger models whose parametric knowledge extends beyond its own. Supervised fine-tuning bootstraps a multi-depth query action, and cost-aware reinforcement learning calibrates whether to query, what to ask, and how much to spend. On 1,158 hard problems across six benchmarks, FlyBy-4B achieves 45.96% pass@8, surpassing Qwen3-14B (41.64%) at 2.7 lower serving cost, while also exceeding Qwen3-8B in pass@1 (16.85% vs 15.31%). Scaling to FlyBy-8B further improves pass@8 to 51.81%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7kgtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates when small reasoning models benefit from further reflection versus external information. Through counterfactual interventions at intermediate reasoning states, it argues that reflection mainly consolidates already-reachable solutions, whereas relevant external information can make new solutions reachable, and that small models are less able to exploit such information. It proposes FLYBY, trained through rescue-based SFT and cost-aware RL to make targeted, multi-depth queries during reasoning. On 6 hard benchmark subsets, FLYBY-4B achieves 45.96% pass@8 versus 41.64% for Qwen3-14B at approximately 2.7x lower estimated serving cost.

Critical strengths
  1. Addresses a practical question about allocating internal reasoning and external assistance. Clearly articulates and empirically supports the distinction between reflection (consolidating reachable solutions) and external information (expanding reachability).

  2. Provides extensive intermediate-state intervention experiments across model families and scales.

  3. Demonstrates substantial accuracy–cost improvements, supported by backend-transfer and price-sensitivity experiments.

Critical weaknesses
  1. The execution/knowledge distinction and the claimed selective diagnosis are not cleanly validated: V(s)=0 under N=8 is a finite-sample label, solution-conditioned oracle hints mix factual knowledge with solution strategies, and Fig. 6a only probes states where the policy chose to query (with Qwen3-4B as probe and a 512-token budget).

  2. Leakage control is limited: the problem is hidden only by an 8-gram overlap filter, queries can paraphrase the core question (e.g., Fig. 10), and the leakage test uses non-thinking Qwen3-4B as the reader.

  3. The baselines do not isolate the learned policy from backend access: Query Opening is untrained and limited to a single depth-1 query, and there is no direct-delegation baseline to DS-V4-Flash or an uncertainty-based cascade.

Questions for authors
  1. Can you separate factual knowledge assistance from strategic reasoning hints using independently estimated state labels, e.g., re-estimating V(s) with fresh and larger samples, and labeling hints as factual vs. strategic.

  2. Can you evaluate query and non-query states using FLYBY itself with matched continuation budgets?

  3. Does the advantage persist against direct delegation to DS-V4-Flash, an uncertainty-based cascade, and equally trained upfront/fixed-timing policies? How does FlyBy differ from ToolOrchestra[1] ?

  4. Can you report full-distribution results, practical answer-selection accuracy, and measured end-to-end latency?

[1] Hongjin Su, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, Pavlo Molchanov Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:115934-115956, 2026.

Reviewer_iNLnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies when extra thinking fails to help small reasoning models. Counterfactual interventions show that self-reflection mainly consolidates already-reachable answers (execution bottlenecks), while knowledge bottlenecks require external information. Building on this, the authors propose FlyBy, which trains a small model with SFT and cost-aware RL to query external LLMs selectively at three cost levels. On 1,158 hard problems, FlyBy-4B reaches 45.96% pass@8, surpassing Qwen3-14B at 2.7× lower cost.

Critical strengths
  1. The counterfactual analysis is careful and large-scale (about 272K continuations, two model families). The random-cue control shows that gains come from relevant information, not from extra generation.

  2. The tool design is simple and practical: the external model never sees the problem, query depth is adaptive, and cost covers both GPU and API usage.

Critical weaknesses
  1. Missing cost-matched baselines: direct external-LLM solving, a simple cascade, and related methods ([1] ToolOrchestra). The gains cannot yet be attributed to the learned policy rather than to access to stronger models.

  2. Practical concerns are unaddressed. Latency is not reported, and robustness to incorrect external replies is untested.

  3. Leakage control (an 8-gram filter) is weak, and multi-turn queries may effectively outsource sub-problems.

  4. Novelty is moderate. The method is close to existing help-seeking frameworks, and its link to the bottleneck analysis is only implicit.

[1] Toolorchestra: Elevating intelligence via efficient model and tool orchestration

Questions for authors
  1. Under matched cost, how does FlyBy compare with direct external-LLM solving, a cascade, and [1]ToolOrchestra?

  2. What is the end-to-end latency compared with Qwen3-14B?

  3. How much does accuracy drop when some external replies are incorrect, and can the model detect them?

  4. Does FlyBy regress on the full benchmarks beyond the hard subset?

[1] Toolorchestra: Elevating intelligence via efficient model and tool orchestration

Reviewer_nyWhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FlyBy trains Qwen3-4B/8B to query stronger DeepSeek models during reasoning, using SFT on selected rescue trajectories followed by cost-aware RL. The paper motivates this approach through a distinction between execution and knowledge bottlenecks. On six hard benchmark subsets, it reports improved pass@8 and lower estimated serving costs, including external API calls, relative to larger Qwen3 models.

Critical strengths

The reported accuracy–cost improvements are substantial on the selected hard problems. Retaining much of the improvement after changing the external backend is also a useful result.

Critical weaknesses
  1. I came away with limited new understanding of the problem. That targeted hints from a stronger model help more than generic reflection prompts is largely expected. The proposed bottleneck distinction does not make this finding much more informative: failing to solve a problem in eight continuations does not show that the model lacks knowledge. It may simply lack the reasoning or search ability to use what it knows. Likewise, a helpful hint may supply the crucial inference rather than a missing fact. The paper does not convincingly separate these explanations.

  2. I do not see a clearly established technical advance over existing trained model collaboration. The SFT-plus-cost-aware-RL recipe is familiar, and the main comparisons leave important factors uncontrolled. Query Opening receives no additional training and makes only one shallow call, while FlyBy is trained and can query repeatedly at different depths. A matched comparison with a trained multi-round router such as Router-R1 is missing. The contribution of carefully selected, strong-teacher-generated rescue data is also not isolated. Missing Qwen3.5-4B/9B and direct inexpensive-backend baselines further limit the practical claim. These gaps require substantive experiments, not just clearer writing.

Questions for authors
  1. What evidence distinguishes missing factual knowledge from insufficient reasoning or search ability?

  2. Does the advantage hold against a trained multi-round router with matched training data, external models, and inference budgets?

  3. How much of the improvement comes from rescue-data selection rather than the proposed querying policy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.