- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates when small reasoning models benefit from further reflection versus external information. Through counterfactual interventions at intermediate reasoning states, it argues that reflection mainly consolidates already-reachable solutions, whereas relevant external information can make new solutions reachable, and that small models are less able to exploit such information. It proposes FLYBY, trained through rescue-based SFT and cost-aware RL to make targeted, multi-depth queries during reasoning. On 6 hard benchmark subsets, FLYBY-4B achieves 45.96% pass@8 versus 41.64% for Qwen3-14B at approximately 2.7x lower estimated serving cost.
Critical strengths
-
Addresses a practical question about allocating internal reasoning and external assistance. Clearly articulates and empirically supports the distinction between reflection (consolidating reachable solutions) and external information (expanding reachability).
-
Provides extensive intermediate-state intervention experiments across model families and scales.
-
Demonstrates substantial accuracy–cost improvements, supported by backend-transfer and price-sensitivity experiments.
Critical weaknesses
-
The execution/knowledge distinction and the claimed selective diagnosis are not cleanly validated: V(s)=0 under N=8 is a finite-sample label, solution-conditioned oracle hints mix factual knowledge with solution strategies, and Fig. 6a only probes states where the policy chose to query (with Qwen3-4B as probe and a 512-token budget).
-
Leakage control is limited: the problem is hidden only by an 8-gram overlap filter, queries can paraphrase the core question (e.g., Fig. 10), and the leakage test uses non-thinking Qwen3-4B as the reader.
-
The baselines do not isolate the learned policy from backend access: Query Opening is untrained and limited to a single depth-1 query, and there is no direct-delegation baseline to DS-V4-Flash or an uncertainty-based cascade.
Questions for authors
-
Can you separate factual knowledge assistance from strategic reasoning hints using independently estimated state labels, e.g., re-estimating V(s) with fresh and larger samples, and labeling hints as factual vs. strategic.
-
Can you evaluate query and non-query states using FLYBY itself with matched continuation budgets?
-
Does the advantage persist against direct delegation to DS-V4-Flash, an uncertainty-based cascade, and equally trained upfront/fixed-timing policies? How does FlyBy differ from ToolOrchestra[1] ?
-
Can you report full-distribution results, practical answer-selection accuracy, and measured end-to-end latency?
[1] Hongjin Su, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, Pavlo Molchanov Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:115934-115956, 2026.