acceptodds
Under review as a conference paper at ICLR 2027

Learning to Ask: Information Acquisition for SLM-LLM Collaboration, under a budget

14 views

Abstract

Collaboration between a small language model (SLM) and a large language model (LLM) offers an opportunity to combine the efficiency of smaller models with the strong reasoning capabilities of larger ones. Existing approaches primarily frame such collaboration as a computation allocation problem, determining which model should handle each portion of the reasoning process. In black-box API-based settings, however, this paradigm can be inefficient due to coarse-grained delegation or repeated transmission of context across model switches. In this work, we instead formulate SLM-LLM collaboration as an problem, under an API budget constraint. The SLM remains the primary reasoner and selectively queries a black-box LLM advisor only when needed, issuing targeted queries rather than delegating the reasoning process itself. To realize this strategy, we develop a three-stage RLVR framework that learns whether to call the advisor, how to formulate useful queries, and how to integrate the collaboration into the reasoning process by jointly refining advisor invocation and information use. Across mathematical reasoning and coding tasks, our approach improves the performance–cost tradeoff over existing collaboration baselines and, in some settings, matches or exceeds oracle problem-level routing. Finally, we show that our strategy can transfer to other advisor model families, without further training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Collaborative reasoning, Reasoning, Black-box LLM, SLM, RLVR, Tool calling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZcFBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames budget-constrained SLM-LLM collaboration as information acquisition. A 7B SLM reasons on its own and, at most once per trajectory, sends a short self-contained query to a black-box advisor instead of routing the whole problem or resending its trace. A three-stage RLVR pipeline learns whether to call, what to ask under a response budget (through a separate query generator, QMaker), and how to use the advice. On math and coding benchmarks, the method is reported to dominate problem-level and step-level baselines on accuracy versus advisor tokens, and to transfer to unseen advisors.

Critical strengths

The problem is timely: strong models are increasingly consumed through metered black-box APIs, and framing SLM-LLM collaboration as information acquisition under a token budget fits that setting. The results on math and coding, and the transfer to unseen advisors, support the approach within the paper’s cost definition.

The paper is well written. The introduction places the work against routing, step-level co-generation, and tool-augmented LMs, and the pilot study motivates each learned capability with a measured failure of the zero-shot SLM.

The formulation in Section 3 is clean and explicit about its assumptions, and the three-stage reward design follows from it directly.

Critical weaknesses

Cost. The comparison is not compute-matched. The cost in Equation 4 counts advisor tokens only, and Appendix A excludes all local inference by design. On every calling trajectory, the method runs three 7B generations: the SLM up to the call, QMaker over the full reasoning trace so far, and the SLM again with the advice in context. A problem-level router runs a BERT-scale classifier; the cascades run the SLM once plus a verifier. QMaker is itself a second Qwen2.5-7B-Instruct. Replacing advisor output with local work is what reduces the quantity on the x-axis, so the omission affects the main result. The local side has a public price. OpenRouter lists Qwen2.5-7B-Instruct at $0.10 per million input tokens and $0.20 per million output tokens, and the paper’s advisor, Qwen3.5-35B-A3B, at $0.065 and $0.75. At those rates, one advisor reply capped at 2048 tokens costs about as much as 7,500 SLM output tokens, which is the scale of one trajectory plus a QMaker pass. Cheaper 7B models exist (Llama 3.1 8B at $0.02 and $0.04); three passes are still a nonzero fraction of each call. Appendix C.5 concedes that on math the advisor reply is often already close to the final answer, and Figure 5c shows that at budgets of 4,096 tokens and above 94 to 97% of math queries are FULL-TASK requests. The paper does not measure how often the reply contains the answer, and the results of the budget-capped LLM baseline (listed in Section 6.1, plotted in Figure 4) are never discussed in the text. At large budgets, the method may reduce to a capped LLM call with a 7B wrapper.

Oracle comparison. The AMC23 result against the oracle is explained in one sentence: the method has access to a richer collaboration space (Section 6.2). The oracle pays for an uncapped LLM solution on every routed problem while the method’s replies are capped at b, so the gain is a cost effect rather than a routing effect. An oracle allowed to choose the budget-capped LLM for each problem would be a fair comparison, and AMC23 has only 50 problems.

Questions for authors
  1. Can the authors report the comparison with SLM tokens (both SLM segments and the QMaker pass) priced alongside advisor tokens, for the method and the baselines? Counting both sides would make the comparison with routing and cascading methods more balanced, since those methods spend far less locally per problem.

  2. Since the advisor reply on math is often already the final answer (Appendix C.5), what accuracy and cost does the advisor reach when its output is capped at about 100 tokens (for example, by prompting it to give only the answer within 100 tokens), enough for an answer and little else? And if that capped advisor is used as the LLM side of a plain SLM/LLM router (the paper’s RouteLLM or oracle setup), does the resulting frontier match or beat the proposed method? This would show whether the gain comes from the learned queries or simply from capping the advisor’s output.

  3. The method assumes that the SLM’s partial trajectory reveals impending failure and that the missing information fits in a short query; math and coding satisfy both. Do the authors expect task types where either assumption fails (long-context QA, open-ended generation, difficulty revealed late in the trajectory), such that routing or cascading the whole problem would be better on average? How would such tasks be identified in advance?

7 comments
AuthorsOpenReview ↗

W1, Q1 - Cost. The comparison is not compute-matched.

We sincerely thank the reviewer for suggesting the use of publicly available pricing from OpenRouter to account for local model inference costs. This is a valuable evaluation perspective that we had not previously considered. We report the results and analysis that our method remains competitive, on Appendix C.4 of the revised manuscript.

However, we would like to clarify the motivation behind our original cost formulation. Minimizing remote API expenditure is a practically meaningful optimization objective in its own right, distinct from minimizing the total computational cost of the system. Our work considers a hybrid inference setting in which smaller models are executed on locally available, user-controlled hardware, while a stronger black-box model is accessed through a usage-based API. In such settings, local inference consumes computational resources but does not incur the same per-request external API charges as remote inference. Consequently, the primary budget constraint can naturally be the expenditure associated with accessing the remote model. Our original metric was designed to capture this aspect of deployment cost, rather than to serve as a comprehensive measure of total computational overhead.

Prior work also adopts cost metrics centered on expensive model utilization, rather than fully compute-matched comparisons that account for all inference costs. For instance, Hybrid LLM [1] and RouteLLM [2] primarily focus on reducing the cost of expensive model invocations, without explicitly accounting for the computational cost of increased smaller-model inference in their primary cost metrics. Both use the fraction of queries routed to the large model as a proxy for cost, implicitly treating the inference cost of the smaller model as negligible. More closely related to our setting, Router-R1 [3] uses model-weighted output tokens from invoked candidate models in its cost related evaluations, while TRIM [4] measures cost through stronger-model output tokens, despite employing two additional local models for generation and step-level verification. These studies similarly treat larger model utilization as a meaningful efficiency metric without explicitly incorporating all local inference costs into their primary cost formulations. Alongside these works, our cost formulation is a step forward toward explicitly capturing real-world remote black-box API charges through a weighted sum of input- and output-token counts using their respective API prices.

We also conducted an additional cost analysis using OpenRouter's publicly listed model prices and included the results in Appendix C.4, following the reviewer’s suggestion. In summary, our method remains competitive in terms of performance–cost trade-offs even when accounting for additional local inference costs, although some baselines achieve comparable or better trade-offs at certain budget levels on the easy split.

[1] Ding, Dujian, et al. "Hybrid llm: Cost-efficient and quality-aware query routing." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Ong, Isaac, et al. "Routellm: Learning to route llms from preference data." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Zhang, Haozhen, Tao Feng, and Jiaxuan You. "Router-r1: Teaching llms multi-round routing and aggregation via reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 141233-141265.

[4] Kapoor, Vansh, et al. "TRIM: Hybrid inference via targeted stepwise routing in multi-step reasoning tasks." arXiv preprint arXiv:2601.10245 (2026).

AuthorsOpenReview ↗

W1 - How often the reply contains the answer?

We appreciate the reviewer for raising this concern. We acknowledge that this is an important concern that warrants further investigation.

To address this, we conducted an additional analysis measuring how frequently the advisor's response contains the student's final answer. For mathematical reasoning tasks, we measured the proportion of queries in which the student's final submitted answer also appears in the advisor's response. For coding tasks, we measured the similarity between the advisor-generated code and the student's final submitted code. Specifically, when the advisor's response contained code, we computed the string similarity using Python's difflib.SequenceMatcher.ratio(), where a score of 1.0 indicates an exact match. We consider the advisor's response to contain the answer when the similarity is above 0.9. We split the results into FULL_TASK and Non-FULL_TASK query types.

The results show that only 23% of advisor responses contain the student's final answer for Non-FULL_TASK queries, where the original question is not directly forwarded to the advisor. Interestingly, even for FULL_TASK queries, where the original question is provided in its entirety, 46% of advisor responses do not contain the student's final answer.

CapFULL_TASKNon-FULL_TASKAll
5120.38 (51%, 905)0.12 (49%, 859)0.25 (1,764)
1k0.43 (34%, 597)0.16 (66%, 1,171)0.25 (1,768)
2k0.53 (45%, 837)0.36 (55%, 1,037)0.43 (1,874)
4k0.49 (86%, 1,588)0.24 (14%, 268)0.45 (1,856)
8k0.60 (93%, 1,725)0.35 (7%, 122)0.58 (1,847)
16k0.67 (90%, 1,544)0.41 (10%, 179)0.65 (1,723)
All caps0.54 (66%, 7,196)0.23 (34%, 3,636)0.44 (10,832)

This is because the FULL-TASK delegation does not necessarily translate into receiving a complete, self-contained solution from the advisor. In practice, advisor responses often terminate before completing the full reasoning process, leaving substantial portions of the solution unfinished. In such cases, the SLM continues reasoning based on the advisor's response, filling in the missing steps to produce the final answer. Thus, even when FULL-TASK requests are frequent, the SLM can still play an important role in completing the solution, rather than merely reproducing the advisor's output.

We will add these results on the Appendix.

AuthorsOpenReview ↗

W1 - The results of the budget-capped LLM baseline are never discussed in the text.

We have newly added a brief discussion about the budget-capped LLM baseline in Appendix E. Please See the paper for further information.

Reviewer_ahUBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Learning to Ask, a framework for budget-constrained collaboration between SLMs and LLMs. Instead of delegating entire problems or reasoning steps to a stronger model, the SLM selectively queries an LLM advisor for useful information while remaining the primary reasoner. The authors introduce a three-stage RLVR training pipeline to learn when to ask for help, how to formulate useful queries through a separately trained QMaker, and how to incorporate the advisor's responses into subsequent reasoning. Experiments on mathematical reasoning and coding benchmarks show favorable accuracy–API cost trade-offs compared with existing routing and collaborative reasoning baselines. The authors also investigate budget adaptation and transfer to different LLM advisors.

Critical strengths

The paper addresses a practically relevant problem of reducing external LLM API costs while maintaining strong reasoning performance, with a clear motivation for budget-constrained collaboration.

The three-stage training framework is reasonably well structured, explicitly addressing selective advisor invocation, query formulation, and response integration.

Critical weaknesses

1. Limited novelty compared with closely related SLM–LLM collaboration methods.

I find the conceptual novelty of this work relatively limited. Several recent studies have already explored training small language models to selectively seek assistance from stronger LLMs, such as Learning to Seek Help and SWE-Protégé. Although the authors discuss these works and highlight differences in optimization objectives and interaction protocols, I am not convinced that these distinctions represent a substantial conceptual advancement. In particular,the proposed "Whether to Ask +How to Ask+How to Use" framework closely resembles capabilities already explored in prior help-seeking and tool-augmented reasoning systems. Earlier works such as Toolformer and Router-R1 also investigated learning adaptive tool invocation and model collaboration. Overall, the contribution appears to be an incremental combination and refinement of existing ideas, with budget-aware query construction being the main additional component.

2. Unaccounted QMaker overhead and limited generality of the proposed framework.

A major concern is the necessity and efficiency of introducing a separately trained 7B QMaker. The authors train different QMaker models for different API budgets, which introduces substantial additional training and inference overhead. However, the main efficiency evaluation only accounts for the external LLM advisor's input and output token costs, excluding QMaker inference, local computation, and end-to-end latency. Although this limitation is acknowledged in Appendix A, it significantly weakens the overall efficiency claims. Furthermore, the generality of the approach remains unclear. While the authors evaluate transfer to different LLM advisors, QMaker is trained separately for different budgets and task domains, and its effectiveness across different SLM architectures or unseen budgets is not established. I would like to see a more comprehensive evaluation of total system cost and transferability.

3. Insufficient evidence that learned information acquisition is necessary or substantially better than simpler alternatives.

The paper emphasizes targeted information acquisition as its central contribution. However, Figure 5 shows that QMaker frequently generates full-task queries rather than targeted subproblem queries. For example, at a 4K-token budget, approximately 94% of the mathematical queries are classified as full-task requests. This raises the question of how much the proposed mechanism truly differs from conventional task delegation with response-length constraints. More importantly,the paper lacks a direct comparison against simpler alternatives, such as using a fixed prompt to reformulate the SLM's current reasoning state into a concise, self-contained advisor query. Although the authors include an LLM-only baseline with budget instructions, this does not isolate the benefit of the separately RL-trained QMaker. Without such comparisons, it is difficult to determine whether the improvements come from learned information acquisition or simply from query reformulation and output-length control

Questions for authors

Novelty: Could the authors further clarify the fundamental methodological differences from closely related works such as Learning to Seek Help and SWE-Protégé? Beyond introducing token-level budget constraints and a separate query generator, what is the main conceptual or algorithmic advancement?

QMaker overhead and generalization: What are the actual inference latency and computational costs introduced by QMaker? How would the accuracy–cost trade-off change if these costs were included? Additionally, can a single QMaker generalize across different API budgets, task domains, or SLM architectures without retraining?

Necessity of learned query construction: Given that a large fraction of QMaker-generated queries are full-task requests, could the authors compare against simpler prompt-based query reformulation with explicit response-length constraints? How much of the improvement can be attributed specifically to RL-trained QMaker rather than query reformulation or output-length control?

8 comments
AuthorsOpenReview ↗

W1, Q1. Conceptual or algorithmic advancement of our approach - 1

W1. Limited novelty compared with closely related SLM–LLM collaboration methods.

I find the conceptual novelty of this work relatively limited. Several recent studies have already explored training small language models to selectively seek assistance from stronger LLMs, such as Learning to Seek Help and SWE-Protégé. Although the authors discuss these works and highlight differences in optimization objectives and interaction protocols, I am not convinced that these distinctions represent a substantial conceptual advancement. In particular, the proposed "Whether to Ask + How to Ask + How to Use" framework closely resembles capabilities already explored in prior help-seeking and tool-augmented reasoning systems. Earlier works such as Toolformer and Router-R1 also investigated learning adaptive tool invocation and model collaboration. Overall, the contribution appears to be an incremental combination and refinement of existing ideas, with budget-aware query construction being the main additional component.

Q1. Novelty: Could the authors further clarify the fundamental methodological differences from closely related works such as Learning to Seek Help and SWE-Protégé? Beyond introducing token-level budget constraints and a separate query generator, what is the main conceptual or algorithmic advancement?


We agree that, if our work is viewed merely as enabling a small language model to selectively seek help from a stronger LLM, its novelty may appear limited. However, there exists two gaps that prior works insufficiently addressed, as they primarily pursued different objectives. Our contributions lie in addressing these gaps: (1) a reliable training recipe for learning whether-to-call behavior, and (2) a budget instruction mechanism that enables effective learning under real-world API budget constraints. These contributions go beyond incremental refinements, as we demonstrate through evidence from our reproductions of Learning to Seek Help (revised as [1]) and Router-R1 [2]. We provide the reproduced results and detailed discussion in Appendix B.2.

Below, we summarize the head-to-head comparison between our method and these prior works.

Learning to Seek Help [1]

  • Relies on penalty-based joint optimization for invocation decisions, less effective whether-to-call behavior.
  • Utilizing LLM call frequency, which does not adequately reflect actual token-based API costs. Substantially less favorable performance–cost trade-off.

Router-R1 [2]

  • Relies on penalty-based joint optimization for invocation decisions, less effective whether-to-call behavior.
  • Incorporates output-token costs into its training objective, but lacks an explicit mechanism to control its response length according to a given budget, less favorable performance-cost trade-off than ours.

SWE-Protégé [3]

  • Enforce every trajectory to call the advisor, not adaptively call.
  • The main objective is not about performance–cost trade-off perspective of expert invocation. We discuss this distinction further below.

(1) Whether-to-call behavior

Co-optimization of whether-to-call behavior with penalty-based objectives do not reliably induce whether-to-call behavior. To clarify the methodological contribution of our invocation training, we reproduced Learning to Seek Help and Router-R1 both methods using the authors’ implementations on our training data and tasks. For Router-R1, we used its official repository. Since Learning to Seek Help is concurrent work and we could not find a public repository, we used the code provided in the supplementary materials accompanying its concurrent submission. We extensively swept the hyperparameters governing invocation behavior—\alpha in [0.1, 0.3, 0.5] for Router-R1 and \beta in [0.1, 0.3, 0.5] for Learning to Seek Help. For both settings, we conducted the sweep at advisor response budgets both with 1,024 and 4,096 LLM tokens, so we swept a total of 6 training points for each method. Implementation details, and code to reproduce the results are provided in Supplementary. We first report the spearman correlation between the problem difficulty from the SLM, and the invocation ratio.

MethodSettingSpearman ρ
Oursγ\gamma 4+0.721
Oursγ\gamma 12+0.747
Router-R1LLM reply ≤1k, α\alpha=0.1n/a (never calls)
Router-R1LLM reply ≤1k, α\alpha=0.30.21
Router-R1LLM reply ≤1k, α\alpha=0.50.07
Router-R1LLM reply ≤4k α\alpha=0.10.22
Router-R1LLM reply ≤4k α\alpha=0.3n/a (never calls)
Router-R1LLM reply ≤4k α\alpha=0.50.29
LFILLM reply ≤1k, β\beta 0.1n/a (never calls)
LFILLM reply ≤1k, β\beta 0.3n/a (never calls)
LFILLM reply ≤1k, β\beta 0.5n/a (never calls)
LFILLM reply ≤4k,β\beta 0.1n/a (never calls)
LFILLM reply ≤4k,β\beta 0.3n/a (always calls)
LFILLM reply ≤4k,β\beta 0.5n/a (never calls)
AuthorsOpenReview ↗

W1, Q1. Conceptual or algorithmic advancement of our approach - 2

Across the tested configurations, neither method except for ours reliably learned to invoke the advisor selectively according to the SLM’s standalone ability. Theoretically, penalty-based objectives can induce selective invocation by balancing the expected benefit of advisor consultation against its cost, and more intensive hyperparameter search may find a good whether-to-call behavior. However, as the selective invocation is only implicitly rewarded, the rise of this capability is not guaranteed. In contrast, our approach directly rewards difficulty-aligned invocation decisions using the SLM's standalone success rate as a supervision signal, with a separated training stage. This decouples learning whether to call from the quality of query construction and response utilization, avoiding the need to induce selective behavior through a carefully balanced cost penalty. As demonstrated by the substantially higher difficulty–invocation correlations ((\rho=0.721)–(0.747), compared with at most (\rho=0.29) for Router-R1, our carefully designed training method provides a more reliable training mechanism for learning whether to seek assistance, rather than relying on this capability to emerge as a byproduct of unstable joint optimization with implicit reward.

(2) Appropriate cost accounting and corresponding training methods.

Our contribution is not merely to evaluate collaboration using a more realistic API-cost metric, but to introduce a training recipe that explicitly targets token expenditure by controlling the query formulation, together with advisor response verbosity. In particular, LLMs can generate unnecessarily lengthy responses even to focused queries, resulting in substantial API costs. In fact, this is also the case for our query formulations, where the advisor still answers unnecessarily long for transform type queries, even longer than Full-task type queries in some cases. We evaluate the resulting advisor response length for queries in each budget, with sufficient response token budget.

CapTask TypeNo InstrBudget-Inst
512FULL_TASK2,058286
512TRANSFORM3,252342
1kFULL_TASK2,362902
1kTRANSFORM2,378500
2kFULL_TASK2,2631,101
2kTRANSFORM1,719983

Therefore, optimizing query formulation alone does not necessarily ensure cost-efficient advisor interaction.

We address this by training the query generator to produce both a self-contained query and a budget instruction specifying the desired level of response detail. We further evaluated the practical benefits of our approach against the reproduced Learning to Seek Help and Router-R1 baselines under token-based API cost accounting. As shown below, our method generally achieves a more favorable performance-cost trade-off across MATH500, AMC23, and AIME, compared with Router-R1, and Learning to Seek Help. The full results are in the Appendix B.2. Below, we don’t report the results with never-call training configurations, as they have very low performance.

MATH500

BaselineBaseline CostBaseline AccuracyOurs Cost at Same AccuracyCost ReductionOurs Accuracy at Same Cost
Router-R1 1k, cc0.33,2390.80285474%0.862
Router-R1 1k, cc0.53,8920.78163484%0.866
Router-R1 4k, cc0.14,5350.8351,53866%0.866
Router-R1 4k, cc0.54,5120.79780382%0.866
LFI llm4k-turnpen0.39,1360.899(Ours max: 0.866 at 3,414)—0.866

AMC23

BaselineBaseline CostBaseline AccuracyOurs Cost at Same AccuracyCost ReductionOurs Accuracy at Same Cost
Router-R1 1k, cc0.35,2150.7622,14059%0.858
Router-R1 1k, cc0.55,6070.7562,08363%0.866
Router-R1 4k, cc0.17,7740.8816,34318%0.912
Router-R1 4k, cc0.59,2180.8846,48630%0.931
LFI llm4k-turnpen0.313,2340.8977,05947%0.931

AIME 25+26

BaselineBaseline CostBaseline AccuracyOurs Cost at Same AccuracyCost ReductionOurs Accuracy at Same Cost
Router-R1 1k, cc0.38,0740.2194,89139%0.295
Router-R1 1k, cc0.58,3130.1924,33048%0.300
Router-R1 4k, cc0.116,9760.41515,01312%0.445
Router-R1 4k, cc0.517,3950.42715,7759%0.450
LFI llm4k-turnpen0.320,3840.43816,41019%0.483
AuthorsOpenReview ↗

W1, Q1. Conceptual or algorithmic advancement of our approach - 3

(3) Comparison with SWE-Protégé

SWE-Protégé [3] shares our broad goal of SLM-led expert collaboration, but our training objectives differ in two important respects. First, SWE-Protégé does not explicitly optimize whether-to-call behavior. Its second RL stage assigns a −10 penalty to trajectories without any expert calls, thereby encouraging the SLM to invoke the LLM advisor on every task. In contrast, our objective explicitly encourages the SLM to avoid unnecessary expert calls, encouraging to call the LLM advisor only for the problem that it can’t solve on its own. Second, SWE-Protégé primarily optimizes the SLM's ability to solve software engineering tasks through expert collaboration, rather than the performance–cost trade-off perspective of expert invocation. Its training objective does not directly penalize API token expenditure, and Its evaluation does not isolate the cost-effectiveness of the learned expert-invocation policy through comparisons even with problem-level random routing under matched API token budgets. In contrast, our method directly optimizes task performance against expert API expenditure, demonstrating high efficiency. We did not reproduce SWE-Protégé in our experiments because its training pipeline relies on SWE-specific expert-augmented trajectories and reward signals, requiring substantial task-specific adaptations that go beyond directly applying the method to our setting.

[1] Zeng, Hang et al. “Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLM.” (2026).

[2] Zhang, Haozhen, Tao Feng, and Jiaxuan You. "Router-r1: Teaching llms multi-round routing and aggregation via reinforcement learning." Advances in Neural Information Processing Systems 38 (2025): 141233-141265.

[3] Kon, Patrick Tser Jern, et al. "SWE-Prot'eg'e: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents." arXiv preprint arXiv:2602.22124 (2026).

Reviewer_Bjp6OpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper casts collaboration between a small language model (SLM) and a black-box LLM as budget-constrained information acquisition. The SLM may call the LLM once via an ask_advisor tool; the LLM sees only the query, and cost is a weighted token count (Eq. 3-4). Three RLVR stages train the SLM to decide whether to call, using labels from its own solve rate (Eq. 5-7); a separate model, Qmaker, to write a self-contained request plus a response-length instruction under a per-call output cap bb (Eq. 8-9); and the SLM again on full trajectories with an asymmetric reward (Eq. 10). Experiments on math and LiveCodeBench v6 compare against routing, cascading and step-level methods.

Critical strengths

S1. Large gains at low-to-moderate cost. In Fig. 4, at about 2-3k cost on AMC23 the method reaches about 0.80 while all baseline curves stay below about 0.70; on LCB medium at about 5k it reaches about 0.48 against at most about 0.40. Why this matters to my assessment: my concerns are about attribution, not existence.

S2. Unfavourable results are reported. C.4 says that under call-count cost "our method does not consistently outperform them" (l.1058). App. G, MATH500 Problem 287: "The efficiency gain in this case solely comes from the budget instruction" (l.2595-2596). Why this matters to my assessment: it makes the claims checkable.

Critical weaknesses

W1. The gain is not attributed to targeted queries. The abstract credits "targeted queries rather than delegating the reasoning process itself" (l.020-021). It is not separated from routing plus a length instruction, or from RL training of the SLM.

  • Fig. 5c/5d: at 4k/8k/16k, full-task requests are 94/97/96% of math queries and 69/85/77% of code queries. Problem 287 is a full-task query at budget 512.
  • No baseline pairs problem-level selection with the same instruction and cap. Routers and the Oracle use the LLM's Avg@k (App. E, l.1239-1241); "LLM-only + budget-instr" calls on every problem (l.357-358). §6.2 credits beating the Oracle to a "richer space of collaboration" (l.376-377).
  • Stage 1, Qmaker versus SLM-written queries, the instruction ss, the gate G(q)G(q) and the asymmetric reward are never removed.
  • Eq. 10 rewards λRtask\lambda R_{task} on trajectories without a call, so stage 3 also trains solo solving. On code, stage 3 adds up to 12.4 points while call rates change little (Fig. 9c/9d). The trained SLM's solo accuracy and the checkpoint behind the SLM-only and routing curves are not reported.
  • Fig. 4 has no error bars or repeats. Fig. 15's first row (training advisor, identical endpoints) shows about 0.88 on MATH500 at about 4.3k, above every point in Fig. 4. Fig. 7a gives LCB easy solo accuracy 0.41; Fig. 4 shows SLM-only at about 0.61.

Why this matters to my assessment: the central claim concerns the mechanism, which the evidence cannot yet isolate.

W2. Lack of novelty; the stated differentiator is not what is trained. The parts exist (RLVR tool use, l.122-124; GRPO, l.257). The paper calls Learning to Seek Help "closely related to our work in training an SLM to adaptively seek assistance from a stronger LLM" (App. B, l.807-808), says Router-R1 "learns adaptive model invocation using reinforcement learning" (l.837-838), and says SWE-Protégé "trains an SLM to consult a stronger LLM" (l.128-129). The claimed difference is the token-level API budget (l.801-802, l.842-843).

  • The average budget BB of Eq. 3 enters no reward (Eq. 7, 9, 10). Training enforces only a per-call cap bb (l.280); call frequency is set by γ\gamma (Eq. 6).
  • One Qmaker is trained per bb (l.318-319); nothing selects one for a target BB. "can make the joint optimization unstable" (l.215) is untested.
  • Eq. 1 has the SLM write the query; D.1 says Qmaker "rewrites this query" (l.1211-1212).
  • No RL-trained help-seeking method is a baseline (§6.1, l.356-366).

Why this matters to my assessment: without a budget-level objective or these comparisons, this is a training recipe for a known setting.

Questions for authors
  1. Can you add a router or Oracle that sends problems to the LLM with the same instruction and cap? Please also stratify by query type.
  2. What is the trained SLM's accuracy with the advisor disabled, per budget? Which checkpoint do the SLM-only and routing curves use?
  3. Are Fig. 4 and Fig. 15's first row the same checkpoint and evaluation? Please explain 0.41 versus about 0.61 on LCB easy, and tabulate all six budgets with intervals.
  4. What is the trained system's calling utility, as in Fig. 2c? In a GRPO group, a call decision matching n(x)n(x) earns at least 1 before the call and a mismatch at most λ\lambda, so for λ∈(0,1)\lambda \in (0,1) the task term cannot reverse that ranking.
  5. Will code and full hyperparameters be released? D.1 defers to "the default optimization settings of VeRL" (l.1205-1207).

One more Suggestion: Move the qualifiers of §6.2 (l.372, cost regime) and §8 (l.481-482, "non-universal" transfer) into the abstract. These parts are important to the paper.

12 comments
AuthorsOpenReview ↗

W1. Inconsistent results.

Fig. 4 has no error bars or repeats. Fig. 15's first row (training advisor, identical endpoints) shows about 0.88 on MATH500 at about 4.3k, above every point in Fig. 4. Fig. 7a gives LCB easy solo accuracy 0.41; Fig. 4 shows SLM-only at about 0.61. Are Fig. 4 and Fig. 15's first row the same checkpoint and evaluation? Please explain 0.41 versus about 0.61 on LCB easy, and tabulate all six budgets with intervals.

Thank you for raising this issue. As we stated in the official comment, we have made a mistake in selecting evaluation points for both figures. For whether-to-call analysis for code (Figures 8, which was Figure 7 before revision), we mistakenly used evaluation results from LCBv5, which were inconsistent with the LCBv6 results reported in Figure 4. Also, for advisor transfer results (Figure 16, which was Figure 15 before revision), we inadvertently selected results from a checkpoint without integration training, rather than the integration-trained model used for the main results in Figure 4.

We have corrected both figures and revised the corresponding discussions in the manuscript. Again, we sincerely appreciate finding and reporting this issue.

Additionally, we have added an error bar on figure 4.

Will code and full hyperparameters be released? D.1 defers to "the default optimization settings of VeRL" (l.1205-1207).

Yes. We plan to publicly release our training and evaluation code, along with the full hyperparameter configurations, to ensure reproducibility. We will also include the implementation details and hyperparameters for the reproduced baselines.

AuthorsOpenReview ↗

W1. Separating the gain from RL training of the SLM.

Eq. 10 rewards on trajectories without a call, so stage 3 also trains solo solving. On code, stage 3 adds up to 12.4 points while call rates change little (Fig. 9c/9d). The trained SLM's solo accuracy and the checkpoint behind the SLM-only and routing curves are not reported.

What is the trained SLM's accuracy with the advisor disabled, per budget? Which checkpoint do the SLM-only and routing curves use?

For the "SLM-only" baseline, we have used the SLM without further training.

To disentangle improvements in the SLM's standalone reasoning capability from gains attributable to advisor collaboration, we additionally evaluated each budget-specific RL-trained SLM checkpoint without access to the advisor and compared it against the original, untrained SLM-only baseline.

Math:

SLM (cap)MATH500AMC23AIME (25+26)
5120.7420.5310.079
10240.7340.5340.085
20480.7310.5340.083
40960.7300.5380.088
81920.7290.5470.065
163840.7260.5530.067
SLM-only anchor0.7210.5500.083

Code:

SLM (cap)easy (322)medium (382)hard (350)
5120.5690.1310.016
10240.5970.1700.036
20480.6210.1650.043
40960.6020.1580.026
81920.5960.1660.033
163840.6130.1640.029
SLM-only anchor0.5860.1900.051

The results show that RL training does not systematically improve standalone SLM performance. On mathematical reasoning benchmarks, the trained checkpoints achieve performance largely comparable to the original SLM, with only modest improvements on MATH500 (up to +2.1 percentage points) and no consistent gains on AMC23 or AIME. On coding benchmarks, the trained checkpoints show occasional improvements on easy problems, but consistently underperform the original SLM on medium and hard problems. These results suggest that the performance gains of our full framework cannot be explained simply by improvements in the SLM's standalone reasoning capability through RL training.

AuthorsOpenReview ↗

W1. The gain is not attributed to targeted queries. - 1

W1. The gain is not attributed to targeted queries. The abstract credits "targeted queries rather than delegating the reasoning process itself" (l.020-021). It is not separated from routing plus a length instruction, or from RL training of the SLM.

  • Fig. 5c/5d: at 4k/8k/16k, full-task requests are 94/97/96% of math queries and 69/85/77% of code queries. Problem 287 is a full-task query at budget 512.
  • No baseline pairs problem-level selection with the same instruction and cap. Routers and the Oracle use the LLM's Avg@k (App. E, l.1239-1241); "LLM-only + budget-instr" calls on every problem (l.357-358). §6.2 credits beating the Oracle to a "richer space of collaboration" (l.376-377).

Can you add a router or Oracle that sends problems to the LLM with the same instruction and cap? Please also stratify by query type.

To resolve the concerns, we report the router + budget-instruction+cap results on figure 19 in Appendix, and report the stratified results in stratify_query_code.pdf and stratify_query_math.pdf on supplementary.

First, we would like to clarify that QMaker frequently generating full-task queries is not inherently an undesirable behavior that our method aims to avoid. As explicitly stated in Line 53 of the Introduction, QMaker may delegate the entire problem when the budget permits. Our primary objective is to maximize performance under a given cost budget, rather than to explicitly enforce targeted subproblem queries. To this end, our framework enables explicit control over the advisor's response verbosity, while QMaker learns to adapt its querying strategy to the available budget. Importantly, our notion of targeted information acquisition is not limited to asking subproblem-specific questions, but more broadly refers to acquiring useful information in a budget-aware manner.

As shown in Figure 19, our method achieves higher cost efficiency even compared with routing baselines equipped with learned budget instructions from QMaker and an output-token cap, outperforming even the oracle with learned budget instructions on AMC23. We conducted experiments comparing our method against four baseline variants. Specifically, we considered two task delegation mechanisms: (1) a problem-level oracle delegator operating at the cost levels considered in our experiments, and (2) RouteLLM. For each delegation mechanism, we evaluated two response-length control strategies: (1) an output-token cap with a heuristic budget instruction, and (2) an output-token cap with a learned budget instruction obtained through QMaker training. This yields four baseline variants. Overall, our method achieves a better performance–cost trade-off than the other baselines, except for the oracle with learned budget instructions in some settings.

For the stratified results, our method achieves Pareto-frontier performance across all query types except the incomplete-query type. First, we grouped the rollouts by query type and evaluated SLM-only and LLM-only performance on the corresponding problem subsets. We then evaluated LLM-only with learned budget instructions and an output-token cap, as well as RouteLLM, oracle and our method, on the same subsets. We excluded subsets with fewer than 30 problems (n<30) due to high statistical uncertainty. The table below summarizes the results under a 1,024-token budget. We report the performance of ours, and RouteLLM’s performance matched our’s cost. For oracle+instr, when there exists no point on the matched budget, we denote the best performance it can achieve. In many cases, the problem-level routing methods like RouteLLM + instr and Oracle + instr often show unfavorable performance–cost trade-offs. Full results are in the supplementary.

Comments on the paper

1 comment

Dear reviewers and AC,

We are grateful that the reviewers recognized our work as addressing a timely and practically relevant problem (ZcFB, ahUB), with a well-structured training design (ahUB) and strong empirical improvements (Bjp6).

We also realized, as pointed out by the reviewers, that some parts of our original manuscript could lead to misunderstandings or would benefit from stronger empirical support, particularly regarding the meaning of "targeted."

In response to reviewers’ valuable feedback, we conducted extensive additional experiments and analyses. Below, we briefly summarize the key updates. Some have already been incorporated into the revised manuscript, while others are provided in our supplementary materials and individual responses, with plans to incorporate them into a future revision.

  • Clarification of "targeted": Slightly revised the manuscript to clarify the meaning of "targeted" in our method and avoid potential misunderstandings.
  • Comparison with closely related works: Added comparisons with Learning to Seek Help and Router-R1, demonstrating substantially stronger difficulty-aligned whether-to-call behavior and generally more favorable performance–cost trade-offs (Appendix B.2).
  • Accounting for local inference costs: Demonstrated that our method maintains favorable performance–cost trade-offs even after accounting for local inference costs using OpenRouter API pricing (Appendix C.4).
  • Evidence beyond simple task delegation: Provided additional analyses showing that the benefits of our method cannot be fully explained by simple task delegation:
    • Comparisons with task delegation under response-length constraints, including a problem-level oracle with learned budget instructions (Figure 19).
    • Stratified performance analysis across query types (stratify_query_{code,math}.pdf in the supplementary materials; also in our response to Bjp6).
    • Analysis of how frequently LLM responses directly contain the final answer (in our response to ZcFB).
  • Generalization: Demonstrated that QMaker generalizes across different API budgets, task domains, and unseen SLM architectures without retraining (Appendix C.8).
  • QMaker overhead: Showed that QMaker accounts for only approximately 10% of total inference latency (in our response to ahUB).
  • Comparison with simpler alternatives: Demonstrated that our trained QMaker substantially outperforms simpler alternatives, including SLM-generated queries and an untrained, prompt-based QMaker (untrained_qmaker_SLM-own-query.pdf in the supplementary materials; also in our responses to ahUB and Bjp6).
  • Extensive ablation studies:
    • Isolated the effects of SLM RL training from the gains of our collaboration framework (in our response to Bjp6).
    • Evaluated the contributions of individual components, including budget instructions, the invocation gate, and asymmetric rewards (in our response to Bjp6).
  • Motivation for stage-wise training: Demonstrated that joint optimization leads to unstable SLM behavior in our evaluated configurations, supporting our stage-wise training design (lagrangian_sweep.png in the supplementary materials; also in our response to Bjp6).
  • Utility of advisor invocation: Demonstrated that our trained system benefits substantially from advisor invocation, unlike its untrained counterpart (in our response to Bjp6).

We have also posted summary comments for individual reviewers to facilitate the review of our detailed responses and additional findings.

We would greatly appreciate it if the reviewers and AC could review these updates and let us know if any further clarification is needed. We would be happy to address any remaining questions during the discussion period.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.