acceptodds
Under review as a conference paper at ICLR 2027

BeaconAgent: Learning Capability-Aware Help-Seeking Policies for Small Language Model Agents

11 views

Abstract

Small language model (SLM) agents offer a resource-efficient alternative to large language model (LLM) agents, but their limited capabilities make them unreliable on challenging tasks. Recent work enables SLM agents to leverage stronger LLMs through self-reflection, confidence estimation, and learned routing. However, existing approaches mainly focus on assistance routing, with limited attention to the design of a fine-grained help-seeking policy. First, the critical step where assistance is most valuable may fail to be identified. Second, assistance decisions may not align with the SLM agent's capability boundary, causing unnecessary or insufficient help. Third, existing approaches often rely on delegation, increasing LLM involvement and reducing SLM autonomy. To address these challenges, we propose , which enables the SLM to execute the full agent trajectory while selectively requesting brief, targeted LLM guidance provided only as an observation. BeaconAgent learns this help-seeking behavior through two training stages. First, a teacher LLM identifies the causing trajectory failure, replaces it with an explicit ASK_HELP step, and completes the remaining trajectory with guidance. The resulting successful trajectories are used for supervised fine-tuning, teaching the SLM agent . Second, capability-aware reinforcement learning aligns help seeking with the SLM agent's capability boundary, promoting assistance when needed while reducing unnecessary help otherwise, thereby teaching the SLM agent . We evaluate BeaconAgent on math reasoning, retrieval-augmented QA, and unseen coding tasks. Results show that BeaconAgent improves task performance while reducing unnecessary LLM usage, achieving a better performance–cost trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Small Language Model Agents, SLM–LLM Collaboration, Help-Seeking Policy, Capability-Aware Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WTK6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces BeaconAgent, a technique enabling collaboration between Small Language Models (SLMs) and Large Language Models (LLMs). BeaconAgent allows SLM agents to collaborate with more capable LLMs while retaining their role as primary executors. By combining "failure-driven supervised fine-tuning (SFT)" with "capability-aware reinforcement learning (RL)," the system learns where, how, and when to seek assistance through mechanisms such as critical error localization, concise and targeted guidance, and capability-matched help-seeking. Across eleven benchmarks—including unseen code generation—BeaconAgent optimizes the trade-off between performance and cost by minimizing unnecessary LLM interventions while maintaining excellent task performance.

Critical strengths

This paper focuses on the critical and pressing topic of collaboration between SLMs and LLMs. The proposed BeaconAgent method was evaluated across 11 benchmarks, including unseen code generation. The results demonstrate that BeaconAgent can reduce unnecessary calls to LLMs while maintaining excellent task performance.

Critical weaknesses

Existing literature already contains studies addressing the same or similar problems—such as "When2Ask," published at the 2024 Reinforcement Learning Conference (RLC). However, the authors of the paper in question fail to provide a literature review; they neither mention these related studies nor compare BeaconAgent with those methods.

The content of Section 2 (especially Section 2.2) is difficult to understand. It is recommended to refine the text and clarify the meanings of the mathematical symbols.

Questions for authors

See the above "Critical Weaknesses" Section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer WTK6

Thank you for your careful review and valuable suggestions. We particularly appreciate your recommendation of When2Ask (Hu et al., 2024), which is an important and valuable study on learning when an agent should interact with an LLM.

(1) Relation to When2Ask. We believe that When2Ask is an important and valuable contribution to learning efficient agent–LLM collaboration, and it provides meaningful insights for our research. We sincerely appreciate the reviewer for bringing this work to our attention. In the revised manuscript, we will cite and discuss When2Ask in the Related Work section and provide a comparison between When2Ask and BeaconAgent.

When2Ask focuses on embodied intelligence scenarios and introduces a Planner–Actor–Mediator architecture, in which a PPO-trained asking policy determines when to request high-level plans from an LLM based on environmental observations. Its experiments in MiniGrid and Habitat demonstrate that learning when to interact with an LLM can effectively reduce unnecessary interactions while maintaining task performance.

BeaconAgent addresses SLM–LLM collaboration in the context of language model agents performing complex reasoning and tool-use tasks. Its key innovation lies in jointly learning where to seek help, how to request targeted assistance, and when to seek help according to the SLM's evolving capabilities. Specifically, failure-driven SFT enables the SLM to identify critical errors and formulate targeted help requests, while capability-aware GRPO aligns help-seeking decisions with its capability boundary. Moreover, BeaconAgent retains the SLM as the primary executor throughout task execution, with the LLM providing only brief, targeted guidance, thereby improving task performance while minimizing reliance on the LLM.

We believe that When2Ask and BeaconAgent represent complementary research efforts that explore efficient agent–LLM collaboration in different application scenarios and from different methodological perspectives. We highly appreciate the contributions of When2Ask and sincerely thank the reviewer for this valuable suggestion, which helps us better position our work within the existing literature.

(2) Clarity of Section 2, particularly Section 2.2. Thank you for pointing out the difficulty in understanding the methodology. We agree that the presentation of Section 2.2 can be improved. In the revised manuscript, we will reorganize this section, explicitly define the mathematical symbols when they first appear, and provide clearer explanations of the corresponding equations. In particular, we will clarify the capability-label construction and dynamic relabeling procedures, the meanings of the rollout-group statistics and reference help-request positions, and the respective roles of the task-outcome reward, repeated-request penalty, and critical-step alignment reward. We will also explain the reward calculation and clipping procedure step by step to make the overall optimization process easier to follow and reproduce.

We sincerely thank the reviewer again for highlighting these important related works and presentation issues. These suggestions will help us better position our contribution and improve the clarity of the paper.

We sincerely thank the reviewer for the valuable comments and constructive suggestions. We have carefully revised our paper accordingly and resubmitted the updated version. The major revisions are summarized as follows:

(1) Improving the readability of Section 2.2. We have revised Section 2.2 to improve its clarity and readability, providing clearer explanations of the mathematical notations, training procedures, and the overall logic of our proposed method.

(2) Adding discussion and experiments on When2Ask. We have expanded the Related Work section to cite, discuss, and compare When2Ask with BeaconAgent, clarifying the differences in their approaches to learning help-seeking behaviors. Furthermore, we have conducted additional experiments by adapting When2Ask to our experimental setting and comparing it with BeaconAgent across all eleven benchmarks. The results show that When2Ask is a strong baseline, achieving a macro-average accuracy of 53.34% and outperforming all three baselines reported in the main paper, namely AdaSwitch (47.80%), R2vAgent (53.32%), and STEER (48.59%). Nevertheless, BeaconAgent further improves the average accuracy to 57.55%, outperforming When2Ask on ten of eleven benchmarks. Moreover, BeaconAgent provides brief and targeted assistance, with only 49.39 LLM output tokens per assisted trajectory. These results demonstrate the effectiveness of BeaconAgent in achieving stronger task performance through fine-grained help seeking and minimal LLM assistance.

We sincerely appreciate the reviewer's feedback, which has helped us improve the quality and clarity of our manuscript. We look forward to the reviewer's further comments and response.

Thank you for the author's response; it addresses my primary concerns. I have updated my rating to "weak acceptance".

We sincerely thank the reviewer for reconsidering our work and for increasing the score. We greatly appreciate your constructive feedback and thoughtful evaluation, which have helped us further improve the quality of our paper.

Reviewer_ZDkCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on the issue of SLM leveraging LLM to enhance performance. The authors propose BeaconAgent, whose core idea is to construct training data containing ask_help to teach SLM where and how to ask questions, while using RL to train SLM on whether to ask questions. Experiments were conducted on Qwen3-4B, showing improvements over AdaSwitch, R2vAgent, and STEER. Overall, the main contribution of this work is proposing a post-training approach to enhance SLM's ability to request LLM assistance.

Critical strengths

This work focuses on leveraging the capabilities of large language models (LLMs) to enhance the performance of small language models (SLMs). This issue is indeed significant in various practical AI application scenarios, as it requires balancing the trade-offs between effectiveness and cost. Compared to existing approaches, it proposes a novel framework for constructing post-training data and reinforcement learning, making the problem itself quite research-worthy.

Critical weaknesses

The current experiments have only verified performance in QA, Math, and Coding. Whether this construction system can function in more complex real-world scenarios, such as customer service systems, remains questionable.

Questions for authors

Can this solution still be applied to more complex real-world application scenarios, such as customer service scenarios and long-horizon task execution?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZDkC

We sincerely thank the reviewer for recognizing the value of our work and providing this valuable comment. We agree that validating the applicability of BeaconAgent to customer service scenarios and long-horizon task execution is important.

We conducted additional experiments on the original τ-bench, which evaluates agents in complex customer service scenarios involving long-horizon task execution, multi-turn user interactions, sequential tool calls, dynamic information gathering, and state-dependent decision-making. We evaluated all 165 test tasks, including 115 retail and 50 airline customer service tasks, using Qwen3.7-Flash as the user simulator. Notably, BeaconAgent is directly evaluated using the original trained checkpoint, without any additional training on customer service data.

MethodRetail (%)Airline (%)
Qwen3-4B23.5032.00
AdaSwitch24.3034.00
R2vAgent25.2036.00
STEER23.5034.00
BeaconAgent (Ours)26.9638.00

BeaconAgent achieves the highest task success rates in both domains, outperforming the original Qwen3-4B by 3.46 and 6.00 percentage points, respectively, while consistently outperforming all three collaboration baselines.

The key innovation of BeaconAgent lies in its ability to jointly learn where, how, and when to seek help, while retaining the SLM as the primary executor throughout task execution. Through failure-driven SFT and capability-aware GRPO, BeaconAgent enables SLM agents to identify critical errors, request targeted guidance, and make assistance decisions according to their evolving capabilities. This design is independent of specific task domains, allowing SLM agents to improve task performance while minimizing their reliance on LLMs.

The additional experimental results on τ-bench further demonstrate that BeaconAgent is applicable and effective in both customer service scenarios and long-horizon task execution, validating the cross-domain generalizability of our proposed framework.

We sincerely thank the reviewer again for this valuable suggestion, which has helped us further strengthen the empirical validation of our framework.

Reviewer_yg3vOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes BeaconAgent, a framework that trains small language model agents to request targeted assistance from stronger models while remaining the primary executor. The method combines supervised fine-tuning on trajectories repaired at their first critical error with capability-aware GRPO. Experiments across mathematical reasoning, retrieval-augmented QA, and code generation report improved accuracy and reduced teacher output-token consumption. However, the experiments do not sufficiently isolate the benefit of the proposed training components.

Critical strengths
  • The paper addresses a practically relevant problem: improving small-agent reliability while limiting stronger-model usage.

  • The complete system achieves encouraging results across multiple tasks, including transfer to an unseen coding domain.

  • The comparisons between random and critical-error intervention positions, and between brief guidance and direct-step execution, provide useful evidence about intervention placement and assistance format.

Critical weaknesses
  • The incremental benefit of the proposed RL design is unclear. The comparison between SFT-only and full BeaconAgent demonstrates the benefit of the complete RL stage, but does not establish whether its particular reward design improves over ordinary RL training. A crucial missing baseline is the same SFT checkpoint trained with the same GRPO pipeline and assistance interface using a simple correctness reward and help-request penalty. There are also no component ablations isolating capability conditioning, dynamic relabeling, and critical-step alignment. Consequently, it is difficult to determine whether the complicated reward construction is necessary.

  • The methodological distinction from related work needs stronger justification. The SFT stage resembles existing failure-driven trajectory repair and distillation approaches. AdaSwitch trains on small-agent mistake-checking and correction trajectories, while Agent-R constructs recovery trajectories using error localization. The RL motivation also overlaps with Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Models, which conditions assistance penalties on whether the small model can answer independently. The paper should explain its incremental contribution more precisely and experimentally distinguish its design from these alternatives.

  • The SFT construction does not directly validate student recovery from minimal guidance. During data generation, the teacher supplies the hint and generates the remaining trajectory. Verifying the teacher’s final answer establishes that the constructed demonstration succeeds, but does not establish that the hint is sufficient for the student to recover. Table 2 supports critical-error placement over random placement, but does not fully separate the benefits of learning help-seeking behavior from distilling successful teacher continuations.

  • Teacher output tokens alone provide an incomplete efficiency evaluation. Reduced teacher output length is encouraging, but end-to-end execution also includes processing the teacher’s input context, additional interaction rounds, and subsequent student generation. Without latency and total token measurements, the practical efficiency advantage remains insufficiently characterized.

  • Some implementation details remain underspecified. Appendix A.6 describes the basic ask_help interface, but the exact assistance prompts, generation limits, and enforcement of restrictions on final answers and full solutions are unclear. The appendix also provides reward preference orderings without fully specifying the numerical base rewards and outcome-clipping bounds. These omissions make the reward design difficult to reproduce and assess.

References

  • Hao Sun, Jiayi Wu, Hengyi Cai, Xiaochi Wei, Yue Feng, Bo Wang, Shuaiqiang Wang, Yan Zhang, and Dawei Yin. 2024. AdaSwitch: Adaptive Switching between Small and Large Agents for Effective Cloud-Local Collaborative Learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8052–8062. Miami, Florida, USA: Association for Computational Linguistics. DOI: 10.18653/v1/2024.emnlp-main.458. ACL Anthology

  • Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. 2025. Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training. arXiv preprint, arXiv:2501.11425. DOI: 10.48550/arXiv.2501.11425. arxiv.org

  • Hang Zeng, Xiangyu Liu, Yong Hu, Chaoyue Niu, Jiarui Zhang, Shaojie Tang, Fan Wu, and Guihai Chen. 2026. Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Models. arXiv preprint, arXiv:2604.17827v1, April 20, 2026. DOI: 10.48550/arXiv.2604.17827.

Questions for authors
  • Can the authors compare against an otherwise matched GRPO baseline using R=1[correct]−λNhelp,R=\mathbf{1}[\text{correct}]-\lambda N_{\mathrm{help}}, sweeping λ\lambda to compare accuracy–cost curves? This is the most important missing experiment.

  • What are the individual contributions of capability conditioning, dynamic relabeling, and critical-step alignment? Why should matching an exact reference step index be rewarded when different trajectories can reach equivalent reasoning states at different positions?

  • How does performance change when the student, rather than the teacher, generates the continuation after receiving the hint during SFT data construction? How does the method compare with ordinary teacher-repaired trajectory distillation?

  • Can the authors include an oracle-assisted reference that identifies the first recoverable critical error using hindsight and lets the teacher complete the remainder? An oracle-timed hint followed by student continuation would additionally help separate intervention-timing errors from assistance-use errors.

  • Can the authors report end-to-end latency, teacher-call counts, and input/output tokens for both models under comparable serving conditions, and provide the complete assistance prompts and numerical reward configuration?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yg3v

We sincerely thank the reviewer for recognizing the value of our work and providing insightful feedback. We address the concerns below.

(1) Comparison with standard GRPO. We compare matched GRPO baselines (same post-SFT checkpoint, pipeline, and assistance interface) with reward = correctness − λ·N_help.

MethodλAcc.% ↑Easy Ask%Hard Ask%Gap (pp)LLM Tokens ↓
GRPO+Penalty058.0086.594.07.5176.69K
GRPO+Penalty0.0557.4077.588.511.0162.50K
GRPO+Penalty0.1056.8069.082.013.0147.82K
GRPO+Penalty0.2055.9055.569.514.0122.36K
GRPO+Penalty0.3054.5038.051.513.587.61K
GRPO+Penalty0.5053.058.019.011.026.43K
BeaconAgent—57.5540.4475.1634.73116.12K

Accuracy and LLM output tokens cover 11 benchmarks; ask metrics cover MATH-500, HotpotQA, 2Wiki, Gaokao, and HumanEval.

BeaconAgent achieves a better accuracy–cost trade-off than GRPO+Penalty at λ=0.20 (57.55% vs. 55.90%; 116.12K vs. 122.36K tokens) and λ=0.10 (57.55% vs. 56.80%; 21.4% fewer tokens). Its ask-rate gap (34.73 pp vs. at most 14.0 pp) demonstrates more capability-aligned help seeking beyond uniform penalty control.

(2) Design motivations of the three components.

  • Capability conditioning: A single rollout may unreliably estimate the SLM's capability due to sampling variability. We therefore use four no-help rollouts to obtain more reliable capability labels.
  • Dynamic relabeling: As the SLM improves during RL, static capability labels may encourage unnecessary help seeking. Dynamic relabeling aligns reward preferences with the SLM's evolving capabilities.
  • Critical-step alignment: Help seeking at critical errors should be rewarded because timely intervention prevents error propagation. Table 2 supports this motivation: Critical-Error SFT improves Hit@0 by 14–24 pp and accuracy by 0.2–14.1 pp over Random-SFT. We use a weak auxiliary reward (+0.05/−0.05) rather than a hard constraint.

These components are supporting mechanisms rather than independent innovations. Our core contribution is a unified two-stage training framework that jointly learns where, how, and when an SLM agent should seek targeted assistance, improving performance while minimizing LLM involvement.

(3) Novelty relative to related work. We appreciate these valuable references and will cite, discuss, and compare them in the revision. Unlike AdaSwitch, which relies on LLM step correction, BeaconAgent requests brief, targeted guidance, reducing LLM involvement while preserving SLM autonomy. Unlike Agent-R, which focuses on self-reflection and error recovery, BeaconAgent leverages external assistance when errors exceed the SLM's recovery capabilities. While Learning to Seek Help primarily addresses when to seek assistance, BeaconAgent additionally learns where and how to seek assistance during multi-step execution. Jointly learning where, how, and when to seek help enables more fine-grained and cost-effective SLM–LLM collaboration.

(4) Student recovery and oracle experiments.

Our SFT stage needs correct execution trajectories. Since student continuations may remain incorrect even with guidance, we use teacher continuations with correctness verification to ensure reliable supervision. To validate student recovery, we evaluate help-seeking success on beyond-capability tasks:

BenchmarkSuccess (%)
2Wiki12.9
HotpotQA18.67
MuSiQue17.92
NQ27.71
Bamboogle34.88
MATH30.3
Gaokao21.2
AIME26.32
OlyMath25
Omni-MATH25.9
HumanEval60
Macro-average27.35

These results demonstrate the SLM's ability to recover from brief guidance.

Additionally, we conducted oracle-assisted experiments across 11 benchmarks, using GPT-6.1-Sol to retrospectively identify the first recoverable critical error:

MethodAvg. Acc. (%)
Oracle+Teacher69.70
Oracle+Student59.60
BeaconAgent57.55

Both oracle settings use the same hint type, followed by teacher or student continuation. BeaconAgent is only 2.05 pp below Oracle+Student, supporting its effective intervention timing.

(5) Efficiency and reproducibility. Teacher output+input tokens per assisted trajectory are AdaSwitch: 403+1,809; R2vAgent: 3,628+2,060; STEER: 663+5,693; and BeaconAgent: 49+1,566=1,615. BeaconAgent reduces total teacher token consumption by 27.0%, 71.6%, and 74.6%, respectively, with 1.03 teacher calls per assisted trajectory. Combined with the highest average accuracy (57.55%), these results demonstrate a superior performance–cost trade-off. On 2×A100 (80GB), with one GPU each for the SLM and LLM, teacher latency is 4.1 s/request and end-to-end latency is 18.7 s/task, demonstrating efficient execution under our deployment configuration. We will disclose complete assistance prompts, generation limits, restriction enforcement, numerical rewards, and clipping bounds in the revised appendix.

We sincerely thank the reviewer for the valuable suggestions. We have revised our paper and submitted it. In the revised paper, we have expanded the Related Work section to include, discuss and compare three closely related studies: Agent-R, Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Models, and AdaSwitch. We further clarify the distinctions and incremental contributions of BeaconAgent as follows.

(1) Comparison with Agent-R. Agent-R employs Monte Carlo Tree Search (MCTS) to identify errors in failed trajectories and construct revision trajectories by combining erroneous prefixes with successful continuations, enabling agents to learn self-reflection and error correction through iterative self-training. In contrast, BeaconAgent focuses on learning when an SLM agent should seek targeted external assistance rather than solely improving its self-correction ability. Specifically, we identify the first recoverable critical error and introduce an explicit ASK_HELP action, allowing the SLM to obtain brief, targeted guidance while retaining control of subsequent execution. Furthermore, capability-aware reinforcement learning aligns help-seeking decisions with the SLM's evolving capability boundary.

(2) Comparison with Learning to Seek Help. This work trains SLMs through end-to-end online reinforcement learning to learn when to consult LLMs and how to formulate requests, while enabling LLMs to provide adaptive feedback by either clarifying underspecified requests or returning relevant information. In contrast, BeaconAgent focuses on failure-driven, fine-grained assistance during agent execution. Rather than learning collaboration primarily through iterative request–feedback interactions, BeaconAgent explicitly identifies the first critical error in failed trajectories and uses teacher-verified interventions to supervise where and how assistance should be requested. Furthermore, capability-aware GRPO aligns help-seeking decisions with the SLM's evolving capability boundary, distinguishing tasks that require external assistance from those that can be solved autonomously. This design provides two complementary advantages for our target setting: it anchors assistance to the earliest recoverable failure point, preventing subsequent error propagation, and provides only brief, targeted guidance as an observation, allowing the SLM to retain control of the entire execution trajectory. Moreover, BeaconAgent extends this failure-driven collaboration paradigm across mathematical reasoning, retrieval-based QA, and code generation.

(3) Comparison with AdaSwitch. AdaSwitch enables small agents to identify problematic reasoning steps through self-reflection and adaptively switch to stronger agents for correction. BeaconAgent adopts a different collaboration paradigm: the SLM executes the full agent trajectory and receives only brief, targeted LLM guidance as an observation when necessary, rather than delegating the execution or correction of subsequent steps to the stronger agent. Combined with first-critical-error supervision and capability-aware reinforcement learning, BeaconAgent learns where, how, and when to seek assistance while preserving SLM autonomy. We have also included AdaSwitch as an empirical baseline and evaluated it under our experimental setting, providing a direct quantitative comparison in the main experiments.

Additional implementation details. We have further expanded Appendix A.11 to disclose the complete help prompts, generation-token limits, and constraints imposed on teacher-generated assistance, including restrictions against providing final answers, complete solutions, or full derivations. Regarding the agent execution budget, Section 3.1 (Experimental Setup) explicitly specifies that the maximum trajectory length is limited to seven agent steps.

We hope these revisions and additional clarifications adequately address the reviewer's concerns. We sincerely appreciate the reviewer's constructive feedback and look forward to further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.