acceptodds
Under review as a conference paper at ICLR 2027

Sibyl: An Efficient Small-large Model Collaboration Framework for Long-horizon Tasks

10 views

Abstract

Small language models (SLMs) offer a promising foundation for on-device agents through low-latency, resource-efficient inference, yet limited reasoning and planning capabilities constrain their performance on long-horizon tasks requiring multi-step interaction with the environment. Step-level collaboration between SLMs and larger cloud-hosted models can bridge this gap, but identifying states that warrant cloud assistance remains challenging: the contribution of each cloud call is entangled with subsequent actions and can be assessed only from the final task outcome. Compounding this challenge, the SLM must balance two competing objectives: maximizing task success and minimizing cloud calls. To address this, we propose Sibyl, an algorithm that trains SLM agents to selectively consult cloud models at the step level and internalize their guidance for subsequent decisions, achieving strong task performance with minimal cloud reliance. Sibyl follows a three-stage training pipeline that (1) builds a robust base policy through consultation-free self-evolving reinforcement learning (RL); (2) cold-starts consultation behavior via decisive-disagreement state mining; and (3) jointly optimizes consultation decisions and guidance internalization through consultation-aware RL. Experiments on ALFWorld and WebShop demonstrate that Sibyl, using only a 0.6B-parameter model, outperforms state-of-the-art baselines, including agent training and routing methods, by 95.2% and 80.4% in success rate while averaging only 0.8 and 3.9 cloud calls per trajectory, respectively.

Primary area:
reinforcement learning
Keywords:
Agentic reinforcement learning, large-small model collaboration, long-horizon agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_A9b4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies collaboration between a small on-device model and a large cloud model on long-horizon agentic tasks. The paper proposes Sibyl, which treats call_cloud as an action of the small model itself. At each agentic step, the action chosen by the small model can be either a task action or call_cloud, and if it chooses call_cloud, the large cloud model selects the action for the current step on its behalf. Sibyl learns when to call_cloud through a three-stage training procedure, (i) it first trains the small model into a fairly strong standalone agent with RL and OPD, without allowing it to call_cloud, (ii) it then identifies the states where replacing a single action with the cloud model's action turns a failure into a success, and uses them for SFT so that the small model learns call_cloud in these states, and (iii) it finally continues RL with call_cloud, using comparisons among matched states to judge whether a call is worthwhile, while distilling the cloud model's guidance into the small model to reduce future calls. Using small Qwen3 models as the local models and Qwen3.7-Max as the cloud model, the paper compares with agent training methods and routing methods on ALFWorld and WebShop, and claims higher success rates with very few cloud calls.

Critical strengths

None, I cannot identify any strength of this paper. The key idea of the paper is highly similar to uncited prior work, so the novelty is incremental. The experiments can barely support the claimed contributions. They are not only insufficient, but, judging from the results, the experimental setup and implementation also seem unreasonable. When the cloud call ratio is higher, the success rate is lower, which seems to suggest that the cloud model performs even worse than the fine-tuned small model. Therefore, the paper does not seem to have any strength in terms of novelty, contribution, or experiments.

Critical weaknesses
  1. The key idea of this paper is highly similar to [1], yet the paper neither cites nor discusses it. [1] has already proposed letting a small model treat asking the cloud for help as an action in its own policy, training it with RL to decide when to ask for help, while limiting the number of cloud calls and without requiring an external router. This is exactly the first contribution claimed in this paper. The only difference is that this paper replaces single-turn reasoning with multi-step agentic tasks, but in essence the idea, the contribution, and the method are almost the same. The idea of letting the model itself learn when to defer to a stronger model, without an external router, has also been explored in [2][3], neither of which is cited. In addition, among the three training stages designed in this paper, Stage 1 largely comes from SEED, and the matched-state grouping in Stage 3 is also similar to GiGPO. Therefore, the novelty is incremental.

[1] Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training. ICML 2026.

[2] Learning to Decode Collaboratively with Multiple Language Models. ACL 2024.

[3] Learning to Route LLMs with Confidence Tokens. ICML 2025.

  1. I have serious concerns about the correctness of the experimental implementation, and the experimental results cannot show that routing is really useful. (i) Asking for help brings almost no benefit. The cloud call ratio of Sibyl (Qwen3-1.7B) is only about 3% on both benchmarks, so the improvement mainly comes from the three-stage training rather than from asking for help. Sibyl (Qwen3-0.6B) hands 64.8% of its steps on WebShop to the cloud, yet its success rate (80.4%) is on par with GiGPO w/SFT (80.6%), which does not use the cloud at all at inference. Therefore, I find that for Sibyl (Qwen3-1.7B) with 3% call ratio, training is the key, and for Sibyl (Qwen3-0.6B) with 64.8% call ratio, calling for help brings no benefit. Hence, the additional benefit of asking the cloud for help is not real or at least not demonstrated. (ii) The setup of the routing baselines looks problematic. RouteLLM hands 85.5% of its steps on WebShop to the cloud, yet its success rate is only 22.8%, far below the 46.4% of using the cloud alone. This makes it look as if the cloud model performs even worse than the fine-tuned small model.

  2. Training the proposed method requires fairly strong assumptions and costs. (i) The decisive-disagreement mining in Stage 2 requires restoring the environment to the exact state at a given step, replacing one step, and re-running the rest. This is only feasible in resettable simulated environments such as ALFWorld and WebShop, and is hard to achieve in real web or device environments. (ii) Training depends on many hyperparameters. The paper neither describes the tuning process nor shows the cost of tuning, including the learning rates of the three stages, αstep\alpha_{\mathrm{step}}, αcf\alpha_{\mathrm{cf}}, βopd\beta_{\mathrm{opd}}, and λopd\lambda_{\mathrm{opd}}.

  3. The objective and the training algorithm are disconnected, and the mechanism for limiting cloud calls is unclear. (i) The paper formulates a cost-regularized objective (Eq. 4) with a trade-off coefficient λ\lambda, but λ\lambda never appears in the training algorithm, and the paper only states that tuning αcf\alpha_{\mathrm{cf}} provides an implicit control over this trade-off. (ii) The reward contains no cost term. The consultation-specific advantage only compares the returns of calling and not calling at matched states, so it is unclear why this design limits cloud calls. (iii) Nothing in the method prevents excessive reliance on the cloud, e.g., Sibyl (Qwen3-0.6B) hands 64.8% of its steps on WebShop to the cloud.

Questions for authors
  1. In the experimental setup, the local models are only as small as 0.6B and 1.7B, while the cloud model is Qwen3.7-Max. Is this setup reasonable? Why are there no other combinations, such as larger local models (e.g., Qwen3-4B or Qwen3-8B), other model families, or different cloud models?
  2. After the three stages of training, the small model seems to have already learned how to solve these tasks, with a call ratio of only about 3% for the 1.7B model. If call_cloud is disabled at inference, what is the success rate of the small model? Is it possible that the small model achieves a higher success rate than the large model? What if the large model generates some reasoning or a plan once before the task starts and then feeds it to the small model?
  3. The ablation study stacks Stages 1, 2, and 3 sequentially. What happens if Stages 1 and 2 are skipped and Stage 3 is learned directly? The ablation study needs to be more comprehensive and in-depth.
Reviewer_jNvUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Sibyl, a framework for small and large language model collaboration on long-horizon agent tasks. The main idea is to let the small model decide when to call a cloud model, instead of using a separate router.

The method has three training stages. First, it improves the standalone SLM by self-evolving RL. Second, it finds important states where one cloud action can change a failed trajectory into a successful one. Third, it jointly trains the local actions and consultation decisions, while distilling the cloud guidance into the SLM.

The authors evaluate on ALFWorld and WebShop. A 0.6B model achieves 95.2% and 80.4% success, with 0.8 and 3.9 cloud calls per episode.

Critical strengths
  1. I think this paper studies an important practical problem. Treating cloud consultation as part of the agent policy is a reasonable design for long-horizon tasks. I also like the decisive-disagreement state mining idea, since it tries to identify states where cloud assistance is actually useful, rather than only predicting model uncertainty.
  2. The empirical results are quite strong, especially on ALFWorld. The paper also includes cloud cost comparison, different consultation budgets, and ablation studies. These experiments make the practical benefit of the proposed framework more convincing.
Critical weaknesses
  1. I am not fully convinced by the cloud cost optimization formulation. Eq. 4 introduces an explicit cloud cost penalty, but the actual training objective in Section 4.4 does not directly optimize this term. Also, matching states using observations and available actions may not always give a fair counterfactual comparison, because the interaction histories can be different. I think the connection between the proposed credit and actual contribution of consultation should be better justified.
  2. It is not completely clear how much improvement comes from the collaboration itself. In Figure 4, Stage 1 already achieves 90.0% success on ALFWorld, while the final result is around 95%. On WebShop, the 0.6B model still sends 64.8% of steps to the cloud, and its final success is close to the strongest cloud-assisted training baseline in Table 1. I would like to see more analysis of when the additional collaboration is really necessary.
Questions for authors
  1. Can the authors compare Sibyl with a standalone SLM under matched total training cost and inference cost? How much of the final gain is from consultation, rather than the stronger policy learned in Stage 1?

  2. Why is the cloud invocation cost not included explicitly in the final RL reward? Have the authors tried directly penalizing each cloud call?

  3. For the matched-state credit in Section 4.4, how do the authors handle two trajectories with the same observation but different interaction histories? Could this introduce bias into the consultation advantage?

Reviewer_k95UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sibyl, a small–large model collaboration framework for long-horizon agent tasks that aims to improve task success while reducing cloud API usage. Sibyl treats cloud consultation as an action within the small model's policy, eliminating the need for a separate router. The framework adopts a three-stage training pipeline: (1) consultation-free self-evolving RL to strengthen the small model, (2) decisive-disagreement state mining to initialize consultation behavior, and (3) consultation-aware RL with guidance internalization to jointly optimize local actions and cloud calls. Experiments on ALFWorld and WebShop show that Sibyl achieves strong task success rates with substantially fewer cloud calls than existing collaboration baselines.

Critical strengths
  1. The paper addresses an important problem of efficient small–large model collaboration for long-horizon agent tasks.

  2. The three-stage training pipeline is clear and coherent, with an intuitive approach to learning when to consult cloud models.

  3. Sibyl achieves competitive success rates on ALFWorld and WebShop while substantially reducing cloud calls and API costs.

Critical weaknesses
  1. Small–large model collaboration has been extensively studied, and prior works such as Hera and AlphaRouter have already explored fine-grained collaboration and RL-based routing. In particular, Hera addresses step-level device–cloud coordination for long-horizon agents, which is very close to the setting of this paper. While Sibyl integrates consultation into the SLM policy and introduces additional training strategies, it is unclear what fundamentally new insights or techniques it offers beyond these existing approaches. The authors should better clarify the key methodological differences and contributions.

  2. The consultation-aware credit assignment is not well justified. The proposed method matches states based on their observations and available actions, but these may not fully capture the decision context in long-horizon tasks. Moreover, comparing the average returns of cloud-assisted and local trajectories does not necessarily isolate the contribution of cloud consultation. It is unclear how reliable this advantage estimate is, especially when the matched trajectories have different histories and subsequent actions.

  3. The paper formulates a cost-regularized objective, but the actual training objective in Section 4.4 does not include a cloud-cost penalty. Instead, the authors rely on consultation-aware credit assignment and guidance internalization to reduce cloud usage indirectly. It is unclear why this should consistently discourage unnecessary cloud calls. A comparison with explicit cost-regularized RL would help justify this design.

Questions for authors
  1. While the paper provides stage-wise ablations, it is still unclear how much each component contributes to the final performance. In particular, a simple baseline that adds call_cloud to the action space and trains with standard GRPO would be useful. The authors should also evaluate the effects of consultation-specific credit assignment and guidance internalization separately.
  2. How frequently do matched states occur during training? Since the counterfactual advantage requires both cloud-assisted and local continuations from matched states, what proportion of consultation actions actually receive a nonzero counterfactual advantage?
  3. Why is cloud cost not explicitly included in the final RL objective? Have the authors compared their approach against directly penalizing call_cloud actions during RL? It would be interesting to see whether explicit cost regularization achieves a similar success–cost trade-off.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.