acceptodds
Under review as a conference paper at ICLR 2027

Lifting Cheaper Models: Self-Evolving Routing for Cost-Efficient Agent Services

Abstract

As agent services built on large language models (LLMs) move into production, their cost accrues with every request, much of it the premium for top models. Model routing lowers this premium by serving a request with a cheaper model whenever one matches the model the user asked for, but it is confined to the models already in the pool. When every cheaper model fails a recurring task, the router can only pay for the expensive one or accept worse results, and its savings stop at the capability boundary of the cheaper models. We observe that these failures are rarely random: a cheaper model tends to break at the same step across tasks of the same kind, and the execution history already records how a stronger model gets past it. We introduce RouteSage, a self-evolving router that edits its own routing table through capability patches, validated entries that let a cheaper action serve a family of tasks on behalf of the requested model. A direct patch reroutes the family to a cheaper model that already suffices. A guided patch pairs a cheaper model that falls short with an Experience, a reusable procedure distilled from past executions, creating an action the pool never had without retraining any model. Direct patches recover savings that existing models allow, and guided patches lift the models that fall short. Across web interaction, tool use, and code generation, RouteSage exceeds requested-model quality in every domain while lowering total online cost, including the cost of building and validating every patch, and on MiniWoB++ it cuts online cost by 29.6% and raises success by 3.4 points. Served from a frozen state, it delivers the highest quality of any method in every setting and cuts cost by 57.4% when GPT-5.6-sol is requested on MiniWoB++, whereas no baseline saves more than 5.2% without losing quality.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Agent Routing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_XtHbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RouteSage, a self-evolving routing framework that reduces LLM agent costs by augmenting cheaper models with reusable Experiences extracted from execution histories. It introduces Capability Regions, quality-constrained routing, and validated Capability Patches. Experiments across three agent benchmarks demonstrate promising cost reductions and quality improvements.

Critical strengths
  1. The paper addresses the important problem of reducing inference costs in LLM-based agent services while maintaining task quality. This problem is highly relevant to real-world deployments, where repeated model calls can lead to substantial operational costs.

  2. RouteSage integrates experience reuse, quality-constrained routing, and capability expansion into a unified framework. The reported improvements across three different agent benchmarks suggest that this approach has practical potential for improving cost efficiency without retraining models.

Critical weaknesses
  1. The paper does not sufficiently establish its novelty over existing routing and experience-based agent learning methods, such as FrugalGPT, Agent Workflow Memory, and ReMe. Since improving weaker models through reusable execution experiences has already been explored, the unique contribution of RouteSage remains unclear, which weakens its claimed novelty.

  2. The baselines primarily focus on conventional model routing, without comparisons to experience-enhanced alternatives. Moreover, the ablation studies do not adequately isolate the contributions of key components, such as Capability Regions and patch validation. This makes it difficult to determine whether the improvements arise from the proposed framework or simply from incorporating reusable Experiences.

Questions for authors
  1. How does RouteSage compare with existing routing methods equipped with the same reusable Experiences?

  2. Can the authors provide additional ablations to demonstrate the necessity of Capability Regions and the proposed patch validation mechanism?

  3. How does RouteSage fundamentally differ from existing cost-efficient routing and experience-based agent optimization methods? Could the authors further clarify its unique advantages and explain why existing approaches are insufficient to address the same problem?

4 comments

Dear Reviewer XtHb,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 3 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Dear Reviewer XtHb,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in 48 hours, we kindly request that you review the rebuttal and confirm whether your concerns have been addressed. Thanks a lot.

Regards from the authors.

Dear Reviewer XtHb,

Thanks for your time and effort in reviewing this paper. As the author-reviewer discussion period ends in about 24 hours, we would be grateful if you could take a look at our rebuttal and let us know whether your concerns have been addressed. We are happy to clarify any remaining questions before the deadline.

Regards from the authors.

Thank you for your detailed rebuttal and additional experiments. While the additional experiments have improved the empirical evaluation, your clarification regarding novelty has not fully convinced me. In my view, the proposed framework appears to be primarily an integration of existing techniques rather than a sufficiently novel methodological contribution. Therefore, I do not believe the paper currently meets the acceptance standards of ICLR, and I have decided to maintain my original rating.

Reviewer_ftCvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RouteSage groups requests into procedural regions and routes them to cheaper models, optionally augmented with guidance distilled from execution history. It constructs capability patches online and validates them against the requested model, combining quality-constrained routing with accounting for construction and validation costs.

Critical strengths

A promising perspective with reasonably comprehensive evaluation. Improving cheaper models' effective capabilities is a useful starting point for reducing agent-serving costs. The evaluation spans three domains, includes online costs and frozen held-out testing, and provides ablations demonstrating that guided patches improve both quality and savings over direct routing alone.

Critical weaknesses
  1. Mismatch between the routing framing and the demonstrated contribution. The central mechanism is effectively training-free distillation through execution guidance, but the paper does not isolate whether its routing design adds substantial value beyond this capability enhancement. Directly relevant methods such as Inference-Time Distillation and Agent Memory Distillation are neither discussed nor evaluated. A comparison using the same enhanced action pool with different routers is needed to substantiate the claimed routing contribution.
  2. Overstated novelty and insufficiently competitive baselines. The characterization of prior routing methods as operating over fixed capabilities is too broad: EvolveRouter already uses routing diagnostics to refine agent prompts, while CORAL jointly optimizes routing and prompts. Nevertheless, the experiments compare only against older fixed-pool methods rather than these closer alternatives. This leaves the claimed advance over existing capability-evolving routing approaches insufficiently supported.
Questions for authors

Does the cost accounting distinguish cached from uncached input tokens? In continuing multi-turn sessions, switching models can lose conversation-prefix KV-cache reuse and require additional prefill, whereas a fixed-model baseline may retain that cache. How would the reported savings change under cache-aware accounting, and what is the impact on time to first token?

3 comments

Dear Reviewer ftCv,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 3 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Dear Reviewer ftCv,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in 48 hours, we kindly request that you review the rebuttal and confirm whether your concerns have been addressed. Thanks a lot.

Regards from the authors.

Thanks for the response. My concern about the limited novelty remains unchanged, and I will keep my score.

Furthermore, I have a significant doubt: is there empirical evidence that this method outperforms existing SOTA LLM routers? The current comparison lacks strong router baselines.

Comments on the paper

3 comments

Dear Reviewer d5xx,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 3 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Dear Reviewer d5xx,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in 48 hours, we kindly request that you review the rebuttal and confirm whether your concerns have been addressed. Thanks a lot.

Regards from the authors.

Dear Reviewer d5xx,

Thanks for your time and effort in reviewing this paper. As the author-reviewer discussion period ends in about 24 hours, we would be grateful if you could take a look at our rebuttal and let us know whether your concerns have been addressed. We are happy to clarify any remaining questions before the deadline.

Regards from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.