acceptodds
Under review as a conference paper at ICLR 2027

Whose Turn Is It? Context-Guided Model Routing for Stateful Agents

Abstract

Model routing can reduce the cost of LLM applications by assigning simpler requests to cheaper models without sacrificing the quality of the response. However, most routers treat requests as independent. Stateful agents do not operate under this assumption: a user turn may require multiple model and tool calls, its difficulty depends on the state produced by earlier turns, and changing models can sacrifice reuse of the conversational prompt cache. We study zero-shot, cost-aware routing at the user-turn boundary, systematically varying the routing rule, access to session history, candidate models that differ in size, reasoning effort, or model provider, and prompt-cache behavior. We evaluate on multi-turn versions of SheetCopilot and SpreadsheetBench 2, where persistent workbooks and cross-turn dependencies make isolated routing decisions especially brittle. At a matched offloading rate, giving the router session history improves task pass rate over the same router without history, with gains for every model pair; history-aware routing remains within one point of always using the expensive model. We also found that cost depends on execution dynamics rather than token prices alone: the cheaper model’s per-task cost advantage narrows from to on longer sessions. A closed-form cache analysis further shows that downward switches (expensive to cheap) recover their cache cost within two LLM calls, whereas upward switches do not break even under the evaluated conditions. Together, these results show that agent routing is a sequential decision problem: effective policies must account for what happened earlier in the session and for the asymmetric cost of switching models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model routing, multi-turn agents, conversational context, cost-efficient inference, model switching, prompt caching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qw2KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies zero-shot, cost-aware model routing for multi-turn spreadsheet agents, with the routing decision made at the user-turn boundary. It varies four things:

  • the decision rule: zero-shot LLM router, escalation, de-escalation, or random;
  • what the router sees: the current request only, or the session history;
  • the candidate pair: GPT-5.6 Sol/Luna, Sol/Haiku 4.5, or Opus 4.8 at high/low effort;
  • prompt-cache behavior.

Experiments use multi-turn versions of SheetCopilot (112 tasks, 2.88 turns on average) and SpreadsheetBench 2 (221 tasks, 4.76 turns on average). The main findings are:

  • At a matched offloading rate, giving the router session history raises pass rate by 3.3 points on SheetCopilot.
  • Price per token is a poor proxy for cost per task. Haiku’s cost advantage over Sol shrinks from 6.8× to 3.2× on longer sessions.
  • A closed-form cache analysis shows that switching down to the cheaper model recovers its cache cost within about 1.25 calls, while switching up never breaks even.
Critical strengths
  • I like the fact that this paper is addressing the very practical constraints of cache hits in model routing in multi-turn trajectories, and I also like that they are building a very systematic approach to routing where they consider several axes of routing choices.
  • The motivation for zero-shot routers in agentic settings is convincing. Labels require running full tool loops under each candidate, the number of trajectories grows as 2^T, and candidate models change every few months.
  • The asymmetric cost of switching models is a good observation / result to make. The break-even expression k* (Eq. 3) is simple and gives actionable guidance: switch down early and switch up rarely.
  • The observation that a model cheap per token is not necessarily cheap per task (Fig. 3) is practically useful. Single-turn routing evaluations cannot reveal it.
Critical weaknesses
  • Are the authors citing enough prior literature on model routing? Relevant missing works include xRouter, Router-R1, Adaptive LLM Routing under Budget Constraints, Online Multi-LLM Selection via Contextual Bandits, Federate the Router, ProxRouter, Beyond Accuracy and Cost (latency-aware routing), and Not All Turns Are Equally Hard (adaptive thinking budgets). These prior works bring me to question if the contributions of formulating a cost-aware router at multi-turn boundaries and routing being a decision making problem listed in the introduction are genuinely novel contributions.
  • Only two evaluation benchmarks seems a little too small, and both being on spreadsheet agents seems a little weird – the paper definitely needs broader evaluation. Results are also single runs with no confidence intervals, and run-to-run noise is comparable to the reported gains.
  • I want to understand if the evaluations rigorously study the cache misses due to model changes mid-trajectory phenomenon. From my reading, they do not as the cache analysis is a closed-form calculation (Eq. 3), with no measured cache hit rates.
  • I am not sure all the axes (including routing rule, access to session history, candidate models that differ in size, reasoning effort, or model provider, and prompt-cache behavior) are evaluated effectively. For example, the appendix prompts contradict the definitions of escalation and de-escalation in Sec. 2. Also, the “history-aware” router sees only earlier user requests.
  • The main router does not reduce cost - it costs more than always using the expensive model in all six pair × benchmark settings (Tables 3, 4).
  • [Minor] It is not professional that the appendix is not formatted properly. Appendices B and C are empty headings, and numbers disagree between the text, figures and tables.
Questions for authors
  • What exactly was run under “escalation” and “de-escalation”? Please reconcile Sec. 2 and Fig. 1b with Figs. 7–8.
  • What differs in the “separate harness” used for escalation, de-escalation and random routing?
  • How were the routed costs for Sol/Haiku and Sol/Luna computed? Is the router call included?
  • Why does the history-aware router cost more than expensive-only in every cell?
  • The motivation for zero-shot routers is convincing, but I wonder if continually learnt routers also make sense in this case?
Reviewer_mvJZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a zero shot cost aware model routing approach for multi turn agents. For each turn the router decided whether to use the cheap or an expensive model. Authors test this on two benchmark tasks SheetCopilot and SpreadsheetBench2 and compare an llm router with or without session history, within turn escalation/deescalation, and some other baselines. The conclusion is that giving the router session history improves the pass rate at a matched offloading, cheap per token not necessarily meaning cheap per task and model switching has an asymmetric cost.

Critical strengths
  1. The methodology is straighforward. The router overhead and cache effects are being considered for each policy, the history comparison is done at a matched offloading rate to isolate the effect of which turns are offloaded.
  2. The problem of routing is cleanly separated in 3 dimensions in section 2. The router's information set or the context, the frequency of decision , and the escalation deescalation primitives.
Critical weaknesses
  1. The paper produces no new method, the contribution is largely formalizing or framing this phenomenon along with a measurement study on two spreadsheet benchmarks. The router is a handwritten prompt that implicitly classifies a request as cheap or expensive and is fairly specific to the task at hand and skewed towards always expensive. History aware routing is not new, this perhaps is ubiquitous in practice, the paper itself cites many prior works. The novelty and contributions is therefore only slightly incremental.

  2. The size of the datasets used are 112 and 221 respectively, the deltas between the performances are in the order of few percentage points with no confidence intervals or repetitions. The policies are not even run uniformly on the same task set, for example the deescalation uses 99-110, random uses 111, random and other ran in separate harnesses (this I think contradicts authors experimental setup in section 3). With all such variance inducing conditions, the reported differences could be plausible within noise. Authors need to harden their results.

  3. The baselines are weak, they are mostly variants of the authors own method, random is a control etc. Authors do not compare with existing prior work on such routing, not even a simple llm based complexity routing, or confidence based routing.

  4. Authors never justified or explained the per turn granularity. It is motivated as between session level and per call, as user turn boundary. But they never show any ablation on why, or the value of deciding every user turn instead of once per session or once per call. Also the restriction to only two per session seems forced. The within turn escalation deescalation are also a bit contrived and it seems dont work very well, for example the gains of escalation over cheap only cases comes at a very high cost and the deescalation does not meaningfully cost-quality effect.

  5. The study is effectively confined to a single domain, the prompts on skimming looked very specific to this spreadsheet domain only. While I agree maybe this is a tough test bed, but the paper goes on to make claims about generalized behavior during routing. There is not transfer test or ablation done with a different domain to validate their hypothesis.

  6. The main prompt in the paper classifies a request from cheap to expensive from spreadsheet specific jargon, formal wording with default of when in doubt use expensive. The second prompt with session history uses the session history only for reference resolution it seems and is definitely narrower than the papers original claim that history is needed to judge what has been attempted and whether the current turn required recovery. The implemented mechanism seems to have drifted from the earlier in the paper described and motivated one.

Questions for authors

see weaknesses

Reviewer_8SdWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies zero-shot, cost-aware model routing at the user-turn boundary for multi-turn spreadsheet manipulation agents. The authors evaluate several routing policies—including an external zero-shot LLM router with and without conversation history, within-turn self-escalation and de-escalation, and random routing—across three candidate model pairs on multi-turn conversions of SheetCopilot and SpreadsheetBench 2. The study analyzes how conditioning on prior user requests, per-task call multiplication in weaker models, and prompt-cache invalidation during model switches affect the cost-quality trade-off, supplemented by a first-order break-even formula for cache switching costs.

Critical strengths
  1. The paper highlights practical system-level factors in multi-turn agentic routing that are absent in single-turn request routing, specifically the gap between per-token list price and realized per-task cost (due to weaker models making more tool-call loops) and the asymmetric cost of invalidating provider prompt caches when switching models.

  2. Comparing the router with and without prior user requests at a matched offloading rate cleanly isolates the benefit of resolving cross-turn coreferences (such as "Apply the same format to Q3") when estimating turn difficulty.

Critical weaknesses
  1. The proposed history-aware LLM router fails to achieve the paper's central objective of cost reduction. While Section 1 states that a routing policy is useful only if it lowers expected cost relative to the expensive-only baseline without lowering task success, Tables 3 and 4 show that "LLM router, with history" is actually more expensive than "Expensive-only" across nearly every candidate pair and benchmark (e.g., $0.375 vs. $0.364, $0.392 vs. $0.341, and $0.369 vs. $0.365 on SheetCopilot; $1.192 vs. $1.159, $1.208 vs. $1.168, and $2.080 vs. $2.009 on SSB-2), even before adding the cost of calling the router itself. On SSB-2, the router offloads only 0.4% to 1.7% of turns to the cheap model, acting as a near-trivial pass-through to the expensive model while incurring cache-switching penalties.

  2. Moreover, in two of the three evaluated candidate pairs (Sol/Luna and Opus high/low), the "Cheap-only" baseline achieves a higher pass rate than "Expensive-only" at a fraction of the cost on both benchmarks (Tables 3 and 4), strictly Pareto-dominating all routed policies and rendering routing counterproductive for those pairs.

  3. There is a major inconsistency between the problem formulation in Section 2 and the actual router implementation in Appendix C. Section 2 claims that the router's history state h_t contains "earlier requests, agent responses, tool calls and observations, execution errors, and prior routing decisions." However, Figure 6 reveals that the history-aware router prompt is only fed the raw text of earlier user requests ("Turn 1: <request 1> ... Current request: ") to resolve pronouns, without any agent trajectories, tool execution errors, or workbook states.

  4. The methodological contribution is very thin for a top machine learning venue. The router is a single, handcrafted prompt containing Excel-specific keyword heuristics (Figure 5, e.g., routing any query mentioning VLOOKUP, SUMIF, or cell ranges to Expensive), and the cache break-even equation (Equation 3) is an elementary linear arithmetic identity. Despite the paper's analytical findings on cache stickiness and pair capability gaps, the authors never incorporate these principles into an improved, cache-aware or pair-aware routing algorithm.

Questions for authors
  1. Given that "LLM router, with history" has a higher mean cost per task than "Expensive-only" in all six benchmark-pair cells in Tables 3 and 4 (and offloads less than 2% of turns on SSB-2), in what sense does the evaluated zero-shot router provide a viable cost-saving solution?

  2. Why does the history-aware router prompt in Figure 6 only include earlier user request strings when Section 2 explicitly defines h_t as containing agent responses, tool calls, observations, and execution errors?

  3. Why do the captions of Tables 3 and 4 state that "LLM-router costs are exact only for Opus high/low" and exclude router call costs, when exact token and cost accounting is central to the paper's claims?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.