acceptodds
Under review as a conference paper at ICLR 2027

You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference

Abstract

Existing LLM routers choose among models using static per-model costs. We show that open-weight inference markets introduce a second, largely ignored decision axis: after choosing a model, a client must still choose which provider serves it. Measuring live endpoints across many open models, competing providers, multiple task types, and three measurement waves, we find that provider choice cannot be inferred from the price list. The same model can vary sharply in quality, latency, availability, and price across providers; higher-priced providers are consistently faster, but price does not reliably predict quality or availability; and provider feasibility is task-selective, with one deployment nearly normal on knowledge tasks but catastrophically degraded on multi-step reasoning. We formulate same-model provider selection as a price-taker market-aware routing problem. A simple measured-map policy routes to the cheapest provider that is both quality-equivalent and healthy, yielding matched-quality savings while avoiding degraded endpoints. Because the map drifts, we introduce facet, an online provider router that certifies per-(provider × task) feasibility facets and fails safe to an anchor before serving uncertified arms. Across relaxed deployment assumptions, facet tolerates imperfect task assignment and sparse feedback, while systematic evaluator bias exposes a quality-signal trust boundary that can be mitigated with ground-truth probes or audits. Live provider runs further confirm that certification can move real traffic from a premium anchor to a substantially cheaper certified endpoint. Our results suggest that market-aware LLM routing must measure not only which model to use, but also who serves it.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM routing, inference serving, open-weight models, provider selection, market-aware routing, online learning, non-stationary bandits, reliability, inference cost

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vys5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the problem of provider selection in LLM services. The authors highlight that even when the same model is selected, different service providers can exhibit significant differences in inference quality, response latency, service availability, and pricing. To address this issue, the authors propose a market-aware provider routing mechanism and develop FACET, which combines task-level feasibility certification, fail-safe fallback, and online quality monitoring to reduce inference costs while controlling service quality risks.

Critical strengths

1.The paper studies a practically important problem.

2.The experimental results demonstrate that provider pricing does not exhibit a consistent correlation with service quality.

3.FACET effectively reduces inference costs while mitigating service quality risks.

Critical weaknesses

1.The fundamental differences between FACET and existing safe/constrained bandit methods and CUSUM-based change detection approaches are not sufficiently clarified.

2.Feasibility certification relies on quality estimates from small samples and a trusted anchor provider, without rigorous guarantees that quality constraints are satisfied.

3.The theoretical analysis of detection delay is subject to restrictive assumptions, and the real-world online experiments do not sufficiently validate the method's adaptability to sudden quality degradation.

Questions for authors
  1. What are the key technical innovations of FACET compared with existing safe/constrained bandit methods, non-stationary bandits, and CUSUM-based change detection approaches?

2.Why is the quality threshold defined relative to the accuracy of the best-performing provider? How can the system guarantee service quality when all providers experience simultaneous quality degradation?

3.How does FACET ensure that providers certified based on a small number of samples actually satisfy the quality constraints? Can the authors provide a theoretical upper bound on the probability of incorrect certification?

4.Do the theoretical results on CUSUM detection delay apply to quality degradation of arbitrary magnitude? Do these theoretical guarantees still hold when the actual quality change deviates from the assumed detection distribution?

5.Since no sudden quality degradation was observed in the real-world deployment experiments, could the authors conduct longer-term online experiments involving multiple models to validate FACET's safety and cost advantages under actual quality drift?

1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns.

As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification.

If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate.

Thank you for your consideration.

Reviewer_HZFKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies differences in quality, latency, availability, and pricing across providers serving the same open-weight LLM. The authors find that provider performance varies across tasks and over time, making price alone insufficient for provider selection. They propose FACET, which certifies provider quality before routing traffic and monitors performance changes. The evaluation combines real provider measurements, replay-based experiments, and a small live deployment.

Critical strengths

The paper addresses a relevant practical problem. Its measurements across providers, models, and time provide useful evidence that the same model can deliver substantially different performance depending on the serving endpoint. The live deployment also demonstrates that provider certification and migration can work in practice.

Critical weaknesses
  1. Limited novelty and unclear research positioning.

My main concern is the paper's contribution relative to existing literature. Provider-level heterogeneity has already been investigated in Li et al. (2026) and Gao et al. (2026), including differences in performance, cost, and deployment feasibility. On the methodological side, minimizing cost subject to uncertain quality constraints is closely related to cost-subsidy bandits, particularly Sinha et al. (2021) and Juneja et al. (2026).

FACET combines established ideas such as feasibility certification, fallback routing, and change detection. While their integration is practically useful, the paper does not convincingly establish a substantial methodological advance. The positioning is therefore unclear: as a measurement study, its findings overlap with prior work; as a methodology paper, its algorithmic novelty appears limited.

  1. Limited validation of the proposed methodology.

Much of FACET's claimed advantage comes from constructed failure scenarios rather than naturally observed provider changes. In fact, the severely degraded provider in the measured replay cell is expensive, whereas the main safety experiments emphasize hypothetical cheap degraded providers. The 36-hour live experiment includes only 538 served queries and observes no actual quality collapse, so it does not validate the central drift-detection mechanism.

The incremental benefit is also not fully convincing. Appendix U reports 46% median savings under confidence-supported measured-map routing, compared with 45% from simply selecting the most accurate provider. Although this does not directly measure FACET's online benefit, it raises questions about how much of the reported improvement comes from the proposed methods rather than basic provider measurement and selection.

The presentation further weakens the paper. It moves between descriptive market analysis, routing optimization, and online monitoring without clearly identifying the central research contribution.

Questions for authors
  1. What is the precise methodological novelty of FACET relative to existing cost-subsidy bandits and safe or non-stationary bandit methods? Can the authors provide a direct comparison under matched information and measurement budgets?

  2. What new empirical insight does this study establish beyond Li et al. (2026) and Gao et al. (2026)? The paper acknowledges related measurement work, but the substantive research gap remains unclear.

  3. Can the authors provide stronger evidence of FACET's benefit under naturally occurring provider degradation rather than constructed failures? In particular, how much improvement remains over a simpler provider-selection policy equipped with the same monitoring mechanism?

Li, H., He, Z., Liu, X., Xu, Y., Liu, D., Ma, J., Wu, L., Wu, Y., Tang, X., and Shi, T. (2026). When Is the Same Model Not the Same Service? A Measurement Study of Hosted Open-Weight LLM APIs.

Gao, Y., Wang, M., and Yu, Y. L. (2026). Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference.

Sinha, D., Sankararaman, K. A., Kazerouni, A., and Avadhanula, V. (2021). Multi-Armed Bandits with Cost Subsidy.

Juneja, I., Joe-Wong, C., and Yağan, O. (2026). Cost-Ordered Feasibility for Multi-Armed Bandits with Cost Subsidy.

1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns.

As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification.

If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate.

Thank you for your consideration.

Reviewer_Fg4iOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies provider selection after an open-weight model has already been chosen. Measurements across providers, tasks, and time show that price alone does not determine whether an endpoint meets a task's quality requirements. The proposed FACET system certifies each provider-task pair before admitting user traffic, uses an already certified fallback provider while probing candidates, and monitors admitted providers with CUSUM and cohort comparisons. The evaluation combines live endpoint measurements, routing experiments, component comparisons, feedback and drift stress tests, and a live deployment to examine the trade-off between serving cost and exposure to providers below the quality threshold.

Critical strengths
  1. A relevant problem supported by concrete measurements. Choosing a model does not settle which provider should serve a particular task. The paper provides useful evidence that same-model providers can differ in quality across tasks and that their suitability changes over time. This supports the practical value of task-conditioned provider selection based on current evidence.

  2. Careful evaluation with a clear connection between claims and evidence. The experiments examine task-label errors, noisy and sparse quality feedback, evaluator bias, different patterns of provider degradation, and live migration. These checks cover several deployment conditions that are easy to overlook. The comparison that gives periodic re-measurement the same feedback monitor is particularly useful for separating the effects of admission and continued monitoring. Together, the component comparisons and stress tests connect the motivation, design choices, and empirical claims clearly.

Critical weaknesses
  1. The methodological advance over existing quality-aware provider routing is modest. Existing systems such as OpenRouter Auto Exacto already use ongoing quality signals and recurring benchmarks to adjust provider routing. FACET adds task-specific admission, fallback service, and targeted probing, but its distinction from this practice needs a more concrete comparison. Table 23 also shows that periodic re-measurement with the same monitor achieves comparable below-floor exposure in the post-admission drift setting. This makes the evidence more compelling for a useful system integration than for a substantial methodological advance. The reported reduction in dedicated probes is valuable, but does not establish a corresponding reduction in total cost.

  2. The practical trade-off under short-lived degradation remains insufficiently characterized. Appendices J and V already examine frequent changes and recovery, and the paper includes probe costs and a break-even analysis. The remaining question is what happens when quality briefly falls below the threshold and quickly recovers. The results do not clearly quantify how evidence age and feedback delay affect detection and re-admission, or whether the avoided low-quality service justifies the additional fallback and re-certification costs in this regime. This is especially relevant for low-volume workloads, where labelled observations accumulate slowly. The limitation concerns the demonstrated range of practical benefit, rather than a requirement for instantaneous detection.

Questions for authors
  1. Which specific capability distinguishes FACET from existing quality-aware provider routing such as Auto Exacto? Please explain the difference in task-specific admission and use of existing monitoring signals, and clarify what the component comparisons establish about additional benefit at a comparable total cost.

  2. For a brief quality drop followed by rapid recovery, how do detection, quarantine, and re-admission interact? Beyond the existing drift and recovery tests, can the authors clarify the effect of delayed feedback and slowly arriving labels, and show when the avoided below-floor service outweighs the additional probing and fallback costs? A focused sensitivity analysis would help establish the relevant deployment conditions.

1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns.

As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification.

If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate.

Thank you for your consideration.

Comments on the paper

3 comments

Dear Authors and Reviewers,

Thank you for supporting ICLR'27!

We are now entering the discussion phase. Please make the most of this time to exchange your views. I encourage each Reviewer, if you have not already done so, to read the other reviews and the Authors' responses. Please respond to the Authors and actively participate in the discussion to help clarify any remaining issues.

Thank you,

AC

Dear reviewers,

The authors have submitted their response, and the discussion will be closed in a few days. Please read the response and the other reviews, and update your review, including ratings, if your assessment has changed.

If any concerns remain, please explain whether and why the response has addressed them. Where your assessment differs from that of other reviewers, please discuss the specific points with reference to the paper and the response.

Thank you for your time and contributions.

AC

Dear Reviewers,

Thank you for your efforts for this paper.

As the discussion period draws to a close, I'd especially appreciate hearing from Reviewer Fg4i, Reviewer HZFK and Reviewer Vys5, who have yet to join the discussion. Please post a brief reply to the authors indicating which concerns their response has addressed and which, if any, remain. If all your concerns have been resolved, a short confirmation is sufficient.

Thank you for your time and input.

Thanks & Best Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.