- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the problem of provider selection in LLM services. The authors highlight that even when the same model is selected, different service providers can exhibit significant differences in inference quality, response latency, service availability, and pricing. To address this issue, the authors propose a market-aware provider routing mechanism and develop FACET, which combines task-level feasibility certification, fail-safe fallback, and online quality monitoring to reduce inference costs while controlling service quality risks.
Critical strengths
1.The paper studies a practically important problem.
2.The experimental results demonstrate that provider pricing does not exhibit a consistent correlation with service quality.
3.FACET effectively reduces inference costs while mitigating service quality risks.
Critical weaknesses
1.The fundamental differences between FACET and existing safe/constrained bandit methods and CUSUM-based change detection approaches are not sufficiently clarified.
2.Feasibility certification relies on quality estimates from small samples and a trusted anchor provider, without rigorous guarantees that quality constraints are satisfied.
3.The theoretical analysis of detection delay is subject to restrictive assumptions, and the real-world online experiments do not sufficiently validate the method's adaptability to sudden quality degradation.
Questions for authors
- What are the key technical innovations of FACET compared with existing safe/constrained bandit methods, non-stationary bandits, and CUSUM-based change detection approaches?
2.Why is the quality threshold defined relative to the accuracy of the best-performing provider? How can the system guarantee service quality when all providers experience simultaneous quality degradation?
3.How does FACET ensure that providers certified based on a small number of samples actually satisfy the quality constraints? Can the authors provide a theoretical upper bound on the probability of incorrect certification?
4.Do the theoretical results on CUSUM detection delay apply to quality degradation of arbitrary magnitude? Do these theoretical guarantees still hold when the actual quality change deviates from the assumed detection distribution?
5.Since no sudden quality degradation was observed in the real-world deployment experiments, could the authors conduct longer-term online experiments involving multiple models to validate FACET's safety and cost advantages under actual quality drift?
Follow-up on our rebuttal
Dear Reviewer,
Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns.
As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification.
If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate.
Thank you for your consideration.