- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks at quality-aware cross-model computation reuse in LLM serving, where the reuse quality is not known in advance and different tasks may share some preparation costs. The authors formulate it as an online optimization problem and propose QARS, which combines confidence-based quality estimation with joint preparation and assignment scheduling. One interesting part is the quality-aware stopping rule, which tries to balance the scheduling accuracy and the uncertainty from learning. The paper also provides regret analysis and experiments showing better quality-aware cost and scheduling efficiency compared to several baselines.
Critical strengths
-
The problem is meaningful and quite practical. Considering uncertain reuse quality together with shared preparation costs makes sense, and it captures some important challenges in cross-model computation reuse that are often overlooked.
-
I think the quality-aware stopping rule is an interesting part of the paper. It connects the optimization gap with the uncertainty in quality estimation, so the scheduler doesnt need to spend too much computation when the current solution is already good enough given what has been learned.
-
The paper provides a fairly complete theoretical analysis, including hardness results, regret bounds and scheduling complexity. The experiments also shows consistent improvements over the baselines, which supports the usefulness of the proposed approach.
Critical weaknesses
-
My main concern is about the experimental evaluation. Most of the regret results are based on a semi-synthetic simulator fitted using LLM outputs, and the linear quality assumption is basically satisfied by construction. This makes the results a bit less convincing, since its not clear how well QARS would work with real online LLM feedback, especially when the quality model is not exactly linear.
-
Another concern is the regret benchmark. It compares against the per-epoch optimal decision under the same observed state, but doesnt really consider how current preparation decisions may change future reuse opportunities. In practical caching or computation reuse systems, these decisions can have longer-term effects. So I think the current regret guarantee might not fully reflect the actual long-term performance of the system.
Questions for authors
-
What is the real performance of QARS in real-world LLM deployment?
-
Please consider how current preparation decisions may change future reuse opportunities?
4 comments
Official Comment by Authors
Q2: Another concern is the regret benchmark. It compares against the per-epoch optimal decision under the same observed state, but doesn't really consider how current preparation decisions may change future reuse opportunities. In practical caching or computation reuse systems, these decisions can have longer-term effects. So I think the current regret guarantee might not fully reflect the actual long-term performance of the system.
A2: Thank you for this constructive comment. We have further clarified the effects of current decisions on future reuse and the scope of the regret benchmark in Section 3.2. We explain these mechanisms and the planning extension below.
- Retained results support future reuse. Preparing result at epoch incurs . If retained, its preparation cost becomes , allowing later compatible requests to reuse it without another preparation charge. QARS uses these updated costs and available results to guide future decisions.
- Regret measures how well QARS learns and schedules. Under the stated conditions, sublinear regret means QARS’s average cost gap to the known-quality optimum under the same observed states tends to zero. This guarantee does not compare policies that create different future reuse opportunities.
- Preparation planning can account for future savings. QARS currently evaluates new preparations using the current epoch’s objective. A promising extension would anticipate future requests and result retention, and optimize cumulative quality-aware cost along the states created by each policy. We have expanded this discussion in Section 7.
Q3: What is the real performance of QARS in real-world LLM deployment?
A3: Thank you for highlighting this important point. For experimental validation, we have added experiments with measured LLM feedback, where QARS reduces quality-aware cost by 10.6–14.1% relative to UCB-LocalSearch across four QA datasets (Appendix I, Table I.1). For deployment, Section 3.1 (Figure 2) presents our design for using QARS as the reuse scheduler in LLM serving systems supported by engines such as vLLM or SGLang. For end-to-end performance, evaluating latency and throughput in large-scale deployments remains future work (Section 7).
Q4: Please consider how current preparation decisions may change future reuse opportunities?
A4: Thank you for this suggestion. Current preparation decisions affect future reuse opportunities: later compatible requests can reuse a prepared result without another preparation charge if it remains available. QARS uses these savings to guide future decisions, while evaluating new preparations for current tasks. Planning preparation for future savings is a promising extension that could explicitly model request arrivals and retention. We have clarified this distinction in Section 3.2 and expanded the discussion in Section 7.
We hope these clarifications and additional results address your concerns, and we would be happy to clarify any remaining points.
I maintain the original score
The additional experiments with real LLM feedback are helpful and partially address my concerns, and the further discussion on future reuse is also useful. Overall, I'm happy to maintain my current score.
Thank you for your follow-up
Thank you for your thoughtful follow-up and continued support. We are glad that you found the experiments with real LLM feedback helpful and the discussion of future reuse useful. Your feedback has helped us strengthen the empirical evaluation and clarify the scope of our guarantees. We would be happy to provide any further clarification you may find helpful.
Official Comment by Authors
We sincerely appreciate your constructive feedback. We hope the responses and revisions below address your concerns, and we welcome any further questions.
A1: Thank you for this constructive feedback. We have added experiments with real LLM feedback on held-out documents (Appendix I, Table I.1). We explain our evaluation design, new results, and extensions to richer quality models below.
Table I.1. Objective per task under real LLM feedback (lower is better). Settings as in Section 6.
References
[1] Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved Algorithms for Linear Stochastic Bandits. NeurIPS 2011.
[2] P. Panda, R. Magazine, C. Devaguptapu, S. Takemori, and V. Sharma. Adaptive LLM Routing under Budget Constraints. Findings of EMNLP 2025, pp. 23934–23949.
[3] B. Howson, C. Pike-Burke, and S. Filippi. Delayed Feedback in Generalised Linear Bandits Revisited. AISTATS 2023.
[4] M. Moghimi, S. T. Jose, and S. Moothedath. Neural Contextual Bandits Under Delayed Feedback Constraints. In 2025 IEEE 64th Conference on Decision and Control (CDC), 2026.