- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies admission control for shared services with different request values, resource requirements, and tenant service targets. It proposes an algorithm, TERRA-CP, whose algorithm intuition makes sense, by ranking requests using an affine combination of debt-repayment density and value density. Then, they run some numerical experiments to support their algorithm.
Critical strengths
The presentation is clear.
The theoretical math model tries to describe single-instance LLM scheduling with an analytical perspective.
Critical weaknesses
-
The model is quite general, which does not capture any LLM serving unique pattern. Therefore, if I replace LLM serving by general cloud computing, the model does not change. However, this is wrong since LLM serving has many unique patterns such as KV usage, which is not described in this paper. This makes a big gap between model and real.
-
The admission policy is similar to single-instance scheduling. I think the authors do not compare with methods from this field. Moreover, a very strange thing is that: the authors abstractly define a term as "value", and use this value to make decisions. However, in real LLM serving, finding this kind of "value" is the hardest thing. For example, shortest first scheduling makes the predicted output length as the "value", but this may not be good. Leaving the definition as abstract and using an abstract term to do admission control, from a practitioner's perspective, I really cannot see any insight from the model and algorithm.
Questions for authors
-
How should we define "value" in real LLM serving?
-
What is the specific model pattern related to LLM serving?
Thanks for the responses. I still don't see how this problem is different from a generic scheduling problem. The answer given in the authors' rebuttal seems to be about implementation details, while the overall problem framework and solution are still the same as any scheduling problem that has been widely studied.