- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper models the online LLM routing in the serving system as a multi-arm bandit problem. Deriving learning-based policy and a square root regret bound. Then, they run some model-based simulation to verify the theory.
Critical strengths
- The writing is clear.
- The theoretical analysis is rigorous.
Critical weaknesses
-
The biggest weakness is the gap between the model and real. In real MaaS system, usually different models are deployed on different GPU groups (each GPU needs to store the model parameters and therefore it is not possible to store a lot of models' parameters on a single GPU). Therefore, in a real MaaS system, waiting is usually not a routing decision. Moreover, from the math model in the paper, I do not see a clear relationship between this model and LLM serving, and I think it can be a valid model on general learning theory and bandit literature. If the authors think this is a complete theoretical work, I suggest removing all LLM motivations, and carefully compare with bandit literatures to make the novelty and contribution clear. If the authors think this is really designed for LLM serving, we need to have some LLM serving patterns in the model. Otherwise, the gap is too huge to provide any insight. Based on my understanding, if the paper is trying to contribute to the application field (using bandit to optimize LLM routing), the authors should spend more efforts on explaining the gap between the model and real routing before deriving beautiful mathematical theorems, because even if the mathematical theorems are beautiful, due to the huge gap between the model and real, these beautiful results still become meaningless.
-
The simulation is mainly model-based simulator, and from my perspective, it can provide few insights.
Questions for authors
Would you like to figure out the gap between your model and real LLM routing in a MaaS system?