- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies routing and model residency when the model pool is larger than GPU memory. CoServe keeps several predicted acceptable models for each request instead of committing to one too early. When considering a resident set, it recomputes the request assignments and estimates serving, waiting, and loading costs. The historical B200 runs show better deadline attainment than CARROT, with a worse tail. A separate controlled experiment with three models gives some evidence that recomputing assignments helps.
Critical strengths
-
I like the main observation. If a request can use either A or B, the memory controller should know this before deciding to load or evict a model. Just passing a chosen model identifier loses that option. The examples make the issue easy to understand, and the paper implements the idea on an existing serving backend.
-
The authors are quite careful about what the measurements mean. In particular, they include waiting before HTTP dispatch and separate timestamp uncertainty from run-to-run uncertainty. They also report the stronger static result and the quality limitations. I appreciate this, since otherwise it would be easy to get a misleading latency improvement by moving the queue elsewhere.
Critical weaknesses
There is still a few things I am unsure about in the evaluation. The quality numbers in the historical runs come from incomplete offline labels, with some training/test prompt overlap. The newer experiment improves this setup, but does not establish a gain in correct-and-timely answers. The static baseline is faster, although it also relaxes the candidate restriction. So I would be careful about drawing a broad quality-latency conclusion from these numbers.
The controlled ablation is useful, but it uses a different model pool, predictor, and memory budget from the main deployment. It helps explain the mechanism, without fully explaining the historical deadline gains. I would like a clearer connection between the two experiments. These limitations make the results somewhat preliminary, but I still think preserving model alternatives is a useful idea worth sharing.
Questions for authors
-
What quality conclusion can we draw from the labels and generated answers that are available? How often does the fallback select a model below the threshold, and exactly what candidate requirement is relaxed by the static baseline?
-
Which result best supports the connection between recomputing assignments and the gains in the larger-model run? A short explanation using the existing measurements would help; I am not asking for the whole eleven-model experiment to be repeated.
2 comments
Response to Reviewer dSAi (Part 2/2)
Q2. What the larger-model ablations establish
Question. Which result best supports the connection between recomputing assignments and the gains in the larger-model run?
What we have done and where. Both tests now run in the same load-cost regime, with 70B models and 52 s loads. Section 4.4 tests what the memory controller learns from a request's alternatives, and Appendix C tests recomputation.
Answer. The planner-input test is the clearest link, and it is your eviction example measured. With the planner and dispatch unchanged, a planner that sees only each request's cheapest acceptable route makes 4.0 loads per unit and answers 57.9% of issued requests correctly within 100 s. Given the candidate sets, it makes 1.0 load per unit and answers 74.3%. The gain is 16.4 points [11.0, 21.9], positive on all six units.
Evidence and experiments. FULL versus FIXED is the narrower test of recomputation alone. On two GPUs, both planners received the candidate sets and used the same dispatch. Recomputing assignments for each proposed resident set, instead of keeping the assignment fixed within a planning tick, changed goodput by 0.0 percentage points (paired 95% interval [−0.9, +0.9]). The planner was active in this test: CoServe changed the resident set in every unit.
Analysis. Taken together, the two tests show what matters at these load times: the alternatives reaching the memory controller, not recomputing assignments for each proposal. A route to a small model can make a load look necessary. With the candidate sets, the planner recognizes which of those requests the resident 70B model can already answer, and it keeps that model in memory.
Commitment to revise. Section 4.4 reports the planner-input result and points to the FULL/FIXED test in Appendix C.
Response to Reviewer dSAi (Part 1/2)
Thank you for highlighting the decision at eviction time. Based on your comments, we have improved the paper. Section 4 of the revised paper replaces the eleven-model study with a one-build evaluation (Appendix D relates the two). It grades the answers each policy actually generates and reports both versions of the static baseline you asked about.
Q1. Quality conclusion, fallback rate, and the static relaxation
Question. What quality conclusion can be drawn from the available labels and generated answers? How often does the fallback select a model below the threshold? Exactly which candidate requirement does the static baseline relax?
What we have done and where. We replaced the offline label lookup with live grading of every generated answer, on held-out prompts with no near-duplicate in training (Section 4.1 and Appendix A).
Answer. CoServe delivers more correct answers on time, with accuracy among served requests similar to the other policies. The fallback applies to 33.1% of requests, identically in every policy. The relaxed static baseline drops the rule that a request may be served only by a model in its candidate set.
Evidence and experiments. Quality. For the policies in Table 1 that serve only from C(q), accuracy among served requests lies between 80% and 86%; CoServe's is 85.1%. The strongest rule under the same candidate restriction is static Llama-70B with EDF dispatch and CoServe's concurrency limit. Against it, CoServe puts more requests on time (83.8% vs 80.4%) and gives the on-time slots to requests more likely to be answered correctly (88.7% vs 87.0% correct among on-time answers). Overall it answers 74.3% of issued requests correctly within 100 s, against 70.0%.
Fallback. C(q) holds the models whose predicted correctness reaches the threshold τ = 0.80. For 33.1% of test requests no model reaches τ, and these go to the highest-scoring model. The threshold was chosen on held-out calibration data. At τ = 0.80, routing each request to its cheapest candidate loses 1.0 point of accuracy against routing it to its most accurate model.
The static relaxation. Under the candidate rule, a static set serves a request only if a resident model is in C(q). The relaxed version serves every request with the best resident model, even outside C(q). We now report both.
The relaxed set finishes every request on time, and 70.3% of its answers are correct. Of CoServe's served answers, 85.1% are correct.
Analysis. CoServe and static Llama-70B have closely matched accuracy among completed answers (85.1% and 85.2%). The difference that matters to a waiting user is the share of all issued queries answered correctly by the deadline: 74.3% against 70.0%.
Commitment to revise. Section 4 and Appendix A report live correctness, the fallback rate and both static variants. Section 3.1 states the threshold rule.