acceptodds
Under review as a conference paper at ICLR 2027

CoServe: Preserving Model Choice Under GPU Memory Pressure

Abstract

Multi-model LLM services often offer more models than GPU memory can hold simultaneously. Selecting one model for a request can trigger loading and eviction, even when an already resident alternative could answer it. We present CoServe, which preserves predicted acceptable model alternatives and coordinates request routing with model residency. For each proposed resident set, CoServe recomputes request assignments and compares estimated batch-service costs, waiting penalties, and activation costs. Our structural analysis shows that committing requests to single model identifiers can discard information needed to select a coverage-optimal resident set. On a single B200 deployment with eleven configured model footprints totaling nominal GPU memory, CoServe increases the fraction of requests completed within 100 seconds by at least 16.1 and 3.4 percentage points over CARROT on a Prism-based backend in two arrival realizations. Mean arrival-to-completion latency decreases by 14–17% in the first realization, while tail latency remains higher in both. Numerical ranges reflect timestamp reconstruction uncertainty, not confidence intervals. These results support preserving model alternatives when coordinating routing and GPU residency, with benefits that depend on the workload and latency objective.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Model LLM Serving, LLM Routing, Model Residency, GPU Memory Management, Model Loading and Eviction, Resource-Aware Serving, Inference Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dSAiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies routing and model residency when the model pool is larger than GPU memory. CoServe keeps several predicted acceptable models for each request instead of committing to one too early. When considering a resident set, it recomputes the request assignments and estimates serving, waiting, and loading costs. The historical B200 runs show better deadline attainment than CARROT, with a worse tail. A separate controlled experiment with three models gives some evidence that recomputing assignments helps.

Critical strengths
  • I like the main observation. If a request can use either A or B, the memory controller should know this before deciding to load or evict a model. Just passing a chosen model identifier loses that option. The examples make the issue easy to understand, and the paper implements the idea on an existing serving backend.

  • The authors are quite careful about what the measurements mean. In particular, they include waiting before HTTP dispatch and separate timestamp uncertainty from run-to-run uncertainty. They also report the stronger static result and the quality limitations. I appreciate this, since otherwise it would be easy to get a misleading latency improvement by moving the queue elsewhere.

Critical weaknesses

There is still a few things I am unsure about in the evaluation. The quality numbers in the historical runs come from incomplete offline labels, with some training/test prompt overlap. The newer experiment improves this setup, but does not establish a gain in correct-and-timely answers. The static baseline is faster, although it also relaxes the candidate restriction. So I would be careful about drawing a broad quality-latency conclusion from these numbers.

The controlled ablation is useful, but it uses a different model pool, predictor, and memory budget from the main deployment. It helps explain the mechanism, without fully explaining the historical deadline gains. I would like a clearer connection between the two experiments. These limitations make the results somewhat preliminary, but I still think preserving model alternatives is a useful idea worth sharing.

Questions for authors
  1. What quality conclusion can we draw from the labels and generated answers that are available? How often does the fallback select a model below the threshold, and exactly what candidate requirement is relaxed by the static baseline?

  2. Which result best supports the connection between recomputing assignments and the gains in the larger-model run? A short explanation using the existing measurements would help; I am not asking for the whole eleven-model experiment to be repeated.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dSAi (Part 1/2)

Thank you for highlighting the decision at eviction time. Based on your comments, we have improved the paper. Section 4 of the revised paper replaces the eleven-model study with a one-build evaluation (Appendix D relates the two). It grades the answers each policy actually generates and reports both versions of the static baseline you asked about.

Q1. Quality conclusion, fallback rate, and the static relaxation

Question. What quality conclusion can be drawn from the available labels and generated answers? How often does the fallback select a model below the threshold? Exactly which candidate requirement does the static baseline relax?

What we have done and where. We replaced the offline label lookup with live grading of every generated answer, on held-out prompts with no near-duplicate in training (Section 4.1 and Appendix A).

Answer. CoServe delivers more correct answers on time, with accuracy among served requests similar to the other policies. The fallback applies to 33.1% of requests, identically in every policy. The relaxed static baseline drops the rule that a request may be served only by a model in its candidate set.

Evidence and experiments. Quality. For the policies in Table 1 that serve only from C(q), accuracy among served requests lies between 80% and 86%; CoServe's is 85.1%. The strongest rule under the same candidate restriction is static Llama-70B with EDF dispatch and CoServe's concurrency limit. Against it, CoServe puts more requests on time (83.8% vs 80.4%) and gives the on-time slots to requests more likely to be answered correctly (88.7% vs 87.0% correct among on-time answers). Overall it answers 74.3% of issued requests correctly within 100 s, against 70.0%.

Fallback. C(q) holds the models whose predicted correctness reaches the threshold τ = 0.80. For 33.1% of test requests no model reaches τ, and these go to the highest-scoring model. The threshold was chosen on held-out calibration data. At τ = 0.80, routing each request to its cheapest candidate loses 1.0 point of accuracy against routing it to its most accurate model.

The static relaxation. Under the candidate rule, a static set serves a request only if a resident model is in C(q). The relaxed version serves every request with the best resident model, even outside C(q). We now report both.

Static small set (Gemma-2-9B, OpenMath2-8B, Mistral-7B)   Goodput   CoServe − static (pp)
Under the candidate rule.391+35.2 [+33.4, +37.0]
Relaxed (56% of requests served outside C(q)).703+4.0 [+2.1, +5.9]

The relaxed set finishes every request on time, and 70.3% of its answers are correct. Of CoServe's served answers, 85.1% are correct.

Analysis. CoServe and static Llama-70B have closely matched accuracy among completed answers (85.1% and 85.2%). The difference that matters to a waiting user is the share of all issued queries answered correctly by the deadline: 74.3% against 70.0%.

Commitment to revise. Section 4 and Appendix A report live correctness, the fallback rate and both static variants. Section 3.1 states the threshold rule.

AuthorsOpenReview ↗

Response to Reviewer dSAi (Part 2/2)

Q2. What the larger-model ablations establish

Question. Which result best supports the connection between recomputing assignments and the gains in the larger-model run?

What we have done and where. Both tests now run in the same load-cost regime, with 70B models and 52 s loads. Section 4.4 tests what the memory controller learns from a request's alternatives, and Appendix C tests recomputation.

Answer. The planner-input test is the clearest link, and it is your eviction example measured. With the planner and dispatch unchanged, a planner that sees only each request's cheapest acceptable route makes 4.0 loads per unit and answers 57.9% of issued requests correctly within 100 s. Given the candidate sets, it makes 1.0 load per unit and answers 74.3%. The gain is 16.4 points [11.0, 21.9], positive on all six units.

Evidence and experiments. FULL versus FIXED is the narrower test of recomputation alone. On two GPUs, both planners received the candidate sets and used the same dispatch. Recomputing assignments for each proposed resident set, instead of keeping the assignment fixed within a planning tick, changed goodput by 0.0 percentage points (paired 95% interval [−0.9, +0.9]). The planner was active in this test: CoServe changed the resident set in every unit.

Analysis. Taken together, the two tests show what matters at these load times: the alternatives reaching the memory controller, not recomputing assignments for each proposal. A route to a small model can make a load look necessary. With the candidate sets, the planner recognizes which of those requests the resident 70B model can already answer, and it keeps that model in memory.

Commitment to revise. Section 4.4 reports the planner-input result and points to the FULL/FIXED test in Appendix C.

Reviewer_ebS6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoServe keeps multiple predicted model choices per query and recomputes assignments when evaluating GPU resident sets. Its planner accounts for substitutions, queue costs, and model loading. An eleven-model B200 experiment reports higher 100-second completion rates than CARROT on two arrival schedules, but worse p95 latency. A separate three-model ablation tests the benefit of recomputing assignments during planning.

Critical strengths
  1. Preserving query-level alternatives is a useful design choice. Propositions 1-2 show how committing to one model can discard information needed for residency planning. The controller puts this idea into practice by recomputing assignments for each proposed set (Eqs. 6-8).

  2. The deadline gains over CARROT are 16.1-16.9 and 3.4-5.2 percentage points on the two schedules, even after accounting for timestamp uncertainty. The paper includes admission waiting and reports the worse tails. The same-build ablation in Appendix D also reduces p95 on three of four primary schedules.

Critical weaknesses
  1. It is unclear how much of the main deadline gain comes from recomputing assignments. The historical comparisons differ in build and configuration (Sec. 4.1; Appendix B.1). The same-build FULL/FIXED ablation helps, but uses three small models with 4-14 s activations, compared with roughly 44-50 s for the largest models in the main setting. Its deadline attainment is already 99.5-100%. This leaves a gap between the controlled mechanism test and the main deployment result. The reported intervals also do not quantify run-to-run variability.

  2. The quality comparison remains inconclusive. Main-run quality is an offline lookup, missing labels differ across policies, and 76 test prompts overlap with training (Table 1; Sec. 4.1). Static A is faster on all reported latency metrics, but its quality cannot be compared reliably. Appendix D grades live outputs, yet the average FULL-minus-FIXED correct-and-timely difference is bounded by [-6.75, +5.50] percentage points. Its static policy also relaxes the candidate restriction. The case for dynamic residency over simpler policies is therefore still unclear.

Questions for authors
  1. Which build and configuration differences remain in the main comparison? Can a matched FULL/FIXED test in the main activation-cost regime confirm that the scoring step accounts for the deadline gain? Please also specify the Eq. 7 settings used for Table 1.

  2. Are generated responses available for the main runs? Can you compare live correctness and timely correctness on common queries without training overlap, and compare static or simple candidate-aware policies under the same candidate restriction?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ebS6 (Part 1/2)

Thank you for identifying the gap between the small-model ablation and the main deployment. Based on your comments, we have significantly improved the paper. We ran both tests in the 52 s load regime: whether recomputing assignments helps, and what the planner loses when it receives a single route per request.

Q1. Build differences, a matched test of the scoring step, and the Eq. 7 settings

Question. (a) Which build and configuration differences remain in the main comparison? (b) Can a matched FULL/FIXED test in the main activation-cost regime confirm that the scoring step accounts for the deadline gain? (c) Which Eq. 7 settings were used for Table 1?

What we have done and where. (a) We replaced the main evaluation with a one-build study of five open models (Section 4.1 and Appendix A; Appendix D relates it to the submitted study). (b) We ran FULL/FIXED and a planner-input test at 52 s loads (Section 4.4 and Appendix C). (c) The settings are listed below and in Appendix D.

Answer. (a) None remain. All policies now use the same serving build and evaluation settings. (b) On two GPUs at 52 s loads, FULL minus FIXED is 0.0 goodput points [−0.9, +0.9]. FULL and FIXED both plan from the candidate sets and differ only in whether assignments are recomputed for each proposal, so recomputation is not what accounts for the gain; the candidate sets they share are. On one GPU, giving the same planner each request's cheapest acceptable route in place of its candidates costs 16.4 points [11.0, 21.9]. (c) The submitted Table 1 settings are listed below.

Evidence and experiments. (a) One build. Every policy uses the same executor, backend, client, predictor and memory setting. Table 1 identifies the two static variants that relax the candidate restriction. Every run starts a fresh backend. One B200 serves five open models with 308 GiB of weights against 176 GiB of usable memory; a 70B load takes 52 s.

(b) What the scoring step is given, at 52 s loads. The planning chain (Eq. 8 of the submission, Eq. 7 of the revision) scores each proposed resident set through the assignments it enables over C(q). We kept CoServe's planner and dispatch and changed only what the planner scores: committed routes (the cheapest acceptable model per request, as a cost-aware router sends them) instead of C(q).

Residency planner scores   Goodput   Model loads per unit
Assignments over committed routes.5794.0
Assignments over C(q) (CoServe).7431.0

The candidate-set planner led on all six units. (The .579 here is a separate run from the quality-router baseline, which has the same mean.)

(c) Eq. 7 settings for the submitted Table 1. w_q = 1.0; w_f = 0.3, with a causal 120 s projection from a 180 s window; aggregation A = mean; λ_load = 1.0; D_t enforced as dwell guards (at least 60 s, at most 300 s); η = 600 and a₀ = 60 s; replanning every 15 s. The revised scoring step replaces these weights with one objective, V(R) = the sum of p(q, m) over requests whose slot finishes by their deadline. It has three settings, all fixed on calibration data.

Analysis. A request routed to a small model appears to require a load. Its candidate set tells the planner whether the resident 70B model could also answer it, and the planner can then keep the 70B model in memory. That is why the committed-route planner makes four loads per unit where the candidate-set planner makes one. In the two-GPU FULL/FIXED test the planner was also active: CoServe changed the resident set in every unit.

Commitment to revise. Section 4.4 reports the planner-input test, Appendix C the FULL/FIXED comparison at 52 s loads, and Appendix D the Table 1 settings.

AuthorsOpenReview ↗

Response to Reviewer ebS6 (Part 2/2)

W1. Run-to-run variability

Question. The reported intervals do not quantify run-to-run variability.

What we have done and where. Every comparison in Section 4 now uses six independent units with paired intervals. We also reran units on identical schedules (Appendix A).

Answer. On identical arrival schedules, CoServe's goodput changed by 1.0 point on one rerun and 0.1 on the other.

Evidence and experiments. Across six disjoint units, CoServe led the strongest rule under the same candidate restriction (static Llama-70B with EDF dispatch and CoServe's concurrency limit) on every unit. The paired mean gain was 4.3 points [1.7, 6.9].

Analysis. The reruns measure repeatability on a given schedule. The paired interval measures how the policy difference varies across schedules.

Commitment to revise. Section 4 and Appendix A report the paired intervals and the reruns.

W2/Q2. Live correctness, training overlap, and simple policies under the same restriction

Question. Are generated responses available? Can live correctness and timely correctness be compared on common queries without training overlap, and can static or simple candidate-aware policies be compared under the same candidate restriction?

What we have done and where. Every response in the revised evaluation is generated live and graded. No test prompt has a near-duplicate in the predictor's training data. We added static and simple candidate-aware policies under the same candidate restriction (Section 4.2 and Appendix A).

Answer. The historical logs retain answer text for four of the seven Table 1 runs (CoServe A and B, Static A, and CARROT B). The revised comparison does what you describe: each policy receives the same issued queries, and every completed answer is graded. CoServe answers 74.3% of issued queries correctly within 100 s. Static Llama-70B with EDF and the same concurrency limit answers 70.0%.

Evidence and experiments. One GPU, six units, same candidate restriction for every row:

Policy   Correct within 100 s   Served   On time   Correct among served
CoServe.74396.1%83.8%85.1%
Static Llama-70B, EDF, same concurrency limit.70093.2%80.4%85.2%
INFaaS-style, EDF, same concurrency limit.69596.8%80.0%84.3%
Static Llama-70B.58493.2%68.1%85.3%
Quality router + backend.57987.5%69.4%83.1%
CARROT-style cost-aware router + backend.39683.0%51.4%80.5%

"Served" means dispatched and completed, on time or late. The static small set under the same restriction scores .391.

Analysis. Accuracy among served requests is nearly the same for CoServe and the static 70B rule (85.1% and 85.2%), so the difference is in which answers arrive in time. CoServe met the deadline for 83.8% of issued requests, against 80.4%. Among the answers that arrived on time, 88.7% and 87.0% were correct.

Commitment to revise. Table 1 reports goodput from live grading. Appendix A reports the served, on-time and accuracy fractions.

Reviewer_qLnwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents CoServe, a multi-LLM serving system that jointly considers request routing and GPU model residency under memory constraints. The key observation is that conventional routing systems commit each request to a single model before residency decisions are made, potentially causing unnecessary model loading and eviction even when alternative resident models could serve the request. CoServe instead preserves a set of predicted acceptable models for each request and evaluates candidate GPU-resident model configurations by recomputing request assignments and estimating serving costs, waiting penalties, and model activation overheads. Experiments on a single NVIDIA B200 GPU with 11 models show improvements in the fraction of requests completed within 100s compared with baseline solutions.

Critical strengths

S1: Multi-LLM serving is an interesting and practically relevant problem. S2: The paper formulates the interaction between routing and residency decisions and proposes a unified cost model incorporating model coverage, estimated batch-service costs, request waiting penalties, and model activation costs.

Critical weaknesses
  1. Limited technical novelty. The core idea of jointly considering model routing and resource allocation is relatively straightforward and has been explored in related systems. CoServe has proposed an heuristic approach. However, the technical distinction from existing joint routing and resource management approaches is not sufficiently compelling.

  2. The main evaluation considers only a single B200 GPU and two arrival realizations, each involving 1,000 requests. This is a limited setting for evaluating a multi-model serving system.

Questions for authors
  1. Could the authors clarify the fundamental technical differences between CoServe and prior multi-LLM serving systems?
  2. Since the main motivation for multi-model routing is to balance response quality and serving efficiency, could the authors report end-to-end quality-constrained goodput, such as the number of correctly answered requests completed within the deadline? How sensitive are the results to inaccuracies in the predicted acceptable model sets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer qLnw (Part 1/3)

Thank you for the review and for the questions on novelty, scale and the quality metric. Based on your comments, we have significantly improved the paper. The revised paper replaces the evaluation with a one-build study (five open models, live grading, six test units) and states the scoring step as one objective. Its primary metric is the one you asked for: the fraction of issued requests answered correctly within 100 s.

W1/Q1. Novelty and differences from prior multi-LLM serving systems

Question. What are the fundamental technical differences between CoServe and prior joint routing and resource-management approaches?

What we have done and where. We implemented the decision rule of each prior system as an analogue in the same executor, with the same predictor, backend, memory budget and candidate rule (Section 4.1 and Appendix A of the revised paper). Table 1 (Section 4.2) gives the comparison, and Section 5 the positioning.

Answer. The difference is what reaches the memory decision. In prior systems, the layer that loads and evicts models sees one model per request (RouteLLM- and CARROT-style routers, QLM) or per-model arrival rates (Proteus). INFaaS does consider acceptable alternatives, but for one arriving request at a time, and it loads on a miss. SneakPeek chooses models for timely accuracy without a memory limit or a resident set. CoServe keeps every waiting request's candidate set, with predicted correctness, and makes the memory decision over the whole queue. Before it evicts a model, it checks which waiting requests another resident model could still answer by their deadlines. It scores each proposed resident set by assigning the queue to the batch slots that set would provide.

Evidence and experiments. One B200 serves five open models with 308 GiB of weights against 176 GiB of usable memory, so each 70B model runs alone and a 70B load takes 52 s. Each of six units replays 1,800 held-out requests at 3.0 requests/s.

Decision rule   Serving layer receives   Residency driven by   Goodput   CoServe − rule (pp)
QLM-styleone model per requestdeadline-ordered swaps.196+54.7 [+47.5, +61.9]
CARROT-style cost-aware router + backendone model per requestload on miss.396+34.7 [+32.9, +36.5]
Proteus-stylerates per modelarrival-rate plan.456+28.7 [+16.7, +40.7]
SneakPeek-stylemodel chosen for timely accuracychosen models.523+22.0 [+9.3, +34.6]
INFaaS-styleacceptable modelsload on miss.576+16.8 [+5.5, +28.1]
Quality router + backend (RouteLLM-style)one model per requestload on miss.579+16.4 [+13.8, +19.1]
CoServecandidate set and predicted correctnesswaiting requests over their candidate sets.743

Goodput is the fraction of issued requests answered correctly within 100 s. Intervals are paired 95% intervals over the six units. Every contrast stays significant after Holm correction over the 14 baselines of Table 1.

Analysis. The closest rule is INFaaS-style dispatch given EDF ordering and CoServe's concurrency limit. It uses the candidates at dispatch only and scores .695. CoServe scores .743, a gain of 4.8 points [2.1, 7.5]. Among joint formulations, RouterWise and Cascadia plan routing together with resource allocation for pools that fit their hardware, StageRoute chooses the deployed set under a count limit from per-model statistics, and POLAR caches LoRA adapters. None of them decides full-model loads and evictions from the waiting requests' alternatives and deadlines. The next comment measures what the candidate set is worth at each decision.

Commitment to revise. Table 1 (Section 4.2) and Appendix A give the comparison and the definition of each rule. Section 5 positions CoServe against QLM, Proteus, INFaaS, SneakPeek, RouterWise, StageRoute, Cascadia and POLAR.

AuthorsOpenReview ↗

Response to Reviewer qLnw (Part 2/3)

W1 (continued). What changes when the serving layer sees the candidates?

Question. Is CoServe more than a heuristic, and is its distinction from joint routing and resource management measurable?

What we have done and where. The scoring step is now a single objective in place of hand-set penalty weights (Section 3). Three controlled tests each change one thing (Sections 4.3 and 4.4).

Answer. The controller now optimizes one stated quantity, the expected number of correct answers delivered by their deadlines, and the distinction is measurable at both decisions. At dispatch, with EDF and the concurrency limit held equal, passing the candidate set raises goodput from .586 to .695. At the memory decision, with CoServe's planner and dispatch unchanged, replacing the candidate sets by one route per request lowers goodput by 16.4 points.

Evidence and experiments. The objective. The serving layer assigns each waiting request q to a candidate m in C(q) and a batch slot on m. A resident set R is valued by V(R), the expected number of correct answers it delivers by their deadlines: the sum of p(q, m) over requests whose slot finishes in time. This is a transportation problem, solved with HiGHS in 13 ms per planning tick at the median. Three settings (switching margin, horizon, late weight) were fixed on calibration data.

(a) Dispatch, one GPU, six units. The first two rows use the same EDF dispatch and concurrency limit.

Policy   Goodput   Gain (pp)   Model loads per unit
Committed quality router.5862.0
Pass the candidate set (INFaaS-style).695+10.9 [+6.2, +15.5]1.3
CoServe.743+4.8 [+2.1, +7.5]1.0

The final row adds valued assignment and residency planning together. Their combined gain over the candidate-aware EDF policy is 4.8 points.

(b) The residency planner's input, one GPU, six units. We kept CoServe's planner and dispatch and gave the planner committed routes (the cheapest acceptable model per request, as a cost-aware router sends them) instead of candidate sets. Goodput falls from .743 to .579, −16.4 points [−21.9, −11.0], lower on all six units. Model loads rise from 1.0 to 4.0 per unit. (This is a separate run from the quality-router baseline of Table 1; the two means coincide.)

(c) Commitment with both models resident, two GPUs, six units. Llama-70B and OpenMath-70B are held resident, one per GPU. About 730 of every 1,800 requests list both as candidates.

Dispatch   Commit to the most accurate candidate   Choose within C(q)   Cost of committing (pp)
EDF.498.64214.4 [12.9, 15.8]
CoServe's valued assignment.662.7084.5 [3.5, 5.6]

Analysis. (a) and (c) show that dispatch needs the candidate set to use resident models well. (b) shows that the residency planner needs it too. Fed committed routes, the planner sees demand that only the small models can serve, and it swaps the 70B model out. This is the setting of Propositions 1 and 2, and the result is consistent with them. The two-GPU result is consistent with Proposition 3: a choice made after observing serving state cannot do worse than a commitment made before it.

Commitment to revise. Section 3 states the objective and the solver. Sections 4.3 and 4.4 report tests (a)–(c).

AuthorsOpenReview ↗

Response to Reviewer qLnw (Part 3/3)

W2. Scale of the evaluation

Question. The main evaluation uses a single B200 and two arrival realizations of 1,000 requests. Is this enough to evaluate a multi-model serving system?

What we have done and where. We replaced the main evaluation with six independent test units of 1,800 requests each, and added a two-GPU setting with six more units (Section 4 and Appendix A).

Answer. The evaluation now covers 10,800 requests per policy on one GPU and a second setting with two GPUs at twice the arrival rate. CoServe leads in both.

Evidence and experiments. One GPU. Each policy serves six disjoint units. CoServe leads all 14 baselines, and every contrast stays significant after Holm correction. The strongest rule under the same candidate restriction is static Llama-70B with EDF dispatch and CoServe's concurrency limit. CoServe leads it by 4.3 points [1.7, 6.9], with a positive difference on every unit.

Two GPUs. Two backend servers sit behind one controller at 6 requests/s, over six units.

Two GPUs, six units   Goodput   CoServe − policy (pp)
CoServe.718
EDF with the same concurrency limit, Llama-70B + OpenMath-70B resident.642+7.6 [+6.4, +8.7]
EDF with the same concurrency limit, Llama-70B + three small models resident.650+6.8 [+4.8, +8.8]

Variability. Two one-GPU units rerun on identical schedules moved CoServe by 1.0 and 0.1 points.

Analysis. With two GPUs, a request can have an acceptable model on either device, so the choice among candidates also balances load across devices. CoServe leads EDF on both placements tested.

Commitment to revise. Section 4 and Appendix A describe the units and the two-GPU placements.

Q2. Quality-constrained goodput and sensitivity to wrong candidate sets

Question. Can the authors report correct answers completed within the deadline? How sensitive are the results to inaccuracies in the predicted acceptable model sets?

What we have done and where. The metric you describe is now the primary metric everywhere (Section 4.1). We also perturbed the candidate sets directly (Appendix A).

Answer. Yes. Every goodput figure in this response is that metric. The denominator is every issued request. A request counts only if its graded answer is correct and arrives within 100 s of the request's arrival. For sensitivity, we flipped 10% of candidate memberships at random on two test units. The valued assignment, with the resident set held at its initial state, went from .734 to .714 on those units.

Evidence and experiments. Building the sets. The threshold τ = 0.80 was chosen on held-out calibration data. At this threshold, routing each request to its cheapest candidate loses 1.0 point of accuracy against routing it to its most accurate model. For 33.1% of test requests no model reaches τ; these go to the highest-scoring model, identically in every policy.

Perturbing the sets. On test units 1 and 2, 10% of the memberships were flipped at random, which both adds models to C(q) and removes them. The test isolates the assignment step: the resident set stays at its initial state, and the valued assignment serves the perturbed sets.

Analysis. Calibration measures the accuracy cost of serving a request on its cheapest candidate. The perturbation changes the assignment's goodput by two points when one membership in ten is wrong.

Commitment to revise. Section 4 reports goodput as the primary metric, and Section 3.1 states the threshold rule. Appendix A reports the membership perturbation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the three reviewers for their careful reviews. Based on your comments, we have significantly improved the paper. The revised PDF is uploaded, with new and changed text in blue. This note summarizes what changed and where. The individual responses give the details.

Scope of the revision. The title, the problem, the candidate interface (the router passes each request's set of acceptable models, not one model) and Propositions 1–3 are as submitted. The controller's scoring step and the evaluation are revised, as listed below.

Change in the revision   Addresses   Where
One build for every policy, on one pool of five open models, with a fresh backend per runebS6 Q1Section 4.1, Appendix A
Every answer generated live and graded; no test prompt has a near-duplicate in trainingebS6 Q2, dSAi Q1Section 4.1
Correct answers within 100 s, over all issued requests, as the primary metricqLnw Q2Section 4.1
Six disjoint test units of 1,800 requests, paired intervals, Holm correction and reruns; a two-GPU settingqLnw W2, ebS6 W1Section 4, Appendix A
The decision rules of QLM, Proteus, INFaaS and SneakPeek, RouteLLM- and CARROT-style routers, and static and EDF rules under the same candidate restriction, all in one executorqLnw Q1, ebS6 Q2, dSAi Q1Section 4.2, Appendix A
The scoring step replaced by one objective, the expected number of correct answers delivered by their deadlines, in place of hand-set weights; the Eq. 7 settings of the submitted Table 1 are listedqLnw W1, ebS6 Q1Section 3, Appendix D
A planner-input test and FULL/FIXED at 52 s load timesebS6 Q1, dSAi Q2Section 4.4, Appendix C
The fallback rate, both static variants and the threshold ruledSAi Q1Section 3.1, Appendix A

Results that bear on all three reviews. One B200 serves five open models with 308 GiB of weights against 176 GiB of usable memory, over six test units.

  1. CoServe answers 74.3% of issued requests correctly within 100 s. It leads all 14 baselines after Holm correction. The lead is 4.3 points [1.7, 6.9] over the strongest rule under the same candidate restriction and 34.7 points [32.9, 36.5] over a CARROT-style cost-aware router.
  2. With EDF dispatch and the concurrency limit held equal, passing the candidate set instead of a committed route raises goodput from .586 to .695, a gain of 10.9 points [6.2, 15.5].
  3. With CoServe's planner and dispatch unchanged, giving the planner one route per request instead of the candidate set lowers goodput by 16.4 points [11.0, 21.9]. Model loads rise from 1.0 to 4.0 per unit.
  4. FULL and FIXED both plan from the candidate sets and differ only in whether assignments are recomputed for each proposal. Their difference is 0.0 points [−0.9, +0.9] on two GPUs, so the gain is tied to the candidate sets reaching the planner.
  5. On two GPUs with both 70B models resident, committing each request to its most accurate candidate costs 14.4 points [12.9, 15.8] under EDF and 4.5 points [3.5, 5.6] under CoServe's assignment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.