acceptodds
Under review as a conference paper at ICLR 2027

ARI: Adaptive Rule-Interpolated KV Cache Migration Scheduling for Multi-GPU LLM Serving

Abstract

Large language model serving maintains a request-level key-value (KV) cache that grows during decoding. Although PagedAttention reduces within-instance fragmentation, request growth and completion can imbalance multi-GPU placement. Existing routing and migration mechanisms leave request-level migration value under interacting memory, transfer-cost, and remaining-duration conditions insufficiently modeled. We present , an adaptive rule-interpolated decision layer for runtime Live-KV migration. After a global imbalance detector fires, candidates are represented by source pressure, KV size, estimated migration cost, global imbalance, and remaining execution time. Gaussian rule matching and normalized consequent interpolation yield continuous migration priorities. A separate destination controller plans feasible moves on a virtual cluster state, stopping at the first balanced prefix along its greedy path (minimum Top-) when one exists. The priority rule base grows only in dense, insufficiently covered regions, and observed post-migration utility updates rule confidence. retains MELL's initial allocation and reuses request-level Direct-KV or Token-RePrefill execution paths. Across six model–platform settings on 4 and 8 GPUs, achieves higher output-token throughput than MELL. On 4-GPU Llama-2-7b-chat, throughput increases from 436.82 to 675.31 tokens/s (54.6%) and P95 latency decreases by 30.9%. In selected settings, also improves serving performance or migration efficiency over other baselines. Upon acceptance, we will publicly release all source code and experimental scripts to support reproducibility.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model Serving, KV Cache Migration, Multi-GPU Scheduling, Adaptive Rule Interpolation, Load Balancing, Online Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qNCiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses runtime KV-cache imbalance in multi-GPU LLM serving. Although PagedAttention and continuous batching improve within-instance memory management, request growth and completion can create cross-instance imbalance, causing queueing, preemption, and tail-latency spikes. The authors observe that existing systems (Llumnix, MELL, L4) lack a load-driven, request-level migration decision that models the conditional value of a KV cache, and that fixed Top-1/Top-3 migration-set sizes either under- or over-migrate. The paper proposes ARI, a decision layer built on top of MELL's initial allocator and request-level execution paths (Direct-KV transfer or Token-RePrefill). Its three main components are:

  1. Rule-interpolated migration priority: Five normalized features (source pressure, request KV size, estimated migration cost, global imbalance, remaining execution time) are matched against a sparse rule base via Gaussian kernels; normalized consequent interpolation yields a continuous migration priority, capturing conditional feature interactions that a fixed additive linear score cannot express (formally shown in Proposition 7).
  2. Online rule growth and confidence update: Post-migration utility (balance improvement, memory released, migration time, extra latency) feeds back to update rule confidence (Eq. 4), and recurrent, poorly covered states in a dense region of the feature buffer trigger the creation of new rules (Eq. 5), with density/coverage tests restricting growth.
  3. Destination-aware minimum Top-k planning: A fixed destination score selects feasible targets on a virtual cluster state; the planner greedily applies CV-reducing moves and stops at the first balanced prefix (Eq. 6), avoiding over-migration without searching for a globally minimal migration subset.

Experiments on 4-GPU and 8-GPU A100 testbeds with Qwen2.5-1.5B, Llama-2-7B, and Llama-2-13B, against MELL, Llumnix, FastServe, L4, and a Linear Score ablation, show throughput gains over MELL in all six settings (6.7%–151.5%), e.g., 436.82→675.31 tokens/s (+54.6%) with 30.9% lower P95 on 4-GPU/7B, and first place in four of six settings. Ablations attribute most benefit to remaining-duration and adaptive components.

Critical strengths
  1. Conditional, non-additive candidate valuation. Unlike the fixed linear scoring of prior systems, ARI's Gaussian rule interpolation explicitly models conditional interactions — a large KV cache is valuable only when source pressure is high, transfer is cheap, and substantial decoding remains — and this advantage is backed by a formal separation result (Proposition 7) showing orderings no affine score can realize.
  2. Online rule-base evolution with controlled growth. The combination of coverage/density gating (Eq. 5) and observed-utility confidence updates (Eq. 4) allows the priority mechanism to adapt at runtime while provably bounding the feedback state (Proposition 4), addressing the previously unverified "how should the rule base evolve" question in this scheduling framework.
  3. Destination-aware minimum Top-k planning. Coupling request ranking, destination selection, and migration-set size in one planner — stopping at the first balanced greedy prefix (Eq. 6) — directly answers the unanswered "how many to move" question, and the ablation shows it trades modest throughput for 33.3% fewer migrations versus fixed Top-k.
  4. Clean separation of concerns with honest scoping. Planning, reservation, execution, and adaptation are clearly delimited; ARI reuses MELL's initial allocation and Direct-KV/Token-RePrefill execution paths rather than redesigning them, and the appendix carefully distinguishes decision-time vs. feedback-time predictions, NE vs. N/A vs. zero, and planned vs. committed migrations.
  5. Demonstrated gains where conditions are favorable. On 4-GPU/7B, ARI achieves +54.6% throughput and −30.9% P95 over MELL, a 33.7% throughput advantage over the Linear Score ablation with 19.1% fewer migrations at nearly unchanged P95, and a clear joint benefit over Llumnix on 8-GPU/13B (+25.3% throughput, −3.7% P95, −31.2% migration execution time).
Critical weaknesses
  1. The 4-GPU/13B failure case is unexplained and undermines the contribution claim. ARI's throughput (268.77) is not only below Llumnix (320.36) and FastServe (310.02) but below its own ablation baseline Linear Score (308.11), and P95 is uniformly worse; the paper only states that "the observed advantage concentrates on the tested 7B configurations" (lines 408–409) without analyzing the root cause — a clear tension between the contribution statement and the experimental evidence.
  2. The benefit–cost account of "smarter scheduling" is not settled. Table 8 shows ARI's planning time (6.56 ms) is 22× that of Llumnix (0.30 ms), and on 4-GPU/7B ARI performs fewer migrations than MELL (10.03 vs. 11.00) yet takes longer migration execution time (2,875.02 vs. 1,940.73 ms, lines 427–429). Whether the added planning/execution cost is justified across settings is left as aggregate, setting-dependent observations.
  3. Statistical reporting contradicts the stated protocol. The paper claims "rigorous run-level statistical analysis" (line 719), yet sample standard deviations exist for only one of 18 configurations (Table 7, 4-GPU/13B), with no confidence intervals or significance tests anywhere, and run-level raw records, failed trials, and exclusion counts are absent from the aggregate files (lines 415–418).
  4. Adaptation does not demonstrably improve prediction. Prediction error remains 0.30828 with or without confidence updates (Table 4 discussion, line 452), the confidence update is explicitly "not guaranteed to move toward the target," and the repeated-identical-feedback example in Appendix B shows the default update can leave predictions stuck — so the online-learning component's causal benefit is established only through P95 ablation numbers in one setting.
  5. Key execution-path claims lack commit-level evidence. Backend selection (Direct-KV vs. Token-RePrefill) is "separate from scoring" with an auto setting that "does not establish which backend executed" (lines 284–285); without per-run commit records, the migration-efficiency comparisons (transfer bytes, re-prefill tokens, execution time) cannot be attributed to a specific execution path.
Questions for authors
  1. 4-GPU/13B regression: What is the root cause of ARI underperforming its own Linear Score ablation on 4-GPU/13B (268.77 vs. 308.11 tokens/s, higher P95, higher KV imbalance 0.4402 vs. 0.3197)? Can you decompose the failure into feature-scale, rule-matching, or destination-selection effects, and does the trained rule base differ qualitatively from the 7B case?
  2. Overhead accounting: Given planning time of 6.56 ms (22× Llumnix) and longer migration execution time than MELL on 4-GPU/7B despite fewer migrations, under what workload/cluster conditions does the benefit–cost balance of ARI turn negative? Please provide a per-setting break-even analysis and clarify which backend (Direct-KV vs. Token-RePrefill) actually executed each migration via commit records.
  3. Statistical rigor: The paper claims "rigorous run-level statistical analysis," but standard deviations exist only for 4-GPU/13B. Will you release run-level raw records and report confidence intervals or paired significance tests for all 18 configurations, particularly for the 7B settings where the headline claims rest on single-point aggregate numbers?
2 comments

We thank the reviewer for the detailed assessment. Our responses below draw on the submitted manuscript and appendix, together with quantitative reanalysis of retained experimental records. We have retained run-level results and migration-event logs for the archived experiments discussed here. We are prepared to provide the relevant supporting records through an anonymized supplementary artifact, with explicit links to the corresponding tables, configurations, and runs, to facilitate verification.

Response to Question 1 and Weakness 1: Understanding the 4-GPU/13B Regression

ARI’s contribution is a request-level KV-cache migration policy for multi-GPU LLM serving, combining context-dependent priority interpolation, destination-aware virtual planning, and state-dependent stopping. The four-GPU/13B comparison is particularly informative because it tests ARI against its own Linear Score baseline on both serving performance and KV balance.

Our analysis separates three stages:

Feature-scale alignment. Model geometry and decoding speed affect KV-byte and remaining-duration inputs relative to the configured normalization scales.

Rule activation. In a separate 13B diagnostic trace, 11 of 12 candidates at the first recorded decision have a zero remaining-duration feature. Their priority scores cluster around 0.5, with a total span of approximately 5.95×10−65.95 \times 10^{-6}. This provides a concrete example of weak feature and priority separation at a decision point.

Destination selection and execution. Candidate priority is followed by capacity checks, destination scoring, and virtual imbalance evaluation. The accepted migration therefore depends on the interaction between ranking and target selection. The snapshot-level imbalance condition also differs from the time-averaged imbalance measured during continuing arrivals and KV growth.

The implementation initializes rule prototypes from configuration and adapts the priority library online. Model-specific behavior is therefore reflected in feature distributions, rule activation, and subsequent growth. We will present the frozen initial configurations alongside these activation profiles and use controlled replays with initial placement and destination selection held fixed to isolate the ranking contribution.

Response to Question 2 and Weaknesses 2 and 5: Overhead Accounting and Executed Backends

The benefit–cost account requires accumulated scheduling cost and actual migration cost, rather than migration count alone. Fewer migrations can take longer when the selected requests carry larger contexts or incur longer execution pauses.

Our retained records already provide a stage-level breakdown. Across 134 recorded ARI migration-decision events in the historical eight-GPU 7B track, average priority computation takes 0.454 ms, virtual planning takes 32.923 ms, and total scheduler time is 33.903 ms. Priority computation accounts for approximately 1.34% of the recorded scheduler time. Virtual planning is the dominant measured scheduler component in these events.

For a migrated request with nn remaining tokens, a simple amortization condition is

n ΔTPOT>Textra,n\,\Delta\mathrm{TPOT} > T_{\mathrm{extra}},

where ΔTPOT\Delta\mathrm{TPOT} is the expected per-token improvement and TextraT_{\mathrm{extra}} is the incremental pause and queueing cost on the request’s critical path. Using the quoted 2.875 seconds as an illustrative pause budget, amortization requires more than 11.23 ms/token improvement over 256 remaining tokens, or 5.62 ms/token over 512 tokens, before additional queueing costs. This identifies the unfavorable regime: expensive migrations of short-lived requests, or destinations offering insufficient decoding improvement.

Executed-backend identities are also recorded. In the archived eight-GPU 7B auto matrix, all 481 committed migrations executed Token-RePrefill, including 134 ARI commits. Separately, the fixed Direct-KV 1.5B archive contains 495 verified Direct-KV commits. These records distinguish backend availability, selector configuration, and the path actually executed.

The supporting evidence can provide selected and executed modes, transferred bytes, re-prefill tokens, execution times, and commit/rollback outcomes. Scheduling costs will be associated with their actual invocation counts, including recorded empty-plan invocations, rather than inferred from migration counts.

Response to Question 3 and Weakness 3: Statistical Reporting and Supporting Records

The aggregate tables are not the full extent of the retained evidence. Complete archived comparison groups include per-run records that support paired reanalysis.

For example, in the historical four-GPU 7B confidence-on/off comparison, the mean paired change in per-run end-to-end P95 latency is −625.80 ms, with a paired 95% confidence interval of [−3314.20, 2062.60] ms. This distinguishes a favorable point estimate from a statistically established improvement.

We are prepared to provide the corresponding per-seed results and available request/event records, together with experiment versions, workload identifiers, seeds, and archive hashes. The evidence index will also identify completion, failure, rerun, and exclusion records. Complete paired experiments and aggregate-only summaries will be explicitly distinguished, so that uncertainty and significance claims are tied to the records supporting each comparison.

The submitted manuscript and appendix provide the primary results and analyses; these additional records provide the finer-grained evidence needed to audit the comparisons.

Response to Weakness 4: Prediction Quality and Adaptation Benefits

Confidence adaptation affects relative rule influence and candidate ordering. Aggregate utility-prediction MAE therefore does not fully characterize its scheduling value. Our historical confidence-on/off ablations show favorable point estimates for P95 TPOT and end-to-end P95 latency in both model settings: the respective reductions are 3.74% and 8.91% for 1.5B, and 7.21% and 7.27% for 7B. The paired interval reported above illustrates the uncertainty accompanying these observations.

The repeated-feedback example identifies a specific limitation of the original heuristic update. We reproduced this behavior on CPU: with two equal-match rules having consequents 0.2 and 0.8, the original update keeps the prediction at 0.5 under repeated toy feedback of 0.8. A separately implemented normalized-gradient variant with projection and backtracking reduces absolute error from 0.3000 to approximately 0.2379 after 100 updates, using a learning rate of 0.05 in the same construction.

This is a mechanism-level result for the separate variant, not a GPU performance result or a retrospective claim about the original experiments. It provides a concrete direction for improving credit assignment. Targeted matched-trace comparisons of original, frozen, and normalized-gradient confidence updates will assess ranking quality, pre-update error, and serving outcomes across additional model sizes and GPU configurations.

Reviewer_d4eFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ARI, a scheduler for runtime KV-cache migration in multi-GPU LLM serving. It combines Gaussian rule-based request ranking, destination-aware greedy planning, and online adaptation. The planner accepts only migrations that reduce KV imbalance and stops at the first balanced prefix. Experiments on four- and eight-GPU A100 platforms show higher output-token throughput than MELL in all six model–platform settings. On four-GPU Llama-2-7B, throughput increases by 54.6% , while P95 latency drops by 30.9%.

Critical strengths
  • The paper addresses a practical problem: KV-cache usage changes during decoding, so initially balanced placement can become uneven. Request selection, destination choice, and migration-set size are closely coupled.
  • The planner updates a virtual cluster state after each accepted migration and reassesses subsequent destinations. Its stopping rule avoids unnecessary moves once balance is restored, without claiming global optimality.
  • The appendix provides pseudocode, hyperparameters, complexity analysis, and formal properties. Proposition 7 shows that Gaussian rule interpolation can represent certain conditional rankings that linear scoring cannot.
  • The evaluation covers three models, two GPU configurations, multiple baselines, and component ablations, offering useful evidence of both performance gains and design trade-offs.
Critical weaknesses
  • The core innovation remains incremental. Gaussian rule interpolation and online adaptation are established techniques, and the paper does not show that the learned rules capture useful feature interactions in practice. Proposition 7 establishes representational capacity, not improved scheduling performance.
  • Performance varies across configurations. On four-GPU Llama-2-13B, ARI achieves 268.77 tokens/s, below Llumnix (320.36) and Linear Score (308.11), with higher P95 latency than both. On eight-GPU Llama-2-7B, ARI improves throughput over Llumnix but has substantially higher P95 latency. The reasons for these differences remain unclear.
  • The benefit of online adaptation is not convincingly established. Disabling rule growth slightly increases throughput (679.63 versus 675.31 tokens/s), while disabling confidence updates reduces it by only about 1.1%. Prediction error remains 0.30828 with or without confidence updates. Although both ablations worsen P95 latency, the results do not establish that adaptation improves prediction quality or consistently benefits serving performance.
  • The long-term scalability of rule growth remains unclear. The feedback buffer is bounded, but rules are never merged or deleted, leaving the rule base without a hard size limit. Moreover, feedback covers only attempted migrations, so the utility of skipped candidates remains unknown. The paper does not quantify long-term rule growth or its impact on scheduling overhead.
  • The evaluation does not fully establish robustness. Experiments use A100 PCIe platforms without observed NVLink, and run-level uncertainty estimates are unavailable for all configurations. The accuracy of the remaining-duration and migration-cost estimates, and their effects on scheduling decisions, are also insufficiently evaluated.
Questions for authors
  1. Can the authors show that rule interpolation improves candidate ranking and migration outcomes over Linear Score on held-out workloads?
  2. What explains ARI's underperformance on four-GPU Llama-2-13B and its higher P95 latency than Llumnix on eight-GPU Llama-2-7B?
  3. How does the rule-base size evolve during sustained serving, and how does its growth affect planning overhead?
  4. How sensitive is ARI's end-to-end performance to errors in the remaining-duration and migration-cost estimates?
5 comments

We thank the reviewer for the detailed comments. We clarify ARI’s contribution below and supplement the discussion with quantitative reanalysis of archived experiments.

Response to Question 1 and Weakness 1: Contribution to Multi-GPU Scheduling**

ARI’s core contribution is a request-level KV-cache migration policy for multi-GPU LLM serving. It couples context-dependent priority interpolation with destination-aware virtual planning and state-dependent stopping to coordinate running requests across GPU workers under dynamic memory pressure. This formulation addresses which requests to migrate, where to migrate them, and how much migration the current imbalance requires.

The scheduling rationale is conditional: similarly sized KV requests can merit different priorities depending on migration cost and remaining duration. Proposition 7 establishes that the interpolation model can represent conditional orderings unavailable to a fixed linear score. We use this result as a representational foundation, not as empirical proof that the learned rules improve performance.

Direct held-out ranking validation remains an important extension. We will compare ARI, Linear Score, and a fully frozen rule base using identical features, placement, destination selection, and execution. Candidate-level evaluation will distinguish simulation-derived utility labels from measured GPU feedback.

Response to Question 2 and Weakness 2: Configuration-Dependent Performance

The cited comparisons identify different operating trade-offs: ARI underperforms Llumnix and Linear Score in the four-GPU Llama-2-13B setting, while its throughput advantage over Llumnix in the eight-GPU Llama-2-7B setting accompanies higher P95 latency.

Our reanalysis already provides a concrete breakdown of scheduler overhead. Across 134 recorded ARI migration-decision events in the historical eight-GPU 7B track, average priority computation takes 0.454 ms, virtual planning takes 32.923 ms, and total scheduler time is 33.903 ms. Priority computation accounts for approximately 1.34% of the recorded scheduler time. Thus, virtual planning—not kernel scoring—is the dominant measured scheduler component in these events.

Migration execution and destination waiting can additionally concentrate delays on particular requests, allowing aggregate throughput and P95 latency to move in different directions. These mechanisms are consistent with the observed trade-offs, but the scheduler timing alone does not establish their causal contribution to every cited configuration. We will separate source pauses, destination waiting, and migrated context lengths for the corresponding comparisons.

Response to Weakness 3: Scheduling Value of Online Adaptation

Existing confidence-on/off ablations provide decision-relevant evidence beyond aggregate prediction MAE. In the historical four-GPU experiments, enabling confidence updates reduces the mean of per-run P95 TPOT by 3.74% and 7.21%, and end-to-end latency P95 by 8.91% and 7.27%, in the 1.5B and 7B settings, respectively.

These are favorable point estimates, although the paired 95% confidence intervals include zero. They provide preliminary evidence of potential scheduling benefits, rather than conclusive evidence of an independent learning effect. Aggregate MAE also does not fully characterize candidate-ordering quality, but its equality cannot itself establish that adaptation is useful.

We will strengthen this evaluation through separate confidence-on/off, growth-on/off, and fully frozen comparisons across additional model sizes and four-/eight-GPU configurations. The evaluation will report actual ranking changes, pre-update prediction error, growth activation, and serving outcomes under matched traces and paired seeds.

Response to Question 3 and Weakness 4: Rule Growth and Planning Overhead

The archived traces confirm that rule growth is active in the evaluated system. Across the five historical eight-GPU 7B ARI runs, recorded decision-time libraries contain 3–13 rules, and the final recorded feedback states contain 9–13 rules. These observations characterize growth within the measured runs; they do not establish long-term saturation.

Growth is selective: a new priority rule is eligible only when both the local-density and insufficient-coverage criteria are satisfied. The feedback buffer is bounded, while the rule library currently has no fixed cap or merging/deletion mechanism. Its finite-history bound is R≤R0+FR\leq R_0+F, where FF is the number of feedback calls.

Scoring NN candidates against RR rules requires O(NRd)O(NRd) kernel work, with d=5d=5. The measured timing above quantifies this cost at the observed library sizes. Sustained-serving measurements and controlled rule-count microbenchmarks are needed to assess larger libraries and motivate an explicit rule budget or compression mechanism.

Skipped candidates’ utilities remain unobserved. We will preserve that distinction: simulation-derived counterfactual utilities will be reported separately from measured migration feedback.

Response to Question 4 and Weakness 5: Estimation Sensitivity and Robustness

Existing development experiments show that cost inputs materially affect scheduling decisions. In a separate three-seed four-GPU 13B comparison, changing the cost-calibration inputs increases mean migration count from 11.33 to 14.67 per run, while output throughput changes from 259.08 to 257.03 tokens/s. This demonstrates sensitivity in that development setting and shows why calibration accuracy alone does not guarantee better serving performance. It is not a controlled error-tolerance test.

ARI nevertheless separates feasibility from priority estimation. On a consistent snapshot with accurate KV accounting, accepted virtual migrations must respect modeled capacity constraints and strictly reduce imbalance. Remaining-duration and migration-cost errors can affect selection quality without bypassing these checks.

We will quantify that effect through controlled over- and underestimation of each input, separately and jointly, reporting ranking changes, migration decisions, and serving performance with paired uncertainty estimates. Hardware conclusions will remain scoped to the evaluated A100 PCIe platforms; broader interconnect claims require separately calibrated measurements.

Thank you for the additional analyses. The new results help clarify the scheduling overhead and the potential benefits of confidence updates. However, I still have concerns about whether rule interpolation improves candidate ranking in practice and whether the observed latency gains from online adaptation are statistically reliable, given that the confidence intervals include zero. Could the authors further clarify these points, particularly regarding the actual ranking changes and their impact on migration decisions?

I also noticed that Figure 1 contains Chinese text in panel (d). Is this intentional? The figure should be carefully checked to ensure the presentation is consistent and rigorous.

Reviewer_R96yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ARI ranks active requests for migration using rule interpolation over memory pressure, KV size, transfer cost, imbalance, and remaining duration. It adapts rule confidence from migration feedback and greedily plans destinations until a balance threshold is reached.

Critical strengths
  • Request ranking incorporates transfer cost and remaining duration alongside memory relief, capturing cases where migrating a large request would not pay off.
  • Gains over MELL appear in all six tested settings; on 4-GPU/7B, throughput rises by 54.6% and P95 latency falls by 30.9%.
Critical weaknesses
  • The confidence update is not guaranteed to improve predictions under normalized interpolation. Empirically, prediction error remains 0.30828 with or without updates, leaving the learning benefit unestablished.
  • Adaptive stopping loses to fixed Top-k on both throughput and P95: 675.31 versus 696.05 tokens/s, and 13,081.86 versus 12,675.33 ms. Its demonstrated benefit is fewer migrations, whose value needs to be weighed against these serving losses.
  • The experimental workload is not identifiable from the paper. §4.1 names only candidate corpora and says aggregate rows do not identify arrival traces; Appendix A.7 describes what should be recorded without supplying the actual loads, trace construction, or run counts.
Questions for authors

Please see the weaknesses.

1 comment

We thank the reviewer for the careful evaluation and constructive feedback. We appreciate the opportunity to clarify our methodological choices and experimental protocol, and we address the three weaknesses below.

For weakness 1, confidence adaptation is intended to improve candidate ordering and migration decisions. Aggregate utility-prediction MAE therefore does not fully characterize its scheduling value. Our historical ablations show favorable point estimates for TPOT and end-to-end P95 latency in both models, suggesting potential scheduling benefits.

We will extend the targeted confidence-on/off ablations to additional model sizes, including Llama-2-13B, and to both four- and eight-GPU configurations. Within each model–topology setting, we will use matched request traces and paired seeds while keeping the other policy components unchanged. We will evaluate candidate-ranking quality, pre-update prediction error, and serving performance, reporting paired confidence intervals to assess whether the observed trends generalize across model sizes and GPU configurations.

For weaknees 2, we thank the reviewer for highlighting the need to evaluate migration reduction alongside serving performance. We would like to clarify the value of adaptive stopping from three complementary perspectives.

First, migration restraint with near-baseline throughput. In the comparison under discussion, adaptive stopping retains 97.0% of fixed Top-k’s throughput while reducing migration events by 25.0%, with a 3.2% increase in P95 latency. Viewed conversely, fixed Top-k uses approximately 33.3% more migration events for a 3.1% throughput gain. This represents a potentially useful operating point when controlling migration frequency is an explicit system objective.

Second, state-dependent migration volume. Adaptive stopping determines how many requests to migrate from the current imbalance, instead of prescribing a fixed migration count. Under the chosen request ranking and destination-selection rule, it stops at the first accepted prefix that satisfies the balance criterion, when such a prefix exists. This gives the migration volume an explicit operational rationale and avoids continuing that plan after its balance objective has been met.

Third, complementary latency benefits. The same comparison reports TPOT of 14.17 versus 16.02 ms, an approximately 11.55% reduction. Thus, the result combines fewer migration events and lower per-token latency with throughput and end-to-end P95 trade-offs. These metrics capture different serving objectives and should be assessed jointly.

We will sharpen the presentation around this migration-count/performance trade-off and explicitly acknowledge the P95 cost. The contribution is a state-adaptive mechanism that offers an alternative operating point; the current comparison does not establish universal superiority or lower total migration cost.

For weakness 3, we thank the reviewer for pointing out that the workload description needs to identify the concrete experimental inputs. We clarify that the workload protocol is shared across methods and ablation variants within each GPU topology: all four-GPU comparisons use workload W4, and all eight-GPU comparisons use workload W8. For a given model and seed, the same scheduled request sequence is reused across the compared methods.

Both profiles specify 64 requests per run, a concurrency cap of 16, and scheduled arrivals at ti=0.02it_i=0.02i seconds. Requests are generated from a fixed prompt template, with repetition levels sampled uniformly from {1,4,16,64}\{1,4,16,64\} and output-token limits sampled uniformly from {64,128,256,512}\{64,128,256,512\}, using a seeded generator. Temperature is zero and EOS termination is disabled. Prompt token counts are determined by the corresponding model’s frozen tokenizer and chat template. The seeds are {7,17,27,37,47}\{7,17,27,37,47\}. Scheduled arrivals and actual dispatch times are recorded separately because the concurrency cap can delay dispatch.

The prescribed main matrix contains six methods × five seeds = 30 runs per model and topology; the seven-variant ablation matrix specifies 35 runs per model and topology. We will add a workload-definition table and a result-to-trace manifest linking each reported table row to its workload ID, materialized configuration, seed, trace identifier and hash.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.