acceptodds
Under review as a conference paper at ICLR 2027

When the Second GPU Helps: Topology-Aware Mode Selection for Narrow-Interconnect LLM Serving

Abstract

Modern LLM serving assumes NVLink-class interconnects, yet most multi-GPU deployments (consumer workstations, PCIe-only cloud VMs) link cards at a few GB/s. On such a narrow link, does a second GPU help, and in which mode? Replica places a full model copy per card, roughly doubling throughput when the context fits one card; staged split partitions the layers, extending reach but moving the per-token attention state (KV cache, growing with context) over the slow link every step. We show that this decision reduces to a calibrated discrete decision rule over two measurable coefficients—portable after per-engine recalibration: the per-token transport tax τ and the replica's fixed-overhead gap ΔA. Their ratio gives a crossover context ctx* = ΔA/τ; we are explicit about identifiability: the decision is robust to ±15% coefficient error (first flip ≈31%), while the point estimate of ctx* is the fit's least-identified parameter (LOCO 120–1,281). We measure both at four link settings across three hosts (6.65/13.3/17.6/25.4 GB/s), four engines, two models, up to k = 4 no-NVLink GPUs: the split tax is sub-linear (flat through k = 3), rejecting the shared-root (k−1)² prediction on distinct-bridge hosts, while replica scaling is near-linear (×3.89). Because τ is system-controllable, the coefficients also serve as an interface: a per-token cost reduction (an fp8-KV plugin, quality-failing in the tested configuration) measurably shifts ctx* rightward; we identify where the crossover breaks (fast links, ΔA < 0). The abstraction is embodied in an automatic decision layer that gates on memory, calibrates online, and drives llama.cpp within its 32,768-token admission window while reaching 65,536-token contexts on two 12 GB consumer GPUs through its own 4-bit staged path, with no manual quantization or RoPE tuning; the reach is not exclusive—manual vLLM and ExLlamaV3 configurations also admit 65K, and are faster there.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, narrow interconnect, PCIe, consumer multi-GPU, model parallelism, KV cache placement, cost-model calibration, mode selection

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_3cnKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when additional GPUs help LLM serving over slow interconnects. It proposes a decision layer that checks memory feasibility and uses calibrated latency models to choose between single GPU execution, replication, and layer splitting. Two fitted coefficients summarize the fixed cost difference and how latency grows with context length, providing an estimated switching point between modes. Runtime measurements update the estimates. The implementation integrates with llama.cpp and includes a custom staged execution path. Evaluation covers consumer GPUs, several engines, models, context lengths, and concurrency levels. The authors report high agreement with measured mode choices and demonstrate 65K token contexts on two 12 GB GPUs, with automated configuration as the main practical benefit.

Critical strengths
  1. The paper addresses a practical problem for consumer GPU users. Slow links and limited memory make execution choices difficult, so automatic mode selection could reduce manual setup and avoid configurations that run out of memory.

  2. The evaluation spans several hosts, link speeds, engines, and workloads. These measurements provide useful evidence about when each mode works well, while matched configurations help separate mode choice from differences in engine speed.

Critical weaknesses
  1. The presentation reads like an engineering report, with dense inline formulas and many disconnected implementation details. The decision rule, calibration procedures, and fallback rules are not organized into a clear argument. Important corrections and limits are scattered across appendices, making the core contribution and its supporting evidence very difficult to assess.

  2. The central explanation of the split cost conflicts with the implementation. The abstract describes remote KV transfers at every decoding step, while Appendix 24 reports no such transfers and attributes the cost mainly to synchronization. This undermines the physical meaning of the fitted coefficient and the claimed relationship between bandwidth and the mode switching point.

Questions for authors
  1. What is the core decision rule used at runtime? Please distinguish the roles of the fitted latency model, memory checks, and empirical fallback rules in producing the reported decisions.

  2. How do you reconcile the abstract's remote KV transfer explanation with Appendix 24’s finding that KV remains local? What does the fitted cost coefficient actually measure, and what evidence supports its dependence on link bandwidth?

  3. Based on the existing measurements, what does the calibrated selector add over a simple rule with the same memory checks? Table S55 shows it matching always splitting. Please identify where the fitted model leads to a better choice.

Reviewer_pAyrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a second GPU helps on narrow, non-NVLink interconnects (consumer PCIe workstations, cloud VMs), and in which mode. The existing systems assume fast NVLink/InfiniBand fabrics, which leaves this large deployment population unserved. Before KV-placement, offloading, and parallel processing systems decide where KV lives or how it moves. None of the systems make the decision of using a second device per request adaptively. This paper proposes to reduce this choice to two measurable coefficients, the split’s per-token transport tax and the replica’s fixed-overhead gap. Their ratio gives a crossover context. A decision later combines a memory-feasibility gate, an EWMA-calibrated latency score, and a noise-band fallback to choose among single, staged-split, and replica modes. Their evaluation covers four link speeds, three hosts, multiple engines, and up to four GPU’s. The selector is correct on 21/22 grid cells and 44/47 off-grid cells, and the system reached 65K tokens on two 12GB cards.

Critical strengths

(+) The per-request adaptive GPU selection based on the measure coefficients is interesting. I was not sure that rather than scheduling or parallelizing, splitting requests between GPU could help.

Critical weaknesses

(-) The writing was quite hard to follow, and their motivation was not quantified properly. The per-layer GPU split for models has been in research for a long time; the only addition they added is per-request adaptiveness, but their gain is not completely backed up, in the range of 7% ~ 27% on 3K tokens, as current context lengths can go up to way more than this number, not quite explained if the gain is from per-request adaptiveness or just modified engineering.

(-) The paper’s core contribution is not backed up with quantifiable results properly. The main idea mostly reduces to a simple rule (Appendix 5.5): use a full copy per GPU if it fits, else split. Its only extra wins at short contexts don’t repeat (Table S53).

Questions for authors
  1. Does a static rule with your memory gate added match the calibrated selector’s decisions, and if so, what does calibration add?
  2. Table S7 says the 512/1,024 split wins were superseded, yet Appendix 5.5 still credits them to the selector; which result holds?
  3. Does entropy-aware placement beat a plain balanced split with one boundary, or does the gain come only from fewer boundaries?
Reviewer_6iZ4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LLM inference on multi-GPU machines whose GPUs communicate through narrow PCIe or host-relay links. It compares single-GPU execution, a staged layer split, and request-parallel replicas. The proposed decision layer first gates modes using predicted memory feasibility and then ranks feasible modes with a per-engine latency model. Experiments span several engines, models, link settings, and up to four no-NVLink GPUs; the paper reports long-context reach, near-linear independent-replica scaling, and repaired selector accuracy.

Critical strengths
  • The paper studies a relevant and insufficiently characterized deployment regime. PCIe-only consumer multi-GPU machines are common, and the measurements show that mode choices depend on the complete topology and execution engine rather than GPU count alone.

  • The paper is unusually explicit about negative results and validity limits. It reports that optimized mainstream engines can outperform the custom path, that the tested FP8-KV configuration fails its quality probe, that the exact fitted crossover is weakly identified, and that several selector mechanisms were repaired after falsifying measurements. The feasibility and long-context results remain useful even where the broader coefficient claim is uncertain.

Critical weaknesses
  • The evidence does not yet isolate the value of the proposed two-coefficient decision model. In many evaluated regimes, the decision appears to be determined primarily by memory feasibility and simple workload conditions rather than by the predicted crossover itself. A simpler feasibility-aware static policy is therefore a strong baseline, and the paper does not clearly demonstrate when the calibrated model produces a materially better decision. This weakens the paper's central contribution beyond the useful empirical characterization.

  • The benefit of online mode selection is not demonstrated at the serving-workload level. The evaluation largely studies individual configurations, while replica throughput scaling and split-mode latency are reported separately. A trace-driven comparison with changing context lengths and concurrency, including calibration and switching costs, is needed to show that dynamic selection improves latency, throughput, or SLO attainment over strong static policies and engine-native configurations.

Questions for authors
  1. Under the same memory-feasibility checks, how much does the calibrated decision model improve decision accuracy and end-to-end performance over a tuned static policy based only on context length and concurrency on a frozen held-out workload?

  2. What objective does the online selector optimize when replica execution improves system throughput while staged splitting affects per-request latency and memory reach, and how are these competing objectives combined when arrival rate and concurrency vary?

  3. On a trace-driven workload, how does the complete system compare with single-GPU execution, true multi-worker replication, staged splitting, and the best static policy in terms of TTFT, TPOT, throughput or SLO-goodput, after including the costs of calibration, recalibration, mode transitions, and maintaining mode-specific state?

Reviewer_pXYSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes to include the network topology characteristics when doing inference serving - helping choose between running replicas (ie data parallel) versus splitting across layers (ie pipeline parallel). The authors formulate and develop a cost model for the network to determine the optimized serving strategy.

Critical strengths
  • Network-aware Inference Serving is a valuable topic
  • This work can apply both to NV-link based and PCIe based serving systems
Critical weaknesses
  • Incremental novelty: network-aware serving is a well-understood and well-studied problem
  • Analytically modeling the network misses out on real production issues such as congestion.
Questions for authors

Dear Authors, I enjoyed reading this paper. The problem you described is very real. However, the solution proposes is fairly incremental, and does not rise above the bar for ICLR for me.

In particular, many serving systems use analytical or measured models for the underlying platform for serving decisions. For e.g., check out NVIDIA's AI configurator. Or look into works like LLMServingSim (https://arxiv.org/abs/2602.23036) or MIST (https://arxiv.org/abs/2504.09775) which integrate network simulators - which naturally enable the capability this paper is trying to solve.

From experience in this topic, I will also add that in practice, on production clusters network congestion is a big concern - and a cost model for the network only goes so far.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.