acceptodds
Under review as a conference paper at ICLR 2027

Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

Abstract

Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory. Existing offloading systems, however, typically rely on coarse layer-level or expert-level scheduling, which overlooks substantial heterogeneity among tensors within the same layer and adapts poorly to changing hardware load conditions on such devices. This paper presents ATSInfer, a hybrid CPU-GPU inference system for consumer devices that performs offloading at tensor granularity. ATSInfer combines static tensor placement with load-aware dynamic transfer, and introduces asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends. We implement ATSInfer and evaluate it on representative consumer platforms using both dense and MoE models. Compared with existing systems, ATSInfer improves prefill throughput by up to 1.94× and decode throughput by up to 3.29×, while also increasing GPU utilization and making more effective use of PCIe bandwidth. These results show that ATSInfer can substantially improve the user experience of local LLM deployment on personal consumer devices.

Primary area:
generative models
Keywords:
Machine Learning, Large Language Model, Inference, Consumer Device

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_aSxSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors proposed ATSInfer, for fine-grained offload scheduling with workload heuristics. Built on llama.cpp, it combines profiled tensor-level CPU/GPU placement with temporary GPU execution of CPU-resident weights. Its runtime coordinates asynchronous weight and activation movement, reuses temporary buffers, and updates scheduling decisions when measured execution or transfer costs change.

Critical strengths
  • Highlighting thermal throttling is a distinctive strength, alongside limited GPU memory and background contention. This motivates adapting inference schedules to changing device conditions on consumer platforms.
Critical weaknesses
  • Unsupported novelty premise: The layer/expert-versus-tensor contrast conflates storage/transfer, residency, fetch boundaries, and execution placement. FlexGen §4.2 assigns whole tensors within a layer; ZeRO-3 manages individual parameters through submodule hooks. Neither alone establishes arbitrary CPU/GPU operator placement, but the paper's blanket characterization is inadequate.

  • Missing prior art and controls: HeteGen, PatrickStar/SpongeBob, SpeedLoader, ZeRO-Offload, and ZeRO-Infinity provide relevant scheduling, prefetching, or overlap mechanisms, although their workloads and objectives differ. The paper does not establish incremental value over these capabilities, tuned llama.cpp tensor placement, and simple greedy scheduling. Adding citations alone would not resolve this.

  • Algorithm inconsistency: Algorithm 2, step 17 uses the predecessor's default backend and a switch condition inconsistent with Eq. (4); its recurrence also fails to retain consecutive promotions. These concerns apply to the printed pseudocode; the implementation may differ.

  • Unvalidated thermal adaptation: Thermal throttling motivates the design in the main text (§3.1/Figure 2), yet the follow-up is confined to a qualitative attribution of Appendix B's generation-length curves (Figure 10). No temperature/clock/power evidence or controlled online-versus-frozen thermal experiment supports that attribution; context growth is a confound. Main-text Figure 9a evaluates external load, leaving this distinctive motivation without a dedicated thermal evaluation.

Questions for authors
  • How do you define granularity for storage/transfer, residency, fetch boundaries, and execution placement? Which controlled comparisons establish gains over the prior mechanisms above, tuned tensor placement, and a saved-time/byte greedy policy? How does ATSInfer differ from DynamicInfer, whose Appendix E.5 leaves external contention to future work?

  • Which runs produce the reported 3.29x and 3.15x maxima? Can you provide Figure 8a's numerical data, checkpoint/quantization revisions, baseline commands and commits, and peak-memory accounting?

  • What exactly is timed in Figure 1, including bytes accessed, representation, cache state, threading, and synchronization boundaries? How do approximately 59 MiB in 0.19 ms reconcile with the i7-11800H's 51.2 GB/s DRAM limit and 24 MB cache, given the implied 326 GB/s if the full tensor is read?

  • Can you provide thermal telemetry and an online-versus-frozen comparison controlling for context length? For Figure 9a, what external workloads and load definitions were used, were re-estimation/reconfiguration overheads included, and what were the trigger frequency, run-to-run variation, and execution-correctness checks?

2 comments

Thank you for the detailed review. We have re-examined the measurements, scheduling formulation, and evidence. Below, we clarify our contribution, correct reporting errors, and delimit the supported conclusions.

1. Granularity, related work, and controlled comparisons.

We agree that storage/transfer granularity, residency, fetch boundaries, and execution placement are distinct aspects and should not be conflated. Our point is not that layer/expert-level and tensor-level scheduling represent fundamentally different execution models. A layer or expert can be viewed as a collection of tensors and their associated operators. The same computation and data-transfer mechanisms apply at both granularities; changing the scheduling unit does not require a different execution model. Note that operator decomposition depends on the underlying operator library: ATSInfer uses ggml, whose operator boundaries may differ slightly from PyTorch. Therefore, our discussion refers to the actual ggml execution graph. ATSInfer coordinates tensor placement and movement with the execution of their associated operators, making tensor-level scheduling a joint consideration of data and computation.

ATSInfer exhibits characteristics of both GPU-centric and hybrid execution frameworks, combining static weight placement and computation-result transfers with dynamic data transfers during execution. Although this makes the interactions among storage, computation, and communication more complex, its scheduling granularity remains well-defined. Indeed, the execution model would still work if individual tensors and their associated operators were grouped and scheduled at the layer level. Tensor-level scheduling instead provides finer opportunities to exploit tensor heterogeneity and optimize placement and execution decisions.

DynamicInfer also studies dynamic inference scheduling. Unlike approaches that predict additional neuron sparsity, ATSInfer executes dense operators or model-selected experts and adapts weight residency and temporary GPU execution to measured resource conditions.

Our base+TP baseline uses exactly the same tensor-placement results as ATSInfer on llama.cpp. Thus, their performance difference directly measures the benefit of asynchronous coordination and dynamic promotion under identical static placement. It does not replace direct comparisons against all cited systems or a saved-time/byte greedy scheduler. Relative to an independent saved-time/byte ranking, our formulation explicitly accounts for activation-switching costs and exposed weight-transfer costs; this is a design distinction, rather than an experimentally established advantage over that greedy baseline.

2. Algorithm 2 and Equation (4). We agree that Algorithm 2 as printed does not correctly distinguish default residency from runtime placement. In Equation (4), the activation-switching cost depends on the runtime backends:

ci,1rbi−1≠rbi.c_i\\,\mathbf{1}\\{rb_{i-1}\neq rb_i\\}.

For a promoted tensor, rbi=GPUrb_i=\mathrm{GPU}, so this term becomes

ci,1rbi−1=CPU,c_i\\,\mathbf{1}\\{rb_{i-1}=\mathrm{CPU}\\},

rather than the condition involving the predecessor's default backend in the printed step 17. In particular, two consecutive GPU executions incur no activation-switching cost at their boundary.

With a correctly formulated DP recurrence that accounts for runtime placement in each transition, recording predecessor states is sufficient to fully reconstruct the decision chain of the optimal solution, including consecutive promotions.

3. Performance figures, configurations, and memory. The 3.29× and 3.15× maxima came from an earlier revision and do not correspond to the final experiments. We withdraw both values and base our comparison on the per-configuration results in Figure 8a.

The llama.cpp baseline uses commit 550d684bd13132d4af744cee64c1a3acb0ceaf09. The RTX 3060 decode command is:

./build-release/bin/llama-bench -m "$MODEL_PATH" -o json \
    -r "$round_repeat" --delay "$round_delay" \
    -b 512 -ub 512 -t 4 -ngl "$NGL" -p 0 \
    -d 2048 -n "$decode_len" \
    > "$output_path"

The model configurations are:

ModelWeight representation-ngl
Qwen3-30B-A3BINT412
Qwen3-14BINT420
GLM-Z1-9BFP169
GPT-OSS-20BNative MXFP49

The main decode evaluation uses a 2048-token prompt and 128 generated tokens. CPU threading is set to four, and batch/micro-batch sizes are both 512. Repetitions and inter-run delays are controlled by the script variables shown above.

Base+TP additionally applies tensor-placement overrides matching ATSInfer's static placement. For example, GLM-Z1-9B places 87 tensors in CPU memory and the remaining tensors in GPU memory.

On the RTX 3060, reported peak GPU-memory usage was approximately 5.1 GB for all compared configurations; approximately 5.3 GB was available with the desktop running.

4. Figure 1 measurement correction. We identified an error in the aggregation of the Down-operator measurements and reran that experiment. The corrected execution times are approximately 1.62 ms on the CPU and 0.47 ms on the GPU. The original CPU value of 0.19 ms is incorrect, and we withdraw the interpretation based on that exceptionally low latency.

5. External contention and thermal effects. Figure 9a evaluates adaptation to synthetic compute and data-transfer contention. Workload intensity is controlled through CPU worker-core count and transferred data volume. End-to-end TPOT includes runtime overhead. The separately measured reconfiguration cost is approximately 16 ms per event, comprising 9 ms for scheduling and 7 ms for graph rebuilding.

The configured re-scheduling conditions are a measurement deviation of at least 15% and a minimum interval of five recent TPOTs since the previous event. These describe the trigger policy, rather than the observed event frequency.

We agree that the existing evaluation does not isolate thermal throttling. Temperature, clock, power, and throttling telemetry, together with a matched-context online-versus-frozen comparison, are absent. Generation-length curves also confound thermal effects with context growth.

We thank the reviewer for identifying these issues. These corrections and clarifications will make our claims and experimental evidence more precise.

Reviewer_nejSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Hello, authors! Thank you for your work, local LLM inference on consumer devices is an extremely important and practical topic nowadays. Below is my understanding of the paper. Essentially this is a deeply systems paper, so the implementation is especially important here, but I could not find any code or supplementary materials for it.

The setting is LLM inference on a laptop when the weights do not fit in VRAM, so part of the model stays in CPU memory. Nevertheless the main idea: instead of splitting the model by layers (or by experts in MoE), as existing hybrid systems like llama.cpp and KTransformers do according to the paper, you split it by tensors (Q, K, V, O, gate, up, down separately), because tensors of a similar size can have very different CPU and GPU cost, and you adapt the split on the fly to background load and thermal throttling.

To do this, ATSInfer (built on top of llama.cpp) has three parts: (1) static tensor placement at load time, per-tensor CPU/GPU latencies are profiled and a knapsack-style DP chooses which tensors stay in VRAM, with a penalty for switching backend between neighbour operators; (2) load-aware dynamic transfer, at runtime a second DP picks CPU-resident tensors to copy temporarily to GPU so the copy is hidden behind computation, and it is re-run when measured times deviate by 15%; (3) asynchronous CPU-GPU coordination, weights go through the copy engine on a separate stream, activations through SM copy kernels.

Evaluation is on 8 models (dense and MoE, from GLM-Z1-9B to GPT-OSS-120B) on a RTX 3060 laptop and a RTX 4090 desktop, against llama.cpp, vLLM and KTransformers, with up to 1.94x prefill and 3.29x decode throughput reported over llama.cpp.

Critical strengths
  • the idea itself is simple, but the problem is practical: on a laptop the CPU/GPU/PCIe cost changes with background load and thermal throttling, and a schedule chosen at start becomes outdated. Fig. 2 and Fig. 9a show this clearly, and the rate-limited re-scheduling is a reasonable design. this angle is not covered much in prior hybrid inference work
  • broad evaluation on recent dense and MoE models (up to Qwen3.5-122B and GPT-OSS-120B) on two real consumer platforms, with Nsight traces and an ablation of the three components
Critical weaknesses
  • it feels like a limited contribution, mostly engineering on top of llama.cpp. splitting the model finer than layers is not new: llama.cpp itself already allows per-tensor placement (--override-tensor), FlexGen places each weight tensor inside a layer separately, HeteGen profiles modules and splits them between CPU and GPU, PowerInfer goes down to neurons, and choosing a separate CPU-GPU policy for prefill and decode is done in FlexInfer (Na et al.), which is cited only in passing. the DP also does not seem to add much, in the ablation tensor placement barely changes decode speed. so the new part is mostly the adaptation to background load, and the fair baseline would be a tuned llama.cpp with --override-tensor
  • several key numbers do not match the figures or the hardware. the 3.29x decode speedup from the abstract I could not find in Fig. 8a, on the laptop the gains there look closer to 2x at most. the 40 ms TPOT for Qwen3-14B in Sec. 3.5 does not match the throughput in Fig. 8 (closer to 110 ms). the per-step transfer volumes in Sec. 4.2 would need more bandwidth than PCIe Gen4 can give, while Fig. 9b shows much less. and in Fig. 1, which is the main motivation, the Down tensor is processed faster than DDR4 or the 3060 memory can even read it, so the "heterogeneity" between tensors may come from how it was measured
  • no code and no baseline configs (llama.cpp version and flags, vLLM / KTransformers settings). for a systems paper the implementation is basically the contribution, and with the numbers above not adding up, the results can't be verified
Questions for authors

clarification:

  • where does the 3.29x decode speedup (laptop, vs llama.cpp) come from? in Fig. 8a I could not find it. could you add a table with exact numbers and variance over several runs?
  • how was llama.cpp configured (version, -ngl, was --override-tensor used, was op offload on)? and how was KTransformers configured on a CPU without AMX?
  • do you plan to release the code?

suggestions:

  • compare with a tuned llama.cpp --override-tensor baseline, and position the work against FlexGen, HeteGen and FlexInfer (Na et al.)
  • show how much tensor granularity itself gives vs layer granularity with the same async coordination and dynamic transfer, and compare the DP with a simple greedy placement
  • check that outputs are the same as llama.cpp (e.g. greedy tokens or perplexity)
  • DALI (Zhu et al., 2026) also optimizes CPU/GPU placement for MoE on local PCs and is cited, could you compare with it or HybriMoE, since most of the decode gains are on MoE models?
1 comment

We thank the reviewer for the careful reading and constructive feedback. We appreciate the recognition of the practical motivation and the system design. We address the main concerns below.

1. Contribution and relationship to existing tensor-placement mechanisms

We agree that tensor-level placement itself is not new. In particular, ggml, the backend underlying llama.cpp, already provides flexible tensor memory placement through --override-tensor. ATSInfer builds on this capability rather than claiming to introduce it. The key distinction is that --override-tensor provides a low-level mechanism for manually specifying tensor placement, whereas ATSInfer automates placement decisions through profiling and optimization, and further coordinates runtime tensor promotion, asynchronous transfers, and computation.

Our base+TP ablation uses exactly the same static tensor-placement result as ATSInfer, but disables dynamic promotion. This isolates the contribution of runtime promotion from static tensor placement. The results therefore also show that static tensor placement alone does not explain the full performance gains.

2. Correction of reported performance numbers and TPOT

The reviewer is correct that the 3.29× decode speedup in the abstract does not match Fig. 8. This number, together with the 3.12× figure appearing in the manuscript, was inadvertently retained from an earlier version during revision. These are editing errors. The performance values shown in the figures are the intended results, with decode improvements of approximately 2×. We will correct the outdated numbers throughout the manuscript to ensure consistency.

Regarding the Qwen3-14B TPOT, the approximately 40+ ms value in Sec. 3.5 is consistent with the throughput in Fig. 8: a throughput of 20+ tokens/s corresponds to approximately 50 ms per token. We will clarify this relationship in the revised manuscript.

3. Fairness and reproducibility of the llama.cpp baseline

Our benchmark scripts explicitly configure the GPU layer count, context length (2048), batch and micro-batch sizes (512), output lengths (32–2048 tokens), and CPU thread count (4 on the laptop and 8 on the desktop). The 128-token setting uses three repetitions. The --pipo option selects the ATSInfer tensor-placement configuration, while -do controls whether the decode promotion list is enabled. Operator offloading is enabled by default.

For the llama.cpp baseline, we use layer-level CPU/GPU placement, with the GPU layer count selected to match ATSInfer's peak GPU memory consumption. Importantly, this budget includes ATSInfer's temporary promotion buffers, rather than accounting only for resident model weights. In the base+TP ablation, the static tensor placement is identical to ATSInfer's, providing a direct comparison with tensor-level placement while keeping the placement result fixed. We will make the baseline settings and the distinction between layer-level and tensor-level placement explicit in the paper.

4. KTransformers and CPU instruction-set support

KTransformers can run on CPUs without AMX support. Its CPU execution path supports AVX-family instructions and does not require AMX as a prerequisite.

5. Comparison with DALI and HybriMoE

We selected widely used open-source inference frameworks with broader model and hardware compatibility for our end-to-end comparisons. We did not directly benchmark DALI or HybriMoE. Importantly, ATSInfer applies the same optimizations to both dense and MoE models and does not yet incorporate MoE-specific optimization mechanisms. Expert-residency and swapping strategies are therefore complementary to ATSInfer and could potentially be combined with its tensor-granularity scheduling within experts for further speedups. We will clarify this scope and discuss the relationship to these systems.

6. Implementation availability

Although we have not released the official implementation, community developers have already independently implemented the algorithm described in our paper, reporting substantial speedups. We believe this provides additional evidence that the proposed approach is practically implementable. We acknowledge that releasing the official code and complete benchmark configurations would further improve reproducibility.

7. Figure 1 Correction

We identified an error in the original data aggregation for the Down operator, reran the experiments, and replaced the incorrect values. The corrected execution times are approximately 1.62 ms on CPU and 0.47 ms on GPU. We will update Figure 1 and its discussion accordingly and apologize for the inaccurate original data.

Reviewer_ZXipOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ATSInfer, a tensor scheduling system for hybrid CPU-GPU inference of large language models on consumer devices with limited GPU memory.

Critical strengths

1. Practical and well-motivated problem.

The paper addresses the important problem of memory-constrained LLM inference on consumer devices. The observed tensor-level performance heterogeneity provides a reasonable motivation for fine-grained scheduling.

2. Comprehensive system design.

ATSInfer integrates static tensor placement, load-aware dynamic transfer, and asynchronous CPU-GPU coordination into a unified framework. The design considers both computation and communication costs and supports dense and MoE models.

3. Promising performance improvements.

The evaluation covers eight models across two consumer hardware platforms. ATSInfer achieves up to 1.94× prefill and 3.29× decode speedups over llama.cpp, with additional ablation and hardware-utilization results supporting its effectiveness.

Critical weaknesses

1. Insufficient evaluation of thermal throttling.

Thermal throttling is highlighted as an important motivation for load-aware scheduling. However, the experiments do not systematically evaluate ATSInfer's adaptability under thermal constraints. Although Figure 10 shows latency changes during sustained generation, it does not directly demonstrate whether the proposed scheduling strategy effectively mitigates thermal throttling.

2. Limited comparison with closely related methods.

The evaluation mainly compares ATSInfer with llama.cpp, vLLM, and KTransformers on selected configurations. However, other relevant hybrid CPU-GPU inference systems, such as FlexInfer, are not experimentally evaluated. This makes it difficult to assess ATSInfer's advantages over more closely related scheduling approaches.

3. Insufficient analysis of scheduling overhead and robustness.

The dynamic scheduler uses fixed re-scheduling thresholds, but their sensitivity and generalizability are not evaluated. More comprehensive measurements of re-scheduling frequency, latency stability, and runtime overhead are needed.

Questions for authors

1. Can the authors provide controlled experiments to evaluate ATSInfer's effectiveness under thermal throttling, beyond the sustained-generation results in Figure 10?

2. Can the authors compare ATSInfer with more closely related hybrid CPU-GPU inference systems, such as FlexInfer and Fiddler, or explain why these comparisons are infeasible?

3. How sensitive is ATSInfer's performance to the re-scheduling threshold and interval? Can the authors quantify the amortized scheduling overhead and its impact on latency stability?

2 comments

Thank you for your questions. We clarify our evaluation methodology and implementation as follows.

1. Thermal behavior under sustained generation.

We agree that Figure 10 does not make the relationship between thermal conditions and latency variations sufficiently explicit. The original figure focuses on latency evolution during sustained generation but does not directly show the corresponding GPU thermal state. In the revised manuscript, we will supplement these results with GPU temperature measurements, enabling readers to correlate thermal conditions with the observed latency changes. Our runtime instrumentation also records SM clock, memory clock, utilization, and per-token timestamps, providing additional information to characterize device behavior under sustained workloads.

2. Comparison with related hybrid inference systems.

We selected llama.cpp, vLLM, and KTransformers as baselines because they provide practical and reproducible inference implementations with broad model and hardware support relevant to consumer devices. Our evaluation prioritizes systems that can be executed under compatible hardware, model, and inference configurations, ensuring meaningful and reproducible comparisons.

Research prototypes such as FlexInfer and Fiddler target different execution strategies and may have different hardware and model compatibility requirements. Directly comparing reported results across different hardware platforms, model architectures, or execution settings would not provide a reliable performance ranking. We therefore focus on reproducible comparisons using representative inference frameworks rather than combining results obtained under incompatible experimental conditions.

At the mechanism level, FlexInfer employs phase-specific CPU-GPU execution policies, whereas ATSInfer uses tensor-level weight residency patterns and separate promotion lists for prefill and decode. Fiddler uses CPU computation to avoid costly expert-weight transfers. ATSInfer supports both CPU execution and temporary GPU promotion of host-resident weights, with MoE transfers driven by the expert IDs selected during routing. These differences highlight the distinct scheduling strategies explored by the systems.

3. Runtime overhead and scheduling frequency.

We measured the two major components of reconfiguration overhead. Graph rebuilding takes approximately 7 ms, while the dynamic programming scheduler takes approximately 9 ms, resulting in approximately 16 ms in total. This overhead is substantially smaller than the TPOT in our evaluated settings, by roughly an order of magnitude.

Furthermore, reconfiguration is not performed for every generated token. ATSInfer uses a latency-deviation threshold and a minimum scheduling interval to avoid repeated decisions caused by minor fluctuations. The resulting overhead is therefore incurred only when the scheduling conditions are met and is amortized across subsequent token generations.generation.

Thank you for the detailed response. I appreciate the clarifications regarding baseline selection and the planned additional visualizations. However, my main concerns remain partially unresolved. In particular, adding thermal measurements does not directly demonstrate the effectiveness of load-aware scheduling under thermal throttling. Therefore, while the clarifications are helpful, I will maintain my original rating.

Reviewer_6DTLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents ATSINFER, a tensor-grained CPU-GPU hybrid scheduler for LLM inference on consumer devices. Prior hybrid ffloading uses coarse layer/expert placement and cannot adapt to dynamic hardware load. ATSINFER combines a static knapsack-based tensor lacement pass and a lightweight runtime dynamic promotion mechanism, together with a dual-CUDA-stream asynchronous transfer pipeline. It is built on llama.cpp and supports both dense and MoE models. Evaluations on RTX3060/4090 show notable speedups over llama.cpp, vLLM and KTransformers. Ablations confirm the effectiveness of each component.

Critical strengths
  1. Novel scheduling granularity: tensor-level instead of layer/expert-level. Existing hybrid offloading mostly uses layer-level placement, and prior MoE work only reaches expert granularity. The authors observe that tensors within one Transformer layer have vastly different per-byte GPU acceleration gains. Instead of moving an entire layer together, ATSINFER treats tensors as the basic scheduling unit. It models activation transfer penalties when operators cross devices and accounts for operator dependencies.
  2. Two-stage static-dynamic scheduling. Static placement runs at initialization to select tensors permanently kept on GPU. Runtime dynamic promotion moves selected CPU tensors to GPU temporarily according to hardware load. It reuses temporary GPU buffers and preserves the static resident set, avoiding repeated eviction and reload of heavy weights. A throttling rule is added to prevent frequent rescheduling triggered by minor load fluctuations.
  3. Asynchronous pipeline with separate CUDA streams. Weight transfer and activation transfer are decoupled onto distinct streams. This prevents small activation copies from being blocked by large weight transfers, improves PCIe utilization, and overlaps computation with data movement.
Critical weaknesses
  1. The paper lacks a concrete engineering breakdown of its modules at the tensor, operator and hardware pipeline level. Core scheduling components are presented as black boxes, making it hard to understand how tensor placement interacts with the LLM execution graph.
Questions for authors
  1. Please elaborate on how different types of tensors are partitioned and scheduled across the memory hierarchies of different devices.
  2. How does this work perform compared with PowerInfer?
2 comments

Thank you for asking for the tensor, operator, and hardware-level details. The implementation makes the following distinctions.

1. Tensor partitioning and memory placement. The configuration contains two different objects: host-resident weight patterns and weights selected for temporary GPU execution. Matching resident patterns are loaded into CUDA host buffers; other weights use GPU buffers. Promotion retains the host copy and substitutes a temporary GPU tensor as the consuming operator's input. This is tensor-level backend assignment, rather than splitting every tensor internally into CPU/GPU shards.

The placement tool handles dense and MoE weights separately. For dense models, the embedding table is forced to the CPU and the remaining eligible weights enter the memory-budgeted placement solver. Associated biases are bound to their matrix weights. For MoE, the solver partitions expert-bank and non-expert entries. If non-expert weights fit, they are assigned to the GPU and the residual budget is optimized over expert entries. Otherwise, expert entries remain on the CPU and the solver allocates the budget among non-expert entries.

Static expert-bank capacity and runtime transfer payload are different. The runtime reads the selected expert IDs after routing, marks the experts used by the current input, and groups adjacent selected experts into contiguous transfers. It does not transfer an entire expert bank merely because that bank occupies a host-resident tensor.

2. Mapping placement to the graph and execution pipeline. Graph construction selects the prefill or decode promotion list and replaces matching layer-weight references with temporary GPU tensors. The runtime records source/destination mappings and splits the graph at backend changes and dynamic-weight consumers. The temporary-tensor map allows weights mapped to the same reusable tensor to share GPU storage across their scheduled uses.

Weight copies run on a dedicated nonblocking transfer stream. Before overwriting a temporary tensor, that stream waits for an event recorded on the compute stream. The consuming compute stream then waits for the weight-copy completion event. For MoE, scheduling the copy also waits for the routing/top-k result. These dependencies connect the storage decision to the actual consuming operators.

Activation movement follows a separate path. With mapped host memory, SM copy kernels run on the compute stream. If the host pointer cannot be mapped, the implementation falls back to cudaMemcpyAsync. Host-to-device mappings are cached to avoid repeated pointer queries. GPU-to-CPU transitions synchronize before CPU consumption. Both transfer paths still share PCIe bandwidth.

3. Comparison with PowerInfer. PowerInfer combines placement with predictor-based sparse execution. Predictions can omit active neurons and introduce an approximation, although its reported accuracy effects are small in its evaluated settings. ATSInfer's dense path does not add predictive neuron skipping. Throughput alone therefore does not isolate scheduling unless the model and quality target are matched. PowerInfer, Section 8.5.

I thank the authors for the response, which addresses the main points I raised. On presentation, I would suggest one addition: Fig. 3 and Fig. 4 are well suited to explaining the system structure and storage regions, but tensor-granularity placement is the paper's central claim, and the current schematic does not let a reader verify that tensor-level movement actually happens as described. Fig. 5 already shows a stream-level timeline but carries no tensor identity. I would suggest a supplementary figure that follows a single decode step, with concrete tensors as its subjects: layer and operator, tensor role (resident weight, promoted temporary weight, or activation), bytes and measured latency per transfer, temporary-buffer reuse and event dependencies, and the difference between the prefill and decode promotion lists. If space is tight, an appendix figure referenced from the main text is sufficient.

Reviewer_MJ2BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to optimize LLM inference efficiency in consumer devices, where CPU offloading of model weights is necessary (GPU memory is not enough to hold full model weights). The paper observes that existing hybrid CPU-GPU systems schedule model weights at coarse granularity (layer/expert), ignoring the tensor heterogeneity in the same layer (tensor size and operator computation cost). The paper proposes ATSInfer, which schedules weight placements at tensor granularity via following techniques: (1) asynchronous CPU-GPU coordination that overlaps weight/activation transfers and computation. (2) Static placement algorithm to maintain most valuable tensors in GPU. (3) Load-aware dynamic transfer that estimates overlap window to wisely promote CPU-resident tensors to GPU for quicker GPU execution. ATSInfer is implemented on llama.cpp and outperforms baselines in prefill throughput and decode throughput in testbed experiments.

Critical strengths

A well-implemented work. (1) The paper proposes a nice formulation of hybrid CPU-GPU execution pipeline to utilize tensor-granularity optimization opportunities. (2) The paper implements a real system that supports both dense and MOE models and achieves significant performance improvement.

Critical weaknesses
  • The key innovation of this paper builds on an observation, which is not clear and thorough. The foundation of this paper is thus weak.
  • The solution is not clearly organized; it is hard to tell how different parts are correlated. It is also unclear what are the sub-problems, the key challenges, the potential solution space and the key design insights in each solution part.
  • The methodology proposed by the paper relies heavily on profiled data (e.g. execution times in CPU/GPU) to guide scheduling. However, the paper doesn't clearly explain how to profile these data and how to utilize the profiled data for estimation. In Sec 3.4 and Sec 3.5, the paper relies on profiled per-operator CPU and GPU latency. But the paper doesn't tell the cost of profiling (e.g. repetition count of profile executions and input shapes used in profile experiments)
Questions for authors

Clarification requests:

  • How do you profile the following data at initialization?
    • Per-operator CPU/GPU latencies
    • PCIe bandwidth
  • How are operator latencies modeled for varying prompt token lengths?
  • How do you model latencies of operators with input shapes that have never been profiled?
  • What information does a measurement snapshot contain in Algorithm 3? How is deviation computed?
1 comment

Thank you for your questions about the motivation, component interfaces, and profiling procedure. We clarify these points below.

1. The observation underlying tensor-level scheduling.

Placement is based on measured execution benefit, not tensor size alone. For a tensor of size sis_i, its residency benefit is tiCPU−tiGPUt_i^{CPU}-t_i^{GPU}. The static solver also tracks backend state and estimates activation-transfer costs when switching backends. Thus, independently ranking tensors by benefit per byte may differ from selecting assignments that account for communication boundaries. This motivates the tensor-level placement formulation.

2. How the components fit together.

The implementation separates resident placement from temporary GPU execution. The profiler produces operator costs, and the placement tool generates host-resident weight patterns and a promotion list. At runtime, these assignments determine operator inputs, graph splits, temporary buffers, and transfers. Prefill and decode use separate promotion lists. Resident weights remain in their assigned storage, while selected host weights can execute via temporary GPU copies.

3. CPU/GPU latency profiling and repetition counts.

The profiling utility builds a single-token graph (nctx=nbatch=1n_{ctx}=n_{batch}=1), identifies model weights and their consuming operators, and reconstructs each consumer using its output shape/type, source shapes/types/strides, and operator parameters. Inputs, including valid index tensors, are initialized before timing.

Each case has one untimed execution. MUL_MAT uses five timed executions on either backend. Other GPU cases also use five; other CPU cases use two executions per call across two timed calls, totaling four. Mean latency is elapsed wall-clock time divided by execution count. The graph-compute wrapper synchronizes the backend before returning, so GPU timing includes execution completion, not just launch submission. Allocation and input initialization are excluded. CPU threads are configurable via -t (default: four).

Identical operator signatures are cached within the profiling process. Signatures include source layouts and operator parameters, allowing equivalent operations to reuse measurements. JSON output records weight size, CPU/GPU latency, benefit per byte, consuming node, and transfer-cost information.

4. PCIe bandwidth profiling.

The H2D routine transfers a 128 MiB host byte buffer through the backend tensor-set interface. It performs five warm-up and five timed copies, synchronizing after each. Bandwidth is payload size divided by mean transfer time. Activation-transfer cost is estimated as the size of other operator inputs divided by this measured H2D rate. This measures effective H2D bandwidth, not theoretical interconnect peak.

5. Prompt lengths and unprofiled shapes.

The profiler uses exact operator signatures for caching, not a prompt-length-dependent latency model. Prefill and decode use separate promotion configurations. Prefill uses a fixed default configuration across prompt lengths: weight sizes remain unchanged, while prefill requires substantially more computation than decode, making CPU execution a greater bottleneck. Thus, scheduling and manual tuning prioritize minimizing CPU computation rather than repeatedly optimizing placement for each prompt length. Decode follows the single-token path with its corresponding promotion configuration. We do not claim single-token profiling predicts exact latencies for arbitrary prefill shapes, or use interpolation/extrapolation for unseen shapes.

6. Measurement snapshot and deviation in Algorithm 3.

The snapshot contains CPU/GPU operator latency estimates, activation-transfer costs, and weight-transfer costs: m=(tGPU,tCPU,c,w)m=(t^{GPU},t^{CPU},c,w). Deviation refers specifically to changes in measured operator latency relative to the previous reference, not differences across all snapshot fields. When latency deviation reaches 15% and at least five recent TPOTs have elapsed, ATSInfer considers rebuilding the scheduling plan. The threshold detects meaningful execution-cost changes, while the interval prevents excessive reconfiguration.

Comments on the paper

1 comment

We thank the reviewer for this helpful suggestion. We will add a supplementary figure tracing concrete tensors through a decode step, including their layer/operator associations, roles, transfer sizes and measured latencies, temporary-buffer reuse, and event dependencies. We will also illustrate the differences between the prefill and decode promotion lists. This figure will complement Figs. 3–5 and make tensor-level placement and movement directly verifiable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.