- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors proposed ATSInfer, for fine-grained offload scheduling with workload heuristics. Built on llama.cpp, it combines profiled tensor-level CPU/GPU placement with temporary GPU execution of CPU-resident weights. Its runtime coordinates asynchronous weight and activation movement, reuses temporary buffers, and updates scheduling decisions when measured execution or transfer costs change.
Critical strengths
- Highlighting thermal throttling is a distinctive strength, alongside limited GPU memory and background contention. This motivates adapting inference schedules to changing device conditions on consumer platforms.
Critical weaknesses
-
Unsupported novelty premise: The layer/expert-versus-tensor contrast conflates storage/transfer, residency, fetch boundaries, and execution placement. FlexGen §4.2 assigns whole tensors within a layer; ZeRO-3 manages individual parameters through submodule hooks. Neither alone establishes arbitrary CPU/GPU operator placement, but the paper's blanket characterization is inadequate.
-
Missing prior art and controls: HeteGen, PatrickStar/SpongeBob, SpeedLoader, ZeRO-Offload, and ZeRO-Infinity provide relevant scheduling, prefetching, or overlap mechanisms, although their workloads and objectives differ. The paper does not establish incremental value over these capabilities, tuned llama.cpp tensor placement, and simple greedy scheduling. Adding citations alone would not resolve this.
-
Algorithm inconsistency: Algorithm 2, step 17 uses the predecessor's default backend and a switch condition inconsistent with Eq. (4); its recurrence also fails to retain consecutive promotions. These concerns apply to the printed pseudocode; the implementation may differ.
-
Unvalidated thermal adaptation: Thermal throttling motivates the design in the main text (§3.1/Figure 2), yet the follow-up is confined to a qualitative attribution of Appendix B's generation-length curves (Figure 10). No temperature/clock/power evidence or controlled online-versus-frozen thermal experiment supports that attribution; context growth is a confound. Main-text Figure 9a evaluates external load, leaving this distinctive motivation without a dedicated thermal evaluation.
Questions for authors
-
How do you define granularity for storage/transfer, residency, fetch boundaries, and execution placement? Which controlled comparisons establish gains over the prior mechanisms above, tuned tensor placement, and a saved-time/byte greedy policy? How does ATSInfer differ from DynamicInfer, whose Appendix E.5 leaves external contention to future work?
-
Which runs produce the reported 3.29x and 3.15x maxima? Can you provide Figure 8a's numerical data, checkpoint/quantization revisions, baseline commands and commits, and peak-memory accounting?
-
What exactly is timed in Figure 1, including bytes accessed, representation, cache state, threading, and synchronization boundaries? How do approximately 59 MiB in 0.19 ms reconcile with the i7-11800H's 51.2 GB/s DRAM limit and 24 MB cache, given the implied 326 GB/s if the full tensor is read?
-
Can you provide thermal telemetry and an online-versus-frozen comparison controlling for context length? For Figure 9a, what external workloads and load definitions were used, were re-estimation/reconfiguration overheads included, and what were the trigger frequency, run-to-run variation, and execution-correctness checks?
2 comments
3. Performance figures, configurations, and memory. The 3.29× and 3.15× maxima came from an earlier revision and do not correspond to the final experiments. We withdraw both values and base our comparison on the per-configuration results in Figure 8a.
The llama.cpp baseline uses commit 550d684bd13132d4af744cee64c1a3acb0ceaf09. The RTX 3060 decode command is:
./build-release/bin/llama-bench -m "$MODEL_PATH" -o json \
-r "$round_repeat" --delay "$round_delay" \
-b 512 -ub 512 -t 4 -ngl "$NGL" -p 0 \
-d 2048 -n "$decode_len" \
> "$output_path"
The model configurations are:
| Model | Weight representation | -ngl |
|---|---|---|
| Qwen3-30B-A3B | INT4 | 12 |
| Qwen3-14B | INT4 | 20 |
| GLM-Z1-9B | FP16 | 9 |
| GPT-OSS-20B | Native MXFP4 | 9 |
The main decode evaluation uses a 2048-token prompt and 128 generated tokens. CPU threading is set to four, and batch/micro-batch sizes are both 512. Repetitions and inter-run delays are controlled by the script variables shown above.
Base+TP additionally applies tensor-placement overrides matching ATSInfer's static placement. For example, GLM-Z1-9B places 87 tensors in CPU memory and the remaining tensors in GPU memory.
On the RTX 3060, reported peak GPU-memory usage was approximately 5.1 GB for all compared configurations; approximately 5.3 GB was available with the desktop running.
4. Figure 1 measurement correction. We identified an error in the aggregation of the Down-operator measurements and reran that experiment. The corrected execution times are approximately 1.62 ms on the CPU and 0.47 ms on the GPU. The original CPU value of 0.19 ms is incorrect, and we withdraw the interpretation based on that exceptionally low latency.
5. External contention and thermal effects. Figure 9a evaluates adaptation to synthetic compute and data-transfer contention. Workload intensity is controlled through CPU worker-core count and transferred data volume. End-to-end TPOT includes runtime overhead. The separately measured reconfiguration cost is approximately 16 ms per event, comprising 9 ms for scheduling and 7 ms for graph rebuilding.
The configured re-scheduling conditions are a measurement deviation of at least 15% and a minimum interval of five recent TPOTs since the previous event. These describe the trigger policy, rather than the observed event frequency.
We agree that the existing evaluation does not isolate thermal throttling. Temperature, clock, power, and throttling telemetry, together with a matched-context online-versus-frozen comparison, are absent. Generation-length curves also confound thermal effects with context growth.
We thank the reviewer for identifying these issues. These corrections and clarifications will make our claims and experimental evidence more precise.
Thank you for the detailed review. We have re-examined the measurements, scheduling formulation, and evidence. Below, we clarify our contribution, correct reporting errors, and delimit the supported conclusions.
1. Granularity, related work, and controlled comparisons.
We agree that storage/transfer granularity, residency, fetch boundaries, and execution placement are distinct aspects and should not be conflated. Our point is not that layer/expert-level and tensor-level scheduling represent fundamentally different execution models. A layer or expert can be viewed as a collection of tensors and their associated operators. The same computation and data-transfer mechanisms apply at both granularities; changing the scheduling unit does not require a different execution model. Note that operator decomposition depends on the underlying operator library: ATSInfer uses ggml, whose operator boundaries may differ slightly from PyTorch. Therefore, our discussion refers to the actual ggml execution graph. ATSInfer coordinates tensor placement and movement with the execution of their associated operators, making tensor-level scheduling a joint consideration of data and computation.
ATSInfer exhibits characteristics of both GPU-centric and hybrid execution frameworks, combining static weight placement and computation-result transfers with dynamic data transfers during execution. Although this makes the interactions among storage, computation, and communication more complex, its scheduling granularity remains well-defined. Indeed, the execution model would still work if individual tensors and their associated operators were grouped and scheduled at the layer level. Tensor-level scheduling instead provides finer opportunities to exploit tensor heterogeneity and optimize placement and execution decisions.
DynamicInfer also studies dynamic inference scheduling. Unlike approaches that predict additional neuron sparsity, ATSInfer executes dense operators or model-selected experts and adapts weight residency and temporary GPU execution to measured resource conditions.
Our base+TP baseline uses exactly the same tensor-placement results as ATSInfer on llama.cpp. Thus, their performance difference directly measures the benefit of asynchronous coordination and dynamic promotion under identical static placement. It does not replace direct comparisons against all cited systems or a saved-time/byte greedy scheduler. Relative to an independent saved-time/byte ranking, our formulation explicitly accounts for activation-switching costs and exposed weight-transfer costs; this is a design distinction, rather than an experimentally established advantage over that greedy baseline.
2. Algorithm 2 and Equation (4). We agree that Algorithm 2 as printed does not correctly distinguish default residency from runtime placement. In Equation (4), the activation-switching cost depends on the runtime backends:
ci,1rbi−1=rbi.
For a promoted tensor, rbi=GPU, so this term becomes
ci,1rbi−1=CPU,
rather than the condition involving the predecessor's default backend in the printed step 17. In particular, two consecutive GPU executions incur no activation-switching cost at their boundary.
With a correctly formulated DP recurrence that accounts for runtime placement in each transition, recording predecessor states is sufficient to fully reconstruct the decision chain of the optimal solution, including consecutive promotions.