- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents HeteroEngine, a system for training neural networks concurrently on the GPU and Apple Neural Engine (ANE) in Apple Silicon. A key implementation challenge is that Apple's public Core ML interface does not provide general automatic differentiation for the ANE. The authors therefore manually implement the backward pass and use Core ML state and shared memory to exchange model weights and gradients without relying on Apple's private APIs.
Critical strengths
demonstrate that substantial transformer training can be performed concurrently on Apple's GPU and ANE using only public APIs.
Critical weaknesses
The scheduling mechanism is essentially static microbatch partitioning plus synchronous gradient averaging. There is no new scheduling algorithm, adaptive load balancer, heterogeneous optimizer, communication algorithm, or general placement method.
Extremely platform-specific. A substantial portion of the contribution consists of Core ML/ANE-specific engineering: MLState, ANE-compatible layouts, packing matrices because of state-count limits, output backings, and manual MIL backward code.
Very limited model/workload evaluation. The main experiment is essentially GPT-2 124M. Even that is not full training of all model parameters: the 72 dense transformer matrices are trained, while embeddings, biases, and LayerNorm parameters remain frozen.
Questions for authors
The core approach partitions microbatches between the GPU and ANE, runs them concurrently, and averages their gradients before a synchronous update. Could the authors clarify the key methodological novelty compared with existing heterogeneous and data-parallel training approaches such as QSync, Mandheling, and KAITIAN?
How well does HeteroEngine generalize to larger models, different architectures, longer sequences, and full-model training?
3 comments
Response to Reviewer 84eM (2/3)
W2. Platform specificity
The Core ML-specific engineering answers measured constraints of the only public route to the ANE on the macOS releases we used: weight constants require recompilation, tensor inputs cost about 10 ms/MB (666 ms for 67 MB versus 4.2–4.6 ms state-resident), the ANE compiler rejects in-graph state updates, and on the M4 Pro 52 states compile while 54 fail (Appendix C).
What transfers is a design pattern and a measurement method; the platform-specific part is identified. Abstracted from Core ML, HeteroEngine relies on three runtime properties: accelerator state that the host can rewrite between calls without recompilation, caller-owned outputs in memory shared with the GPU, and a compiler that accepts a backward pass written in forward operations. The update rule (Eq. 1), the fixed-order join with one optimizer step (§4) and the validation procedure (§5: fp32-referenced gradients, partition-mirroring controls, paired endpoint tests) use none of Core ML's specifics, and neither do the fp16 range controls of the backward: loss scaling (λ = 1024), pre-scaled head probabilities (C = 4096), per-layer scaling of attention adjoints, and the shifted-softmax form with a common maximum and denominator across vocabulary chunks (§5; we will document the attention scaling in Appendix C). The MIL graph, the vocabulary chunking, the hand-derived backward and the state packing are what another accelerator would require rewriting. Other inference NPUs also lack a native backward path: FwdLLM (USENIX ATC 2024) avoids backpropagation on mobile NPUs because they lack backpropagation-specific operators; Rösti and Franz (FCCM 2025 poster; arXiv 2504.03083) write forward and backward GEMM kernels for an AMD XDNA NPU with AMD's bare-metal tool-flow; and concurrent work (arXiv 2610.24816) adds input-gradient kernels to an inference backend for Snapdragon NPUs. Within Apple Silicon, the same MIL program, rebuilt and compiled on a Mac mini M2, kept all 1,586 operations ANE-preferred and passed the same Shakespeare equivalence test (+0.0022%, 90% CI [−0.097%, +0.102%], five seeds at Q = 2, K = 2; Appendix C.1), while the measured constants differ (state-count boundary 52/54 versus 54/56; ingestion 9.7 versus 8.7 ms/MB; Table 14). The same program generator also trains the Llama-2 (Stories110M) and Qwen2.5 (0.5B) architectures and the 24-block GPT-2 medium on the ANE; at equal tokens the hybrid is 1.40–1.47× GPU-only (M4 Pro, measured for this response). The hardware setting, a GPU and an NPU integrated on one SoC, is common in current phone and laptop processors (e.g., Qualcomm Snapdragon, AMD Ryzen AI, Intel Core Ultra); whether their runtimes provide the three properties above is untested.
Response to Reviewer 84eM (3/3)
W3 and Q2. Model and workload scope; larger models, other architectures, longer sequences, full-model training
Measured for this response on the M4 Pro (macOS 27.0.1), the same design trains two other architectures and models with up to 4.2 times as many trained parameters, with the weight gradients of every block matrix computed on the ANE through public Core ML:
| Model (trained block-matrix parameters) | Hybrid ÷ GPU-only, equal tokens |
|---|---|
| Stories110M, Llama-2 architecture (84.9M) | 1.46× (Q = 4, K = 6 vs M = 10) |
| GPT-2 medium, 24 blocks (302.0M) | 1.45–1.47× (Q = 4, K = 6 vs M = 10) |
| Qwen2.5-0.5B (357.8M) | 1.40× (Q = 2, K = 3 vs M = 5) |
Throughput ratios are over updates 5–124 in two blocks of rotated order; after 124 updates at equal tokens, held-out cross-entropy is within 0.1% of the GPU-only run (one seed). Embeddings, biases and norm gains stay frozen as in the paper.
The paper's results concern fine-tuning pretrained GPT-2 124M. Every coordinate of its 72 dense transformer-block matrices is trained with full gradients (84.93M parameters; no low-rank adapters), the backward pass runs through all twelve blocks, and the token and position embeddings (39.38M entries; the token embedding is tied to the output layer) and 121,344 bias and LayerNorm parameters stay frozen (§3, Appendix C); microbatches have 256 tokens. Within that scope the evidence covers five ANE allocations on Shakespeare with five paired seeds each, a 20M-token WikiText-103 prefix through 18.06M sampled tokens, and a second chip (M2) on Shakespeare (Table 3, §6.4, Appendix C.1). We will scope the abstract accordingly: "the forward and backward passes of the entire model" becomes "forward and backward passes through all twelve blocks, with weight gradients for the 72 block matrices", and "matches or improves on GPU-only quality" and "without loss of quality" become "perplexity within a prespecified 1% equivalence margin at equal token counts".
- Longer sequences: the program is compiled for 256-token windows, and ANE placement depends on tensor size (Table 14), so another length is a new program that needs its own placement check; training at a longer context was not run.
- Full-model training: the head already splits the 50,257-row embedding into four vocabulary chunks (§5); training the tied embedding adds chunked state and a 50,257 × 768 gradient output of the same form.
Test status: the equivalence tests (±ln 1.01, paired 90% intervals) were prespecified; the fixed-time and same-token M = 24 contrasts are secondary pointwise 95% intervals; the long-run advantage criterion and the fp32 evaluator were specified after initial results on the same seeds, so that result is a confirmation on a deterministic rerun, not an independent replication (§6.4, Appendix A).
Response to Reviewer 84eM (1/3)
W1 and Q1. Static synchronous partitioning; QSync, Mandheling and KAITIAN
On the macOS 26 releases used in our experiments, public software reaches the ANE only through Core ML model execution, and HeteroEngine makes the ANE a synchronous data-parallel gradient worker: one Core ML prediction computes the forward pass, loss and hand-derived backward for all 72 block matrices of a 256-token microbatch (§3–4). HeteroEngine claims no new allocation rule. Giving each device a share of a fixed global batch in proportion to its speed is standard: Whale (USENIX ATC 2022, §3.3.1) sizes replica batches by GPU capacity on mixed V100/P100 clusters, and KAITIAN (§III-C) fixes its GPU and MLU shares from an initial benchmark, leaving online adaptation to future work. The difference is the worker. QSync (IPDPS 2024) trains across T4 inference GPUs and V100 training GPUs with unchanged local batches and per-operator precisions on the T4s; the devices in Whale, QSync and KAITIAN train through standard framework stacks. Mandheling (MobiCom 2022) adds INT8 training operators to a DSP inference library, schedules CPU and DSP to limit context switches rather than to run them concurrently (its §3.3), and reports a 1.9–2.7-point accuracy loss relative to FP32 training (its Table 8). In HeteroEngine the ANE runs a complete microbatch per call, concurrently with the GPU, on weights held in MLState, and writes all 72 gradients through output backings into the optimizer's buffers (§3); averaging its fp16 gradients with bf16 GPU gradients passes a paired-seed equivalence test with a prespecified 1% perplexity margin at all four tested mixed allocations, as does all-ANE training (Table 3).
The static, synchronous rule is deliberate: with one parameter version per update and a fixed-order reduction, concurrent execution reproduces a sequential reference bit for bit (§4), and the update rule stays the one the quality tests hold fixed. It also leaves little for adaptive balancing:
The other items listed in W1 are absent by design. The optimizer is unchanged Adam on the averaged gradient, and the GPU-only controls apply the same arithmetic with the ANE term zero, which is what lets paired seeds attribute quality differences to engine arithmetic. No communication algorithm is needed: output backings place the ANE gradients in the optimizer's buffers, and the per-update weight transfer is one copy of the fp16 weight image into state (3.2–3.3 ms; Appendix A). Placement is per microbatch: Core ML dispatch to the ANE pays off only for batch-scale work (Mohite, 2026), so each ANE call carries a complete forward and backward pass (54.2 ms per microbatch, Table 15).
What the paper adds as method is the worker construction and its validation, not the schedule: a complete training microstep compiled as one stateful inference program (weights as host-rewritable state, gradients as caller-owned outputs), and a gradient validation that compares all 72 matrix gradients with an fp32 reference (sampled and full-vector) and separates engine arithmetic from partition effects with mirrored GPU controls (§5, Appendix B). We are not aware of prior work that trains on the ANE through public Core ML alone, or that trains on the ANE and the GPU in one synchronous update.