acceptodds
Under review as a conference paper at ICLR 2027

HeteroEngine: Concurrent Training on Apple Silicon's GPU and Neural Engine

Abstract

Apple Silicon uses a system-on-chip (SoC) architecture in which the CPU, GPU, and Apple Neural Engine (ANE) share unified memory. The two accelerators, the GPU and the ANE, have opposite characteristics: the GPU offers high throughput but consumes more energy, whereas the ANE consumes less energy but offers lower throughput. Moreover, because the two engines share unified memory, they can access the same data and weights without the inter-device transfers required by discrete accelerators with separate device memory; this favors using them together. Standard training frameworks, however, backpropagate on the GPU alone, leaving the ANE idle throughout training. We therefore aim to use both engines concurrently, each compensating for the other's weakness—the GPU's high energy consumption and the ANE's low throughput—and thereby to exceed GPU-only training in both throughput and energy efficiency. In practice, Apple's public software exposes the ANE only through Core ML, a framework for executing trained models that provides no automatic differentiation and thus cannot compute gradients. Prior ANE training work consequently relied on Apple's unpublished internal interfaces. We present HeteroEngine, which derives the backward pass by hand and implements it as a Core ML program, thereby training on the GPU and the ANE concurrently through Apple's official APIs alone, without any such private interface. HeteroEngine partitions each training batch into microbatches and distributes them between the GPU and the ANE. Starting from the same model parameters, both engines concurrently perform the forward and backward passes of the entire model on their assigned microbatches. HeteroEngine then averages the gradients across all microbatches and applies a single optimizer update. When training GPT-2 124M on an M4 Pro, HeteroEngine achieves 1.419× the throughput of GPU-only training at 30.9% less energy per token; relative to ANE-only training, it achieves more than twice the throughput at higher energy per token. Whereas ANE-only training yields a small but statistically significant increase in perplexity, HeteroEngine matches or improves on GPU-only quality: at equal token counts its perplexity is equivalent to GPU-only perplexity on Shakespeare and WikiText-103; on WikiText-103 it is slightly lower after 18.06M tokens at equal token counts and substantially lower at equal training time; and on a Mac mini M2 the same program passes the same test and reproduces the direction of both gains. HeteroEngine thus uses the otherwise idle ANE alongside the GPU, raising both throughput and energy efficiency without loss of quality, and, built on Apple's official APIs alone, runs on a standard macOS installation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Apple Silicon, Apple Neural Engine, heterogeneous training, energy-efficient training, on-device training, Core ML, data parallelism

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_84eMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents HeteroEngine, a system for training neural networks concurrently on the GPU and Apple Neural Engine (ANE) in Apple Silicon. A key implementation challenge is that Apple's public Core ML interface does not provide general automatic differentiation for the ANE. The authors therefore manually implement the backward pass and use Core ML state and shared memory to exchange model weights and gradients without relying on Apple's private APIs.

Critical strengths

demonstrate that substantial transformer training can be performed concurrently on Apple's GPU and ANE using only public APIs.

Critical weaknesses

The scheduling mechanism is essentially static microbatch partitioning plus synchronous gradient averaging. There is no new scheduling algorithm, adaptive load balancer, heterogeneous optimizer, communication algorithm, or general placement method.

Extremely platform-specific. A substantial portion of the contribution consists of Core ML/ANE-specific engineering: MLState, ANE-compatible layouts, packing matrices because of state-count limits, output backings, and manual MIL backward code.

Very limited model/workload evaluation. The main experiment is essentially GPT-2 124M. Even that is not full training of all model parameters: the 72 dense transformer matrices are trained, while embeddings, biases, and LayerNorm parameters remain frozen.

Questions for authors

The core approach partitions microbatches between the GPU and ANE, runs them concurrently, and averages their gradients before a synchronous update. Could the authors clarify the key methodological novelty compared with existing heterogeneous and data-parallel training approaches such as QSync, Mandheling, and KAITIAN?

How well does HeteroEngine generalize to larger models, different architectures, longer sequences, and full-model training?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 84eM (1/3)

W1 and Q1. Static synchronous partitioning; QSync, Mandheling and KAITIAN

On the macOS 26 releases used in our experiments, public software reaches the ANE only through Core ML model execution, and HeteroEngine makes the ANE a synchronous data-parallel gradient worker: one Core ML prediction computes the forward pass, loss and hand-derived backward for all 72 block matrices of a 256-token microbatch (§3–4). HeteroEngine claims no new allocation rule. Giving each device a share of a fixed global batch in proportion to its speed is standard: Whale (USENIX ATC 2022, §3.3.1) sizes replica batches by GPU capacity on mixed V100/P100 clusters, and KAITIAN (§III-C) fixes its GPU and MLU shares from an initial benchmark, leaving online adaptation to future work. The difference is the worker. QSync (IPDPS 2024) trains across T4 inference GPUs and V100 training GPUs with unchanged local batches and per-operator precisions on the T4s; the devices in Whale, QSync and KAITIAN train through standard framework stacks. Mandheling (MobiCom 2022) adds INT8 training operators to a DSP inference library, schedules CPU and DSP to limit context switches rather than to run them concurrently (its §3.3), and reports a 1.9–2.7-point accuracy loss relative to FP32 training (its Table 8). In HeteroEngine the ANE runs a complete microbatch per call, concurrently with the GPU, on weights held in MLState, and writes all 72 gradients through output backings into the optimizer's buffers (§3); averaging its fp16 gradients with bf16 GPU gradients passes a paired-seed equivalence test with a prespecified 1% perplexity margin at all four tested mixed allocations, as does all-ANE training (Table 3).

The static, synchronous rule is deliberate: with one parameter version per update and a fixed-order reduction, concurrent execution reproduces a sequential reference bit for bit (§4), and the update rule stays the one the quality tests hold fixed. It also leaves little for adaptive balancing:

  • All six tested allocations, Q:K = 3:6 to 5:7, were 1.25–1.40× faster than the fastest GPU-only batch in the unmetered grid (Figure 2a; ratios computed from its archived cell means), and the neighbours of 4:6 were all slower than 4:6, so the gain does not hinge on fine allocation tuning.
  • In the metered panel the GPU and ANE legs take 226.3 and 221.0 ms (host-timer medians) of an update that averages 249.3 ms, closer than one microbatch on either engine (37.7 and 54.2 ms; Appendix A, Table 15); an ideal balancer could recover at most this 5.3 ms, about 2%.
  • Throughput changed by −0.24% and +0.27% between the first and last five minutes of the 44.8- and 33.8-minute sustained traces (§6.3), so on a dedicated machine there is little drift for a dynamic scheduler to track.
  • The hybrid reaches 87.6% of the summed GPU-only and all-ANE throughputs; the remainder comes mainly from the unoverlapped update step (19.3 ms) and slower GPU microbatches (§7), not imbalance. Overlapping the update would require predictions on weights one update old, which changes the update rule.

The other items listed in W1 are absent by design. The optimizer is unchanged Adam on the averaged gradient, and the GPU-only controls apply the same arithmetic with the ANE term zero, which is what lets paired seeds attribute quality differences to engine arithmetic. No communication algorithm is needed: output backings place the ANE gradients in the optimizer's buffers, and the per-update weight transfer is one copy of the fp16 weight image into state (3.2–3.3 ms; Appendix A). Placement is per microbatch: Core ML dispatch to the ANE pays off only for batch-scale work (Mohite, 2026), so each ANE call carries a complete forward and backward pass (54.2 ms per microbatch, Table 15).

What the paper adds as method is the worker construction and its validation, not the schedule: a complete training microstep compiled as one stateful inference program (weights as host-rewritable state, gradients as caller-owned outputs), and a gradient validation that compares all 72 matrix gradients with an fp32 reference (sampled and full-vector) and separates engine arithmetic from partition effects with mirrored GPU controls (§5, Appendix B). We are not aware of prior work that trains on the ANE through public Core ML alone, or that trains on the ANE and the GPU in one synchronous update.

AuthorsOpenReview ↗

Response to Reviewer 84eM (2/3)

W2. Platform specificity

The Core ML-specific engineering answers measured constraints of the only public route to the ANE on the macOS releases we used: weight constants require recompilation, tensor inputs cost about 10 ms/MB (666 ms for 67 MB versus 4.2–4.6 ms state-resident), the ANE compiler rejects in-graph state updates, and on the M4 Pro 52 states compile while 54 fail (Appendix C).

What transfers is a design pattern and a measurement method; the platform-specific part is identified. Abstracted from Core ML, HeteroEngine relies on three runtime properties: accelerator state that the host can rewrite between calls without recompilation, caller-owned outputs in memory shared with the GPU, and a compiler that accepts a backward pass written in forward operations. The update rule (Eq. 1), the fixed-order join with one optimizer step (§4) and the validation procedure (§5: fp32-referenced gradients, partition-mirroring controls, paired endpoint tests) use none of Core ML's specifics, and neither do the fp16 range controls of the backward: loss scaling (λ = 1024), pre-scaled head probabilities (C = 4096), per-layer scaling of attention adjoints, and the shifted-softmax form with a common maximum and denominator across vocabulary chunks (§5; we will document the attention scaling in Appendix C). The MIL graph, the vocabulary chunking, the hand-derived backward and the state packing are what another accelerator would require rewriting. Other inference NPUs also lack a native backward path: FwdLLM (USENIX ATC 2024) avoids backpropagation on mobile NPUs because they lack backpropagation-specific operators; Rösti and Franz (FCCM 2025 poster; arXiv 2504.03083) write forward and backward GEMM kernels for an AMD XDNA NPU with AMD's bare-metal tool-flow; and concurrent work (arXiv 2610.24816) adds input-gradient kernels to an inference backend for Snapdragon NPUs. Within Apple Silicon, the same MIL program, rebuilt and compiled on a Mac mini M2, kept all 1,586 operations ANE-preferred and passed the same Shakespeare equivalence test (+0.0022%, 90% CI [−0.097%, +0.102%], five seeds at Q = 2, K = 2; Appendix C.1), while the measured constants differ (state-count boundary 52/54 versus 54/56; ingestion 9.7 versus 8.7 ms/MB; Table 14). The same program generator also trains the Llama-2 (Stories110M) and Qwen2.5 (0.5B) architectures and the 24-block GPT-2 medium on the ANE; at equal tokens the hybrid is 1.40–1.47× GPU-only (M4 Pro, measured for this response). The hardware setting, a GPU and an NPU integrated on one SoC, is common in current phone and laptop processors (e.g., Qualcomm Snapdragon, AMD Ryzen AI, Intel Core Ultra); whether their runtimes provide the three properties above is untested.

AuthorsOpenReview ↗

Response to Reviewer 84eM (3/3)

W3 and Q2. Model and workload scope; larger models, other architectures, longer sequences, full-model training

Measured for this response on the M4 Pro (macOS 27.0.1), the same design trains two other architectures and models with up to 4.2 times as many trained parameters, with the weight gradients of every block matrix computed on the ANE through public Core ML:

Model (trained block-matrix parameters)Hybrid ÷ GPU-only, equal tokens
Stories110M, Llama-2 architecture (84.9M)1.46× (Q = 4, K = 6 vs M = 10)
GPT-2 medium, 24 blocks (302.0M)1.45–1.47× (Q = 4, K = 6 vs M = 10)
Qwen2.5-0.5B (357.8M)1.40× (Q = 2, K = 3 vs M = 5)

Throughput ratios are over updates 5–124 in two blocks of rotated order; after 124 updates at equal tokens, held-out cross-entropy is within 0.1% of the GPU-only run (one seed). Embeddings, biases and norm gains stay frozen as in the paper.

The paper's results concern fine-tuning pretrained GPT-2 124M. Every coordinate of its 72 dense transformer-block matrices is trained with full gradients (84.93M parameters; no low-rank adapters), the backward pass runs through all twelve blocks, and the token and position embeddings (39.38M entries; the token embedding is tied to the output layer) and 121,344 bias and LayerNorm parameters stay frozen (§3, Appendix C); microbatches have 256 tokens. Within that scope the evidence covers five ANE allocations on Shakespeare with five paired seeds each, a 20M-token WikiText-103 prefix through 18.06M sampled tokens, and a second chip (M2) on Shakespeare (Table 3, §6.4, Appendix C.1). We will scope the abstract accordingly: "the forward and backward passes of the entire model" becomes "forward and backward passes through all twelve blocks, with weight gradients for the 72 block matrices", and "matches or improves on GPU-only quality" and "without loss of quality" become "perplexity within a prespecified 1% equivalence margin at equal token counts".

  • Longer sequences: the program is compiled for 256-token windows, and ANE placement depends on tensor size (Table 14), so another length is a new program that needs its own placement check; training at a longer context was not run.
  • Full-model training: the head already splits the 50,257-row embedding into four vocabulary chunks (§5); training the tied embedding adds chunked state and a 50,257 × 768 gradient output of the same form.

Test status: the equivalence tests (±ln 1.01, paired 90% intervals) were prespecified; the fixed-time and same-token M = 24 contrasts are secondary pointwise 95% intervals; the long-run advantage criterion and the fp32 evaluator were specified after initial results on the same seeds, so that result is a confirmation on a deterministic rerun, not an independent replication (§6.4, Appendix A).

Reviewer_eP2zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HeteroEngine, a system that enables hybrid training on Apple Silicon by leveraging public APIs to utilize both the GPU and the Apple Neural Engine. Since the public APIs do not support automatic differentiation, the authors implemented the backward pass manually. On an M4 Pro, this approach achieves a 1.419x speedup while maintaining training quality. Although the engineering implementation is valuable, I have reservations regarding whether the paper offers insights that beyond an engineering report.

Critical strengths
  • The paper provides a complete engineering implementation, achieving heterogeneous backpropagation by manually constructing the backward pass using existing APIs.
  • The experiments are detailed, including comparisons across different random seeds and measurements of energy consumption.
Critical weaknesses
  • The paper lacks clear, high-level insights and does not propose any distinct methodological innovations.
  • There is no fundamental difference between the computational graphs for the forward and backward passes. The absence of an automatic differentiation API does not imply a functional limitation.
  • The "validation procedure" compares high-precision and low-precision results to locate the source of errors. This is a standard correctness check, so it should not be counted as a contribution of the paper.
Questions for authors

Q1: One of the key challenges of training on NPUs is their limited support for efficient weight updates. However, the ANE already supports mutable tensor through ML State APIs, which is paper uses. Could the authors clarify what system-level contributions the paper makes beyond identifying these APIs?

Q2: Could the authors explain how their validation procedure differs from, and offers advantages over, standard gradient correctness verification methods?

Q3: Beyond a straightforward manual implementation of the backward pass, what specific technical insights or methodologies does this work offer that could be generalized to other inference accelerators?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eP2z (1/3)

W1. High-level insights and the type of contribution

Three findings go beyond the implementation.

(i) On the ANE, the constraint that decided trainability was weight mutability, not differentiation. Weights compiled as constants must be recompiled whenever they change (a fresh build's first load spends several seconds in ANE compilation, against a 249 ms update); weights passed as inputs cost about 10 ms/MB (666 ms per prediction for 67 MB); with the same 67 MB held in host-rewritable state, a prediction takes 4.2–4.6 ms (Appendix A, C, C.1).

(ii) In fp16, the algebraic form of a hand-derived backward decides gradient accuracy, so acceptance cannot rest on loss agreement. Two algebraically equivalent softmax backwards necessarily return the same scalar loss, since only the backward differs, yet their pooled relative gradient errors against an fp32 reference are 0.364 and 0.019 (Table 9), and their training runs end +3.02% and +0.06% from GPU perplexity (Figure 6).

(iii) On unified memory, synchronous data parallelism adds the ANE's throughput while quality stays within a prespecified margin: the hybrid reaches 87.6% of the summed GPU-only M = 24 and all-ANE throughputs (Appendix A), and fp16 ANE gradients averaged with bf16 GPU gradients pass the prespecified 1% perplexity equivalence test at all four tested mixed allocations, as does all-ANE training (Table 3).

The contribution is systems and empirical. The allocation rule and the optimizer are deliberately standard (synchronous data parallelism, one Adam step on the averaged gradient, Eq. 1), so the measured gains are attributable to adding the ANE. We will state these findings in §1 in place of API-level wording.

W2. Forward and backward graphs; the role of automatic differentiation

A backward pass written in forward operations is exactly how HeteroEngine trains on the ANE: one Core ML program computes the forward pass, the loss and all 72 weight gradients in a single prediction, and its compiled plan marks all 1,586 operations ANE-preferred (§3). The limitation that required system design is that the weights change at every update. Public Core ML compiles weight constants into the program and ingests tensor inputs at about 10 ms/MB, and the ANE compiler rejects in-graph state updates (Appendix C); HeteroEngine therefore keeps the 72 matrices in MLState that the ANE reads and the host rewrites between predictions (3.2–3.3 ms per update), and returns gradients through output backings into GPU optimizer buffers (§3). The public update path does not provide this on the ANE: Core ML's MLUpdateTask does not support ML programs (Apple Inc., 2024b), and from a decompilation of the ANE compiler an independent guide reports that the engine has no native gradient operation and that Core ML's on-device updates run the backward on the GPU or the host (Bryngelson, 2026a, §15.1). Orion, the closest prior ANE training system, states that Core ML supports neither direct ANE programming nor on-device training and bypasses it through private APIs (Kumaresan, 2026).

Expressibility is also not correctness in fp16. Our first softmax backward (the log-normalizer variant, §5) returned the same scalar loss as the corrected one (only the backward differs), yet its pooled relative gradient error against an fp32 reference was 0.364 versus 0.019, and its training run ended +3.02% versus +0.06% from GPU perplexity (Table 9, Figure 6). The submitted program also pre-scales head probabilities by C = 4096 (Eq. 3) and scales each layer's attention adjoints by 64/max|dO| before the weight-gradient products; in development probes against an fp32 reference, both were introduced to keep fp16 accumulations out of the subnormal range and improved gradient agreement. A plain reverse-mode derivative makes neither choice; we will document the attention scaling in Appendix C. We did not test exporting an autodiff-generated graph to the ANE. We will correct the abstract's "provides no automatic differentiation and thus cannot compute gradients" to "whose ML Program format does not generate a backward pass".

AuthorsOpenReview ↗

Response to Reviewer eP2z (2/3)

W3 and Q2. How the validation procedure differs from standard gradient checks

The procedure separates engine numerics from partition effects and ties a gradient error to its training cost; a loss check or a per-operator check does neither. In low precision no reference path is exact, so it also measures the ANE path against the GPU arithmetic it replaces rather than against a fixed tolerance.

Attribution inside the split step. GPU-mirror controls keep the hybrid's partition (Q individual microbatches plus one batched group of K) but use GPU arithmetic throughout. With the log-normalizer head (ten seeds, M = 4 and 5, 300 updates), grouping and batched arithmetic changed endpoint CE by +0.0001 to +0.0010, while replacing the GPU arithmetic of the Q individual microbatches by the ANE's added +0.036 to +0.045 (Appendix B). With the corrected head at fixed weights under the actual 6 + 4 partition, partitioning alone moved gradient error outside a ±10% band in 3 of 12 conditions, and the hybrid path (GPU 6 + ANE 4) was still closer to the fp32 reference than the GPU-only path in all 12 (median full-vector error 7.26% versus 10.49%; GPU batch 10: 10.33%; Table 10).

Training consequence. The two head variants return identical scalar losses on the same state and input, as expected for a backward-only change, with pooled relative gradient errors of 0.364 and 0.019 against an fp32 reference (Table 9). The paired five-seed endpoint test with a prespecified ±ln(1.01) margin measures what that difference costs: +3.02% versus +0.06% perplexity (Figure 6).

Per-operator vector–Jacobian checks against closed forms and finite differences (e.g., Bryngelson, 2026a, §15.2) test operations in isolation; this procedure measures the effect on all 72 matrix gradients and on training. We will qualify Appendix B's "Reduction grouping and batched arithmetic are neutral" to the log-normalizer endpoint protocol in which it was measured.

Q1. System contributions beyond the MLState API

MLState makes a tensor mutable by the host, not by the ANE program. Core ML documents states as values that persist and update across inference runs, with a transformer KV cache as the worked example (Apple Inc., 2024c); in our contract probes (macOS 26.5.2, coremltools 9.0) the ANE compiler rejects such in-graph updates (coreml_update_state), and the same program compiles and runs only in CPU-only and CPU+GPU configurations (Appendix C). HeteroEngine therefore never updates weights on the ANE; its system contribution is the training design built around that boundary, each part measured:

  • Residency: the 72 matrices stay in state and the host rewrites them between predictions; for 67 MB of weights, state-resident prediction takes 4.2–4.6 ms versus 666 ms when the same weights are passed as inputs (Appendix C), and installing each update takes 3.2–3.3 ms (Appendix A).
  • State budget: on the M4 Pro, 52 states compile and 54 fail even at half the payload, so the matrices are packed into 24 states and sliced inside the graph for both passes (§3, Appendix C).
  • One program per microbatch: forward, loss and hand-derived backward for all 72 matrices, with all 1,586 operations ANE-preferred under CPUAndNeuralEngine, which excludes the GPU, and ANE activity checked on the power rail (§3).
  • Gradient exchange and one update for two engines: caller-owned output backings let the ANE write all 72 gradients directly into MLX buffers (Appendix C); one compiled GPU step combines the ANE and GPU gradients in a fixed order and applies a single Adam update (§4).

On the earlier head program, which shares the driver, native dispatch cuts prediction latency from 106.9 to 48.2 ms relative to the Python bindings (Table 12). End to end, the hybrid delivers 1.419× the throughput of the grid-selected GPU-only control at 0.691× its CPU+GPU+ANE energy per token (§6.3). We will correct §3's "avoiding the fallback observed with in-graph updates" to "because the ANE compiler rejects in-graph updates (Appendix C)".

AuthorsOpenReview ↗

Response to Reviewer eP2z (3/3)

Q3. Insights and methods that transfer to other inference accelerators

What transfers is a design pattern and a measurement method; the platform-specific part is identified. Abstracted from Core ML, HeteroEngine relies on three runtime properties: accelerator state that the host can rewrite between calls without recompilation, caller-owned outputs in memory shared with the GPU, and a compiler that accepts a backward pass written in forward operations. The update rule (Eq. 1), the fixed-order join with one optimizer step (§4) and the validation procedure (§5: fp32-referenced gradients, partition-mirroring controls, paired endpoint tests) use none of Core ML's specifics, and neither do the fp16 range controls of the backward: loss scaling (λ = 1024), pre-scaled head probabilities (C = 4096), per-layer scaling of attention adjoints, and the shifted-softmax form with a common maximum and denominator across vocabulary chunks (§5; we will document the attention scaling in Appendix C). The MIL graph, the vocabulary chunking, the hand-derived backward and the state packing are what another accelerator would require rewriting. Other inference NPUs also lack a native backward path: FwdLLM (USENIX ATC 2024) avoids backpropagation on mobile NPUs because they lack backpropagation-specific operators; Rösti and Franz (FCCM 2025 poster; arXiv 2504.03083) write forward and backward GEMM kernels for an AMD XDNA NPU with AMD's bare-metal tool-flow; and concurrent work (arXiv 2610.24816) adds input-gradient kernels to an inference backend for Snapdragon NPUs. Within Apple Silicon, the same MIL program, rebuilt and compiled on a Mac mini M2, kept all 1,586 operations ANE-preferred and passed the same Shakespeare equivalence test (+0.0022%, 90% CI [−0.097%, +0.102%], five seeds at Q = 2, K = 2; Appendix C.1), while the measured constants differ (state-count boundary 52/54 versus 54/56; ingestion 9.7 versus 8.7 ms/MB; Table 14). The same program generator also trains the Llama-2 (Stories110M) and Qwen2.5 (0.5B) architectures and the 24-block GPT-2 medium on the ANE; at equal tokens the hybrid is 1.40–1.47× GPU-only (M4 Pro, measured for this response). The hardware setting, a GPU and an NPU integrated on one SoC, is common in current phone and laptop processors (e.g., Qualcomm Snapdragon, AMD Ryzen AI, Intel Core Ultra); whether their runtimes provide the three properties above is untested.

Thank you for the detailed supplementary experiments, which have helped clarify the paper's contributions. However, my primary concern remains that the system relies heavily on existing APIs and established techniques, while its methodological novelty and generalizable insights remain limited. In my view, although this is a solid technical report on a specific architecture, it does not demonstrate sufficiently novel methodologies or broader insights. Therefore, I have decided to maintain my initial assessment.

Reviewer_dySmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem that on Apple Silicon, the ANE is typically exposed only for inference through Core ML, while the public Core ML's ML Program lacks automatic differentiation and training loops, leaving the ANE completely idle during standard framework training. The authors implement hand-written backward propagation using only Apple's official APIs, place all 72 weight matrices of GPT-2 124M into mutable MLStates, write gradients back with zero-copy using output backings, and synchronize and average the microbatch gradients between the GPU and the ANE.

Critical strengths

The Core ML ML Program indeed lacks autodiff and training loops. HeteroEngine bypasses the need for recompilation by leveraging mutable MLState, hand-written backward propagation, and PyObjC public APIs to enable training on the ANE. This has engineering value, and the paper provides detailed experimental protocols.

Critical weaknesses
  1. My primary concern is: what is the practical value of performing AI training on Apple Silicon? Given that Apple chips lack interconnect methods such as NVLink or multi-chip NoC interconnects, their computational scale is extremely limited and only suitable for small-model training in edge computing scenarios. I believe a more reasonable approach is to leverage server-grade GPUs/TPUs for training and fine-tuning, and to perform only AI inference at the edge, which I suspect, is precisely the rationale behind Core ML's design choices.
  2. Even setting aside the question of whether training on-device aligns with practical needs, I find the novelty of this work to be very limited. Its primary contribution lies in engineering implementation rather than the proposal of new methods, models, or theories. It reads more like a technical report, and I am not convinced that ICLR is the appropriate venue for showcasing this type of work.
Questions for authors
  1. Please answer the first question I raised in Weaknesses: what is the practical value of performing AI training on Apple Silicon? If you disagree with my perspective, please provide a detailed discussion of this issue in the appendix.
  2. As an academic paper, the authors should clearly articulate their specific contributions and avoid exaggeration. I consider it inappropriate to frame this work as a novel framework for concurrent GPU+ANE training. The actual contribution should be described more like "ANE training under public APIs + dual-engine microbatch concurrency."
  3. The authors should include comparisons with existing works such as Orion/maderix, e.g., throughput/energy consumption comparisons on the same model (GPT-2 124M or Stories110M) and the same chip.
  4. Regarding Table 4, the authors should add a control comparison under the same M=10, same precision (both bf16 or both fp16), and same optimizer implementation.
  5. For the energy consumption description in the paper, it should be clarified whether thermal throttling is triggered.
  6. (Minor issue) I think the abstract is too long and could be appropriately condensed.
4 comments
AuthorsOpenReview ↗

Response to Reviewer dySm (1/4)

W1 and Q1. Practical value of training on Apple Silicon

A single Apple Silicon machine holds up to 512 GB of unified memory (Mac Studio; up to 128 GB in a MacBook Pro), against 32 GB on a GeForce RTX 5090. Models whose fine-tuning state exceeds any consumer GPU therefore fit on one machine, next to the data, without a multi-GPU server. Such training is slower than on datacenter GPUs, and HeteroEngine raises its throughput with the Neural Engine that standard frameworks leave idle: the ANE trains concurrently with the GPU on the same unified memory, without copying weights or gradients between devices (§3). On GPT-2 124M on the M4 Pro this gives 1.419× [1.415, 1.422] the throughput of GPU-only M = 24, the fastest batch size in a separate unmetered grid, at 0.691× its CPU+GPU+ANE energy per token (powermetrics rail estimates; DRAM and wall power excluded), with perplexity within a prespecified ±1% equivalence margin at equal token counts and lower at equal training time on WikiText-103 (−1.354% against M = 10, −1.943% against M = 24; §6.2, §6.4). For a 10M-token fine-tuning job, Table 4 implies 16.2 instead of 23.0 minutes and 28.9 instead of 41.8 kJ of rail energy (arithmetic on Table 4).

Interconnect bounds scaling across chips; it does not enter this setting. Each synchronous update completes inside one SoC, where the GPU and the ANE share unified memory and the ANE writes its gradients directly into the GPU optimizer's buffers (§3).

Apple's platforms support on-device training: Core ML ships an on-device update API, MLUpdateTask, for neural-network, pipeline and nearest-neighbor models, though not for ML programs (Apple Inc., 2024b); MLX and PyTorch MPS train on the GPU (§1); and Apple's MeBP (EMNLP 2025 Industry Track) fine-tunes LoRA adapters on an iPhone 15 Pro Max. What is missing is training on the ANE: the ML Program format, which provides the mutable state HeteroEngine uses, has no automatic differentiation or training loop, so standard frameworks backpropagate on the GPU alone and leave the ANE idle (§1). A new appendix will discuss this setting, Apple's training paths and the ANE gap, and the derived time and energy.

W2 and Q2. Contribution, scope and venue

The description in Q2, "ANE training under public APIs + dual-engine microbatch concurrency", is Contribution 1 as stated in §1: gradients for all 72 dense transformer-block matrices of GPT-2 124M are computed on the ANE through public Core ML and combined with concurrently computed GPU gradients in one synchronous update (§1, §4). The revised introduction will use that wording. Contribution 2 is the gradient-validation procedure that made Contribution 1 reliable, and Contribution 3 is the measured effect. The paper does not present HeteroEngine as a novel framework and claims no new scheduling or optimization algorithm: the allocation is synchronous data parallelism with unequal per-engine shares and one Adam step on the averaged gradient (Eq. 1).

Three findings go beyond the implementation. (i) On the ANE, the constraint that decided trainability was weight mutability, not differentiation: weights passed as inputs cost about 10 ms/MB (666 ms for 67 MB versus 4.2–4.6 ms state-resident), and the ANE compiler rejects in-graph state updates, so the host must rewrite state between calls (Appendix C). (ii) In fp16, the algebraic form of a hand-derived backward decides gradient accuracy: two algebraically equivalent softmax backwards, which by construction return the same scalar loss, differ in pooled relative gradient error against fp32 (0.364 versus 0.019) and in training outcome (+3.02% versus +0.06% perplexity against GPU training; Table 9, Figure 6). (iii) fp16 ANE gradients averaged with bf16 GPU gradients in one synchronous update pass a prespecified 1% perplexity equivalence test at all four tested mixed allocations, as does all-ANE training (Table 3), and the hybrid reaches 87.6% of the summed GPU-only M = 24 and all-ANE throughputs (Appendix A). Orion, the closest prior ANE training system, states that Core ML supports neither direct ANE programming nor on-device training and bypasses it through private APIs (Kumaresan, 2026). ICLR 2027's call for papers lists infrastructure, software libraries and hardware among its topics.

AuthorsOpenReview ↗

Response to Reviewer dySm (2/4)

Q3. Comparison with Orion and maderix/ANE on the same chip

On each chip, under one OS and in one session, the HeteroEngine hybrid trains 5.0× faster than Orion and 3.0× faster than maderix/ANE on the M2, with 62% and 47% less energy per token, and 4.9× faster than maderix/ANE on the M4 Pro, with 34% less. All-ANE training alone is 4.0× and 2.4× faster on the M2 and 2.2× faster on the M4 Pro, with 74%, 63% and 72% less energy. Mean ± SD over three blocks per chip:

SystemWeight gradients onM2: tok/sM2: J per 1k tokensM4 Pro: tok/sM4 Pro: J per 1k tokens
OrionCPU799 ± 0.28.26 ± 0.06does not run—
maderix/ANE, dynamic pipelineCPU1,353 ± 565.95 ± 0.662,189 ± 143.83 ± 0.57
HeteroEngine all-ANE (Q = 4, K = 0)ANE3,229 ± 332.18 ± 0.124,914 ± 1071.08 ± 0.12
HeteroEngine hybrid (Q = 4, K = 6)ANE and GPU4,010 ± 693.13 ± 0.1510,831 ± 4302.54 ± 0.25

Setup, measured for this response. M2: Mac mini (16 GB), macOS 26.4.1, AC power. M4 Pro: the MacBook Pro used throughout the paper, now on macOS 27.0.1, AC power and High Power. Orion (same commit, its current main) does not run there: its ANE programs fail to compile (Code=10, verifyBundleAtPath: invalid model), as also reported for macOS 27.0.1 and 27.2 in Orion's issue #5. Orion (commit 4dcc1c2) and maderix/ANE (dynamic pipeline, commit d91c984) were built from their official repositories with Apple's Command Line Tools and run with their default training settings, periodic checkpointing disabled, on Stories110M and the same pretokenized TinyStories file; HeteroEngine trains GPT-2 124M. Runs were sequential, one process at a time, in three blocks of rotated order, with the powermetrics settings of Table 4 and CPU+GPU+ANE energy; each window starts after compilation and warm-up (Orion steps 5–60, maderix steps 10–290, HeteroEngine updates 5–124). Under the timing most favorable to each prior system (Orion's self-reported time without its per-step weight reload, maderix's without AdamW), the hybrid is still 4.0× and 2.4× faster on the M2 and 3.6× faster on the M4 Pro. Subtracting each session's idle power (0.46 W and 0.78 W) changes no energy ratio by more than 0.05.

The comparison does not favor HeteroEngine. The three models have the same 84,934,656 parameters in block matrices, hence equal block FLOPs per token, and HeteroEngine's classifier is the largest: 50,257 classes, against Orion's 32,000 and the 9,205 tokens that maderix keeps after compacting its vocabulary to those occurring in TinyStories. Orion ran with its repository default of 10 accumulation steps, which spreads its weight reload over more tokens than the 4 used in its paper. HeteroEngine also trains Stories110M itself: on the M4 Pro it reaches about 12,000 tok/s hybrid and 5,200 tok/s all-ANE, 5.5× and 2.4× the maderix/ANE rate on the same model, data and chip (throughput only, separate session).

The gap follows from where the work runs. Orion and maderix keep weight gradients and Adam on the CPU (4.9–8.4 W of CPU power in our traces), and Orion rewrites and reloads its weight files every step (about 640 ms of 3.2 s on the M2). HeteroEngine computes all 72 weight gradients on the ANE and rewrites the weights inside Core ML state (3.2–3.3 ms per update on the M4 Pro; Appendix A); the hybrid adds GPU microbatches, which neither prior system trains on.

Orion and maderix/ANE reach the ANE through private interfaces; HeteroEngine uses public Core ML. An independent guide describes the private runtime used by direct-access projects as reachable from user space but undocumented and unsupported (Bryngelson, 2026a). §2 describes the initial maderix/ANE release we cite; its later dynamic pipeline, measured here, passes weights as IOSurface inputs through the same private APIs, and we will describe it. We will add this table and its protocol to the revised paper, with a related-work comparison that also covers ANEForge.

AuthorsOpenReview ↗

Response to Reviewer dySm (3/4)

Q4. Same M, same precision, same optimizer

Table 4 contains the batch-matched control: bf16 GPU-only M = 10 processes the same 2,560 tokens per update as the hybrid Q = 4, K = 6 and was measured in the same five metered blocks. With paired 95% intervals over five blocks:

Hybrid Q = 4, K = 6 againstThroughputCPU+GPU+ANE energy per tokenPanel
bf16 GPU M = 101.478× [1.473, 1.484]0.701× [0.696, 0.706]Table 4 panel (record 318)
fp16 GPU M = 101.514× [1.495, 1.533]0.696× [0.683, 0.710]Appendix B panel (record 334)
fp16 GPU M = 201.440× [1.422, 1.457]0.692× [0.679, 0.704]Appendix B panel
bf16 GPU M = 24, same session1.373× [1.356, 1.391]0.712× [0.699, 0.726]Appendix B panel

The bf16 M = 10 energy interval and both fp16 M = 10 intervals come from the archived analyses of their own five-block panels; the paper prints their point values (Table 15; Appendix B). M = 20 is the fp16 batch selected by that panel's grid under a prespecified 1% tie rule (M = 24 had the highest median; M = 20 was 0.97% below).

Precision. The ANE program computes in fp16 and the hybrid's GPU leg is the same bf16 graph as the bf16 control (§3, §6.1), so the hybrid is mixed-precision by construction. The fp16 rows match the ANE's datatype with a stabilized fp16 GPU path (fp32 LayerNorm intermediates, loss scale 1024; median gradient error 1.12% against fp32; Appendix B). A hybrid with an fp16 GPU leg was not measured. At M = 10 and equal tokens, fp16-GPU perplexity differs from the hybrid's by +0.083% [+0.027, +0.140] under the primary bf16 evaluator and −0.020% [−0.051, +0.012] under an exploratory fp32 evaluator (paired 95% CIs, five seeds; Table 11).

Optimizer. Every arm applies the same fused Adam arithmetic to fp32 masters and both moments (β1 = 0.9, β2 = 0.999, ε = 1e-8, bias correction, no weight decay), with the ANE term zero in GPU-only runs. The bf16 controls run the hybrid's own driver with the Core ML path removed, and a 300-update check reproduces the parameters, both Adam moments and the fp16 image of the quality-panel GPU control bit for bit (§6.1, Appendix A); the fp16 controls use a derived driver whose Adam function source is identical. We will add the M = 10 energy interval to Table 4 and the fp16 panel to §6.3, labeled as a separate session.

AuthorsOpenReview ↗

Response to Reviewer dySm (4/4)

Q5. Thermal throttling during the energy measurements

In the runs behind Table 4 and Figure 5, we observed no throughput decline and, in every run that executes GPU microbatches, no GPU clock reduction. These runs used AC power with the unchanged High Power setting (pmset powermode 2), checked before and after each run (Appendix A). The metered brackets last about 20–102 s per run, so the sustained traces are the relevant test. In the two sustained traces (one per arm; 44.8 and 33.8 minutes after warmup; the GPU-only trace ran second), throughput changed by −0.24% for the hybrid and +0.27% for GPU-only M = 24 from the first to the final five minutes, and every complete five-minute bin lies within 0.3% of its trace's full-window throughput (§6.3, Figure 5). A post hoc reading of the archived 1 s powermetrics samples of these traces (2,638 and 1,988 samples inside the two windows; descriptive, no interval) shows the GPU's reported active frequency at 1,575–1,578 MHz in every sample, within 3 MHz of the highest state that powermetrics lists for this GPU (1,578 MHz); the first- and final-five-minute means are 1,576.4 and 1,576.1 MHz for the hybrid and 1,577.7 and 1,577.6 MHz for GPU-only. powermetrics reports no ANE clock, but the hybrid's archived training log shows its host-timed ANE prediction phase at 214.5–216.6 ms (medians over consecutive 200-update windows, read post hoc) throughout the trace. In the five-block metered panel (500 ms samples over updates 5–124, also post hoc), the 20 runs that execute GPU microbatches average 1,577.0–1,577.8 MHz; in the all-ANE runs, where the GPU only reduces gradients, applies Adam and exports fp16 weights, it averages 1,200–1,273 MHz under that light load. Temperature and the macOS thermal-pressure level were not logged, so these data show unchanged clocks and throughput rather than a thermal bound, consistent with Appendix A, which states that the time bins do not establish thermal equilibrium. We will add these traces to Appendix A.

Q6. Abstract length

We will condense the abstract as much as possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.