- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose a mixed-precision training framework for AIMC. The approach circumvents some of the undesirable effects of analog training that arise from non-idealities in analog, by leveraging analog devices' characteristically efficient matrix computations for both inference and training while computing gradients in digital. In simulations, the authors show this approach scales comparably with an end-to-end digital pipeline on a Transformer. Thus theirs is highly relevant to the frontier where improvements in efficient training and energy consumption are most badly needed!
Critical strengths
Although my expertise in AIMC is somewhat limited, I found the paper to be highly informative, comprehensive and enjoyable to read . I found the account of non-idealities and attempts to model them generally convincing as far as individual tiles are concerned. I also generally support work in this direction and think a building more efficient training infrastructure is something that is clearly needed and given insufficient attention. This work is a great introduction to the overall state of the literature for backprop based training.
Critical weaknesses
I have several criticisms, none of which are too damning, but all of which , I think, would strengthen this work.
First and most importantly (this will affect my rec. for acceptance):
I believe you need more than one seed. That is a major weakness of these results.
Also, the whole work is built on the assumption that tiles are infinitely accommodating. In practice, some degree of partitioning would be involved. Again, my understanding of physical implementation of these devices is limited, but this is a major issue in the AIMC literature that I am familiar with . In practice this is going to necessarily introduce further non-idealities that your model doesnt really account for . This especially affects your largest experiments and the 'scaling' claim you're trying to make. I think it would be good to be more explicit about this in your limitations section. While my background is not sufficient to comment on this, I think there are also other sources of noise that would warrant passing mention and are not modelled here (again for 'limitations' section) e.g. low frequency /environmental noise , temperature effects , etc.
Secondly, I am more familiar with the literature on AIMC from the perspective of equilibrium propagation. It seems to me that some of the non-ideality issues can be avoided , though imposing additional constraints on architecture etc that make transformers and "scale" at present difficult . Nevertheless, several of the problems which the digital component of your pipeline aims to address are avoided for free with eqprop since we lose the requirement of separate forward and backward MVMs. One can avoid (at least such extensive) DAC/ADC , expensive outer products, storage of moments etc. Since this is another area of interest in the literature on AIMCs, your paper might do well to at least pay lip-service to it, and say, even very generally , why do we need backprop. While the estimates for energy savings from BP are 10 to 100 times, apparently. EP estimates are closer to 10^4 (see the Yi paper below). You can even say "nonetheless, such approaches do not yet scale to frontier data/architectures etc." But some mentioned of the proposed application of local learning on AIMC would be nice, given a shared broader motivation. Likewise the Wright paper has an approach I'm not sure you discuss here. Some more systematic account of the variety of approaches proposed in AIMC and the relative costs and benefits would be useful for readers.
Some references: Yi et al 2023 Activity-difference training of deep neural networks using memristor crossbars; Wright et al 2022 Deep physical neural networks trained with backpropagation Scellier et al 2023 Energy-based learning algorithms for analog computing: a comparative study; Scellier et al 2024 A fast algorithm to simulate nonlinear resistive networks; Nest et al 2024 Towards training digitally-tied analog blocks via hybrid gradient computation
Finally, I think more detailed estimate of the actual benefits/energy-savings/speedups oriented to be expected from your implementation vs. other proposed approaches would help a less specialized reader like myself to better assess the state of the literature and where your method falls in terms of concrete benefits. Since analog is in a relatively early state of development, we would like to make a realistic case for its application and try to anticipate some of the roadblocks that prevent money from flowing into this infrastructure. Your paper does a good job at stating from a high level "we can avoid this, this, and this issue and achieve backprop like scaling." It doesn't really speak directly to the advantages of your approach. Can we expect the "ten or 100 times" improvements in energy efficiency here , or is that no longer applicable since digital is picking up a significant share of the work? etc.
Questions for authors
I have none other than the above. I would really like to see more seeds for the main result. I'm curious if this will take prohibitively long.
4 comments
Q5: What concrete energy savings or speedups should be expected relative to other approaches? Are 10–100× improvements still applicable when digital computation handles a significant share of the work?
Thank you for requesting a clearer assessment of practical energy savings and speed.
Compared with a purely digital baseline, offloading the forward and backward linear projections onto analog arrays while keeping weight-gradient outer products, attention, and optimizer updates in digital yields an idealized compute-energy reduction of up to ~66% (2/3), reflecting that two of the three primary matrix multiplications per linear layer (forward pass and backward activation propagation) are executed in the analog domain, assuming negligible analog array energy and that linear projections dominate total FLOPs at scale.
End-to-end benefit also depends on tile parallelism, converter sharing, digital gradient computation, and programming latency. We therefore do not claim a quantified training speedup from the present simulations.
Our contribution establishes scalable training under the modeled hardware non-idealities, addressing a prerequisite for realizing analog efficiency gains. Quantifying those gains requires a specified hardware implementation and an energy–latency comparison at a common target validation loss.
We appreciate your constructive suggestions and we hope our response resolves your concerns. Thanks!
I am satisfied that my main concerns have been addressed. I intend to reread the full paper and other rebuttals before considering with an open mind my final score. I will at least not be lowering my recommendation/score which I would have done had the seed issue not been addressed.
Thank you very much for your feedback and support! We would be happy to discuss further if there are any follow-up questions.
Thank you for finding the paper informative and for recognizing the relevance of this research direction. We address each concern below, in the order raised: multiple seeds, finite-array partitioning, additional hardware effects, alternative learning approaches, and concrete energy and speed benefits.
Thank you for highlighting the need to quantify run-to-run variability. We have now completed a three-seed comparison (seeds 1337, 1338, and 1339) at all model scales. The results are already updated in the revision.
Thank you for highlighting the physical partitioning requirement. Full-network training has completed at all four tested tile limits through Large and at the 1024×1024 limit for XLarge. Smaller tiles can reduce quantization error, but reconstructing the full output requires summing multiple partial reads. The completed comparison shows convergence without a monotonic loss trend across tile sizes, with execution differences retained as a limitation; see our response to Reviewer zV7w (W1/Q2) for the endpoints and App. C.2 of the revision for the partitioning scheme and modeling scope.
Thank you for highlighting these effects. The training simulations represent device and cycle variability, noisy updates, and effective stochastic read-path perturbations, alongside quantization and clipping. They do not explicitly model the temporal correlations of low-frequency noise, environmental disturbances, temperature-dependent device and converter behavior, or long-term drift. Effective read noise should therefore not be interpreted as validation against these mechanisms.
We will make these omissions explicit in the limitations and distinguish the assumed calibrated operating point from an implemented calibration or temperature-tracking procedure. Separate circuit tests with injected thermal, 1/f, and random-telegraph noise examine selected noise-propagation mechanisms, but do not establish measurement-calibrated robustness of the training runs; see our response to Reviewer D4o6 (Q1/W2).
Thank you for your insighful discussion! EP is a particularly interesting idea with strong potential for analog training. EP and related activity-difference methods derive local learning signals from relaxed network states, offering a complementary route to explicit backpropagation. We use BP here to retain compatibility with the decoder-only Transformer and AdamW pipeline evaluated in this work. In Appendix A.2 of the revision, we now discuss the suggested works by Yi et al. (2023), Scellier et al. (2023), Scellier (2024), Wright et al. (2022), and Nest and Ernoult (2024), distinguishing local energy-based learning, simulation advances, physics-aware BP, and hybrid BP–EP.
We also believe several design principles in this work remain useful for hardware designed for EP: high-precision W-grad accumulation, particularly when small phase-contrast updates are accumulated across samples, and alignment of physical and logical weights with the conductance operating region and network signal scales. Differences between nearby equilibrium states can produce small net updates that are sensitive to quantization and noise, motivating investigation of how much accumulation precision is needed. These are potential extensions rather than EP requirements established by our BP experiments. Appendix A.2 outlines these opportunities; evaluating them in EP hardware is left to future work.