- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the performance degradation caused by trainable affine parameters in normalization layers during analog in-memory computing (AIMC) training. The authors show that although the gain and bias parameters remain in the digital domain and are not quantized, placing them before the DAC can amplify analog conversion errors by altering the scale and direction of input representations. Furthermore, inaccurate gradient feedback during analog backpropagation prevents training from effectively correcting this degradation. To address this issue, the paper proposes using parameter-free normalization before the DAC or moving trainable affine operations after the ADC. The authors validate their findings on both language and vision tasks, including Transformer and Vision Transformer models, across different model scales, converter precisions, and training configurations. Experimental results demonstrate that pre-DAC trainable affine parameters can significantly impair analog training performance, while appropriate affine placement effectively mitigates the issue.
Critical strengths
- The paper identifies a novel and practically important failure mode in analog training, revealing the coupling between model parameterization and analog hardware interfaces. The authors show that pre-DAC trainable affine parameters can degrade AIMC training performance by altering input scale and direction, even when these parameters remain digital and unquantized. This finding extends beyond traditional studies focusing on quantization errors and hardware noise, providing new insights into analog training degradation and guiding future analog accelerator design.
Critical weaknesses
- Lack of validation on real analog hardware limits the generality of the conclusions. The paper mainly evaluates the proposed failure mode in simulation environments and does not verify whether the phenomenon can be reproduced on real AIMC hardware platforms. This limitation is important for the overall evaluation because the core contribution of the paper is to reveal a potentially general failure mechanism in analog training, and the lack of hardware validation may weaken its practical applicability and the credibility of the conclusions.
- Insufficient analysis of generalization across different analog architectures and training scenarios. The paper mainly focuses on a specific max-absolute scaling strategy and model configurations. The applicability of the proposed findings to different AIMC architectures, scaling strategies, and larger-scale models remains insufficiently explored. Since analog computing systems vary significantly in hardware design and training pipelines, it is unclear whether the identified phenomenon can generalize to broader analog training scenarios.
Questions for authors
- Can the main conclusions of this paper be validated on real AIMC hardware platforms? The current experiments are mainly conducted in simulation environments, without validation on real hardware. The authors are encouraged to clarify whether the identified failure mode still exists when considering practical device non-idealities, such as noise, drift, and ADC/DAC errors, and whether the simulation results are consistent with real hardware behavior.
- Does the observed phenomenon depend on specific scaling strategies and hardware configurations? The paper mainly analyzes the issue under max-absolute scaling and a specific AIMC setting. The authors should further clarify whether the proposed failure mode still holds under alternative scaling strategies (e.g., per-channel or group-wise scaling) or different analog architectures.
4 comments
Supporting experiments (Tables Y.1–Y.2)
Both experiments train enwik8 d128-L4 for 5,000 steps with 6-bit forward/backward conversion and five paired seeds. Y.1 reports mean final validation CE; Y.2 reports mean ± SD.
ADC non-uniformity and calibration (Y.1). We compare the original converter setting, added ADC errors, and the same errors with per-layer correction. The added errors include channelwise gain, offset, and nonlinear distortion at every mapped linear layer. Correction uses a post-ADC affine fitted from voltage ramps without task labels and held fixed during training, separate from the task-trained normalization affine. Within each seed, readouts share distortions and calibration constants. These runs use exact weight updates and forward ADC range 12.
Table Y.1. Added ADC non-uniformity and per-layer affine calibration.
| ADC condition | Exposed CE | Removed CE | Deferred CE |
|---|---|---|---|
| No added non-uniformity; no correction | 3.3132 | 2.9549 | 2.8798 |
| Added non-uniformity; no correction | 3.8270 | 3.6779 | 3.4368 |
| Added non-uniformity; fixed affine calibration | 3.7882 | 3.4661 | 3.3163 |
Device drift during training (Y.2). We compare no drift, two strengths of uniform drift, and heterogeneous drift using AIHWKit's power-law model. The exponent controls drift strength; the heterogeneous condition uses a relative device-to-device spread of 0.2. All conditions use pulsed updates and newly trained controls. Every 100 updates, we apply drift to all mapped linear layers over an idle interval of , where is the reference time. No drift compensation is applied, and final evaluation follows the last drift event.
Table Y.2. Readout affine placement with device drift during training.
| Drift condition | Exposed CE | Removed CE | Deferred CE |
|---|---|---|---|
| No drift | 3.6934 ± 0.0749 | 3.0970 ± 0.0549 | 3.1119 ± 0.1886 |
| Uniform, | 4.1672 ± 0.2776 | 3.4695 ± 0.1862 | 3.9624 ± 0.3166 |
| Uniform, | 3.3994 ± 0.0335 | 3.3819 ± 0.0724 | 3.2797 ± 0.0319 |
| Heterogeneous, | 3.5013 ± 0.2769 | 3.4388 ± 0.1919 | 3.2964 ± 0.0311 |
The paired intervals discussed above are nominal 95% confidence intervals without correction for multiple comparisons.
Please restructure the manuscript and devote more of the main text to the experiments.
Thank you for the follow-up. We would like to clarify how the paper is positioned and why the main text is organized this way.
The paper's contribution is a mechanism: a trainable affine that stays digital and unquantized can still degrade analog training once it is placed before the DAC, and training does not repair this damage. The main text is therefore organized around that causal chain, and each step is tested by a controlled experiment. Head replay and training controls localize the damage to the analog readout (Section 3). Paired scale and input-direction diagnostics test how the affine amplifies conversion error (Section 4). Gradient interventions test whether the misleading gain feedback causes the training gap (Section 5). Comparisons of operations before the DAC and after the ADC test the remedies (Section 6, Table 1). Most of Sections 3–6 are thus experiments designed to isolate a cause. Replacing them with additional benchmark tables would show where the effect occurs but not why it occurs or when the remedies should help.
To make the scope of the evidence easier to assess, we have uploaded a revised manuscript that also moves the experiments most relevant to your concerns into the main text. In addition to Table 2, which compares readouts across model scales up to 39.4M parameters, vocabularies, normalization families, vision tasks, and training methods, Section 7 now reports:
- Alternative input scaling. Under average-max scaling, exposed has higher CE than removed in all five paired seeds, with a mean gap of 0.400 (95% CI [0.260, 0.541]). With a calibrated static readout range, the mean gap is 0.074, and its interval [−0.032, 0.179] includes zero (Appendix D.3).
- Device non-idealities (new Table 3). With channelwise ADC gain, offset, and nonlinear errors, both remedies lower mean CE. After fixed per-layer calibration, the paired CE reductions are 0.322 [0.066, 0.578] for removal and 0.472 [0.190, 0.753] for deferral; without calibration, both intervals include zero. Under uniform drift, paired intervals support removal at and deferral at ; neither benefit is established under heterogeneous drift. Exposed also has higher PPL than removed at every tested read-noise level.
These revisions make the experimental evidence and its scope more explicit in the main text while preserving the mechanism analysis. Thank you for your time and consideration throughout the review process.
Thank you for your comments and for recognizing the practical importance of this failure mode. We address your concerns below. We have uploaded a revised manuscript; Section 8 now also lists other AIMC architectures as open validation. Tables Y.1–Y.2 appear in this response.
We acknowledge that physical-hardware validation remains open, as stated in the original submission and retained in revised Section 8. Like prior AIMC studies (Wu et al., 2025), we use hardware-aware simulations to study training under device non-idealities. AIHWKit (Rasch et al., 2021) allows us to compare affine placements under controlled conditions.
Among the non-idealities you raise, the original submission already examines output noise (original Appendix H.8; revised Appendix H.4), device and update non-idealities (Appendix F in both versions), and conversion precision (original Appendices H.7 and D.8; revised Appendices H.4 and D.3).
We added ADC non-uniformity and per-layer calibration experiments (Table Y.1). Removal and deferral still improve CE after per-layer ADC calibration, reducing mean CE relative to exposed by 0.322 and 0.472, respectively. We also added training-time device drift (Table Y.2). Both changes lower mean CE in all tested drift conditions, but the evidence varies: paired intervals support removal under uniform ν=0.01 and deferral under uniform ν=0.05; the other repair in each condition remains inconclusive. Under heterogeneous drift, neither benefit is established.
The degradation also occurs under average-max scaling, as shown in the original submission's alternative-scaling experiments (original Appendix D.9; revised Appendix D.3). With a calibrated static input range at the readout, the mean CE difference is smaller and its confidence interval includes zero. Channel-wise and group-wise scaling remain to be evaluated.
Our choice of max-absolute scaling also has theoretical support. Wu et al. (2026, Section 2.3) show that it asymptotically minimizes DAC reconstruction distortion within the norm-based scaling family they analyze, under their distributional assumptions as converter precision increases.
The added digital converter-quantization controls also reproduce the degradation (Table F.4 in our response to reviewer F2cy). The effect is therefore not exclusive to analog hardware. These controls do not test another AIMC architecture; validation across AIMC architectures remains open, as noted in revised Section 8.
Regarding your concern about larger models, the original submission also tests a 39.4M-parameter Transformer, approximately 50× the primary model's size (original Appendix H.2, Table 46; revised Appendix H.1, Table 43). Exposed has higher CE than removed in all ten paired seeds, and deferral lowers mean CE further. The same ordering holds with RMSNorm at this scale.