acceptodds
Under review as a conference paper at ICLR 2027

Trainable Affine Parameters in Normalization Can Silently Harm Analog Training

Abstract

Analog in-memory computing (AIMC) offers a route to energy-efficient training by accelerating matrix–vector multiplications (MVMs) in memory. With finite-resolution analog MVMs in both forward and backward passes, trainable normalization gain and bias before the final analog linear layer can degrade training, despite remaining digital and unquantized. Part of the loss gap relative to fixed gain and bias persists when the trained head is replayed with exact multiplication, indicating damage retained after training. To understand this failure, we show how affine parameters can increase absolute logit error through changes in input scale and direction, despite dynamic max-absolute scaling. Training can fail to correct these effects because the supplied gain gradient can predict a loss response opposite to that measured by forward evaluations. These findings motivate parameter-free normalization before the digital-to-analog converter, with any needed affine placed after the analog-to-digital converter. Normalizing the digitized output can also mitigate the damage. Language and vision experiments reproduce the failure and its repairs across converter precisions, optimizers, and model scales, with repair benefits depending on the task, optimizer, and operating point.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
analog AI, analog training, in-memory computing, layer normalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ypq9OpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the performance degradation caused by trainable affine parameters in normalization layers during analog in-memory computing (AIMC) training. The authors show that although the gain and bias parameters remain in the digital domain and are not quantized, placing them before the DAC can amplify analog conversion errors by altering the scale and direction of input representations. Furthermore, inaccurate gradient feedback during analog backpropagation prevents training from effectively correcting this degradation. To address this issue, the paper proposes using parameter-free normalization before the DAC or moving trainable affine operations after the ADC. The authors validate their findings on both language and vision tasks, including Transformer and Vision Transformer models, across different model scales, converter precisions, and training configurations. Experimental results demonstrate that pre-DAC trainable affine parameters can significantly impair analog training performance, while appropriate affine placement effectively mitigates the issue.

Critical strengths
  1. The paper identifies a novel and practically important failure mode in analog training, revealing the coupling between model parameterization and analog hardware interfaces. The authors show that pre-DAC trainable affine parameters can degrade AIMC training performance by altering input scale and direction, even when these parameters remain digital and unquantized. This finding extends beyond traditional studies focusing on quantization errors and hardware noise, providing new insights into analog training degradation and guiding future analog accelerator design.
Critical weaknesses
  1. Lack of validation on real analog hardware limits the generality of the conclusions. The paper mainly evaluates the proposed failure mode in simulation environments and does not verify whether the phenomenon can be reproduced on real AIMC hardware platforms. This limitation is important for the overall evaluation because the core contribution of the paper is to reveal a potentially general failure mechanism in analog training, and the lack of hardware validation may weaken its practical applicability and the credibility of the conclusions.
  2. Insufficient analysis of generalization across different analog architectures and training scenarios. The paper mainly focuses on a specific max-absolute scaling strategy and model configurations. The applicability of the proposed findings to different AIMC architectures, scaling strategies, and larger-scale models remains insufficiently explored. Since analog computing systems vary significantly in hardware design and training pipelines, it is unclear whether the identified phenomenon can generalize to broader analog training scenarios.
Questions for authors
  1. Can the main conclusions of this paper be validated on real AIMC hardware platforms? The current experiments are mainly conducted in simulation environments, without validation on real hardware. The authors are encouraged to clarify whether the identified failure mode still exists when considering practical device non-idealities, such as noise, drift, and ADC/DAC errors, and whether the simulation results are consistent with real hardware behavior.
  2. Does the observed phenomenon depend on specific scaling strategies and hardware configurations? The paper mainly analyzes the issue under max-absolute scaling and a specific AIMC setting. The authors should further clarify whether the proposed failure mode still holds under alternative scaling strategies (e.g., per-channel or group-wise scaling) or different analog architectures.
4 comments

Thank you for your comments and for recognizing the practical importance of this failure mode. We address your concerns below. We have uploaded a revised manuscript; Section 8 now also lists other AIMC architectures as open validation. Tables Y.1–Y.2 appear in this response.

W1/Q1. Can the main conclusions of this paper be validated on real AIMC hardware platforms?

We acknowledge that physical-hardware validation remains open, as stated in the original submission and retained in revised Section 8. Like prior AIMC studies (Wu et al., 2025), we use hardware-aware simulations to study training under device non-idealities. AIHWKit (Rasch et al., 2021) allows us to compare affine placements under controlled conditions.

Among the non-idealities you raise, the original submission already examines output noise (original Appendix H.8; revised Appendix H.4), device and update non-idealities (Appendix F in both versions), and conversion precision (original Appendices H.7 and D.8; revised Appendices H.4 and D.3).

We added ADC non-uniformity and per-layer calibration experiments (Table Y.1). Removal and deferral still improve CE after per-layer ADC calibration, reducing mean CE relative to exposed by 0.322 and 0.472, respectively. We also added training-time device drift (Table Y.2). Both changes lower mean CE in all tested drift conditions, but the evidence varies: paired intervals support removal under uniform ν=0.01\nu=0.01 and deferral under uniform ν=0.05\nu=0.05; the other repair in each condition remains inconclusive. Under heterogeneous drift, neither benefit is established.

W2/Q2. Does the observed phenomenon depend on specific scaling strategies and hardware configurations?

The degradation also occurs under average-max scaling, as shown in the original submission's alternative-scaling experiments (original Appendix D.9; revised Appendix D.3). With a calibrated static input range at the readout, the mean CE difference is smaller and its confidence interval includes zero. Channel-wise and group-wise scaling remain to be evaluated.

Our choice of max-absolute scaling also has theoretical support. Wu et al. (2026, Section 2.3) show that it asymptotically minimizes DAC reconstruction distortion within the norm-based scaling family they analyze, under their distributional assumptions as converter precision increases.

The added digital converter-quantization controls also reproduce the degradation (Table F.4 in our response to reviewer F2cy). The effect is therefore not exclusive to analog hardware. These controls do not test another AIMC architecture; validation across AIMC architectures remains open, as noted in revised Section 8.

Regarding your concern about larger models, the original submission also tests a 39.4M-parameter Transformer, approximately 50× the primary model's size (original Appendix H.2, Table 46; revised Appendix H.1, Table 43). Exposed has higher CE than removed in all ten paired seeds, and deferral lowers mean CE further. The same ordering holds with RMSNorm at this scale.

Supporting experiments (Tables Y.1–Y.2)

Both experiments train enwik8 d128-L4 for 5,000 steps with 6-bit forward/backward conversion and five paired seeds. Y.1 reports mean final validation CE; Y.2 reports mean ± SD.

ADC non-uniformity and calibration (Y.1). We compare the original converter setting, added ADC errors, and the same errors with per-layer correction. The added errors include channelwise gain, offset, and nonlinear distortion at every mapped linear layer. Correction uses a post-ADC affine fitted from voltage ramps without task labels and held fixed during training, separate from the task-trained normalization affine. Within each seed, readouts share distortions and calibration constants. These runs use exact weight updates and forward ADC range 12.

Table Y.1. Added ADC non-uniformity and per-layer affine calibration.

ADC conditionExposed CERemoved CEDeferred CE
No added non-uniformity; no correction3.31322.95492.8798
Added non-uniformity; no correction3.82703.67793.4368
Added non-uniformity; fixed affine calibration3.78823.46613.3163

Device drift during training (Y.2). We compare no drift, two strengths of uniform drift, and heterogeneous drift using AIHWKit's power-law model. The exponent ν\nu controls drift strength; the heterogeneous condition uses a relative device-to-device spread of 0.2. All conditions use pulsed updates and newly trained controls. Every 100 updates, we apply drift to all mapped linear layers over an idle interval of 10t010t_0, where t0t_0 is the reference time. No drift compensation is applied, and final evaluation follows the last drift event.

Table Y.2. Readout affine placement with device drift during training.

Drift conditionExposed CERemoved CEDeferred CE
No drift3.6934 ± 0.07493.0970 ± 0.05493.1119 ± 0.1886
Uniform, ν=0.01\nu=0.014.1672 ± 0.27763.4695 ± 0.18623.9624 ± 0.3166
Uniform, ν=0.05\nu=0.053.3994 ± 0.03353.3819 ± 0.07243.2797 ± 0.0319
Heterogeneous, ν=0.05\nu=0.053.5013 ± 0.27693.4388 ± 0.19193.2964 ± 0.0311

The paired intervals discussed above are nominal 95% confidence intervals without correction for multiple comparisons.

Please restructure the manuscript and devote more of the main text to the experiments.

Thank you for the follow-up. We would like to clarify how the paper is positioned and why the main text is organized this way.

The paper's contribution is a mechanism: a trainable affine that stays digital and unquantized can still degrade analog training once it is placed before the DAC, and training does not repair this damage. The main text is therefore organized around that causal chain, and each step is tested by a controlled experiment. Head replay and training controls localize the damage to the analog readout (Section 3). Paired scale and input-direction diagnostics test how the affine amplifies conversion error (Section 4). Gradient interventions test whether the misleading gain feedback causes the training gap (Section 5). Comparisons of operations before the DAC and after the ADC test the remedies (Section 6, Table 1). Most of Sections 3–6 are thus experiments designed to isolate a cause. Replacing them with additional benchmark tables would show where the effect occurs but not why it occurs or when the remedies should help.

To make the scope of the evidence easier to assess, we have uploaded a revised manuscript that also moves the experiments most relevant to your concerns into the main text. In addition to Table 2, which compares readouts across model scales up to 39.4M parameters, vocabularies, normalization families, vision tasks, and training methods, Section 7 now reports:

  • Alternative input scaling. Under average-max scaling, exposed has higher CE than removed in all five paired seeds, with a mean gap of 0.400 (95% CI [0.260, 0.541]). With a calibrated static readout range, the mean gap is 0.074, and its interval [−0.032, 0.179] includes zero (Appendix D.3).
  • Device non-idealities (new Table 3). With channelwise ADC gain, offset, and nonlinear errors, both remedies lower mean CE. After fixed per-layer calibration, the paired CE reductions are 0.322 [0.066, 0.578] for removal and 0.472 [0.190, 0.753] for deferral; without calibration, both intervals include zero. Under uniform drift, paired intervals support removal at ν=0.01\nu=0.01 and deferral at ν=0.05\nu=0.05; neither benefit is established under heterogeneous drift. Exposed also has higher PPL than removed at every tested read-noise level.

These revisions make the experimental evidence and its scope more explicit in the main text while preserving the mechanism analysis. Thank you for your time and consideration throughout the review process.

Reviewer_F2cyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper targets improving training for mixed-precision analog in-memory computing (AIMC) hardware, an emerging accelerator for AI workloads. The paper shows that trainable affine parameters (immediately before the final layer) can impair analog training in a typical AMC setting but have no effect on digital training. It shows the benefit of removing or differing affine corrections in small-scale experiments.

Critical strengths

The paper provides a nice observation about the impact of trainable affine parameters immediately before the final layer: it stalls training improvement, whereas removing it, or training an affine after the analog-digital converters, allows loss for improving in the studied setup.

Critical weaknesses
  1. The paper only discusses the affine correction before the final layer, which is applied after DAC. However, there is no consideration or discussion of affine correction happening post-ADC for each MVM layer (which, for example used for compensating for ADC non-uniformity).

  2. The motivation for analog foundation training is not justified; particularly, the cost for large-scale training is not affordable (as the reported experiments are limited to very small-scale models); hence, it is not at all clear if such removing/deferring of the affine correction is the actual bottleneck and resolving it can make a significant difference in the way that the field could move forward.

  3. The PPL/CE values reported in Table 2 could be arranged more clearly to show the gain. Is there any downstream evaluation?

Questions for authors

Besides the three points mentioned in the weaknesses, it would be great to provide a fair comparison with digital low-precision and post-training quantization methods, especially since all the experiments are small-scale.

3 comments

Thank you for recognizing the observed training degradation. We clarify the affine placement and lead with the calibration, model-scale, and optimization controls that address your concerns. We have uploaded a revised manuscript: Table 2 now reports CE and accuracy together, and Appendix C.2 adds Table 10. Tables F.1–F.6 appear in this response.

W1. The paper only discusses the affine correction before the final layer. However, there is no consideration or discussion of affine correction happening post-ADC for each MVM layer.

Thank you for raising this point. The original submission already included layerwise controls comparing exposed and removed normalization affines (Section 3.2 and Appendix C.2). We have now extended these controls to compare exposed and deferred placement for both the internal normalization affines and the final readout affine. We have added this comparison to Appendix C.2, Table 10 of the revised manuscript.

Deferring the final affine improves CE whether the internal affines are exposed or deferred (Table F.1). Deferring the internal affines alone produces no clear change in CE, with paired intervals including zero. This supports our focus on the final readout even when the internal normalization affines are also deferred.

Regarding ADC non-uniformity compensation, the separate calibration experiment in Table Y.1 in our response to reviewer ypq9 also shows that removal and deferral remain beneficial after per-layer ADC gain/offset correction.

W2.1. The motivation for analog foundation training is not justified, particularly given its cost and the small scale of the experiments.

Thank you for raising this concern. We agree that large-scale analog training involves substantial costs, and its practical affordability remains an open question. AIMC offers a potential route to more efficient training; our work addresses a specific obstacle to training accuracy. However, the core scientific phenomenon investigated in this paper, the systematic performance degradation caused by learned affine parameters, is not restricted to large-scale settings. We observe it across the tested model scales, ranging from compact ViTs on CIFAR-100 and TinyImageNet-200 to Transformers up to 39.4M parameters. While prior studies have specifically examined the hardware and energy costs of analog training (Gokmen and Vlasov, 2016; Nandakumar et al., 2020), a detailed cost analysis is beyond the scope of this work.

W2.2. It is unclear whether affine placement is an actual bottleneck and whether addressing it can meaningfully advance analog training.

At 39.4M parameters, removal and deferral reduce mean CE from 5.868 to 3.645 and 2.654, respectively (ten seeds; revised Appendix H.1, Table 43). In a separate control on the primary model, a tenfold larger head learning rate retains an exposed-minus-removed gap of 0.383 CE, with a nominal paired 95% confidence interval of [0.313, 0.453] (revised Appendix G.3, Table 37). These results show a substantial, correctable loss penalty in the tested settings; they do not establish affine placement as the dominant bottleneck to scaling analog training. Neither remedy requires additional MVMs. Removal eliminates the input affine, while deferral replaces it with an output-side elementwise affine (revised Appendix G.2, Table 35).

W3.1. The PPL/CE values reported in Table 2 could be arranged more clearly to show the gain.

We have updated Table 2 in the revised manuscript to report CE and classification accuracy together in a simplified layout.

W3.2. Is there any downstream evaluation?

We added a supervised downstream evaluation by fine-tuning our enwik8-pretrained 39.4M models on TREC-6 question classification (Table F.2a). Under matched analog fine-tuning settings, validation accuracy is 16.0% / 19.4% / 24.2% for exposed / removed / deferred, and both remedies lower CE. The accuracy ordering is consistent with the pretraining CE ordering. However, none is clearly above the 23.0% majority-class baseline, and all three are well below FP32 fine-tuning (57.8%). This pilot therefore does not by itself establish a downstream benefit.

We also evaluated CIFAR-100-to-TinyImageNet transfer. This experiment shows small mean accuracy increases but no clear benefit across five seeds (Table F.2b).

Q1. It would be great to provide a fair comparison with digital low-precision and post-training quantization methods, especially since all the experiments are small-scale.

We added comparisons with digital low-precision training, varying the computation across all mapped linear layers (Table F.3) or only at the readout while keeping the trunk in FP32 (Table F.4). In the readout-only controls, exposed performs slightly better under BF16, W8A8, and W6A6, but worse under both converter STE variants. The effect is therefore not exclusive to analog hardware. Its occurrence depends on the quantization and gradient rules. When scale gradients are included across all mapped layers, exposed instead performs slightly better (Table F.3).

We also added PTQ comparisons using frozen pretrained checkpoints, with no further training (Tables F.5–F.6). Within each readout, all methods use the same checkpoint and evaluation examples, with its floating-point result as the reference. Digital W8A8/W8A6 stays close to these references, whereas calibrated converter PTQ incurs larger CE increases. Digital PTQ therefore preserves pretrained performance better than converter PTQ in the tested configurations. These comparisons assess prediction quality and do not establish relative hardware speed or energy efficiency.

Supporting experiments (Tables F.1–F.6)

Internal and final affine placement (F.1). We compare exposed and deferred normalization affines independently in the internal layers and at the final readout. The internal group contains the attention and MLP pre-normalizations in all four blocks. Exposed applies the trainable affine before the receiving analog MVM; deferred replaces it with a trainable output affine after ADC. All mapped linear layers remain analog. We train enwik8 d128-L4 for 5,000 steps with 6-bit forward/backward conversion and pulsed updates. The four configurations share common-parameter initialization, training batches, and evaluation batches within each seed.

Table F.1. Internal and final normalization affine placement.

Internal affineFinal affine exposedFinal affine deferredCE reduction from final deferral
Exposed3.547 ± 0.0533.128 ± 0.0640.419 [0.286, 0.552]
Deferred3.572 ± 0.0993.081 ± 0.0930.491 [0.391, 0.592]

Validation CE, mean ± SD over five seeds; reductions have nominal paired 95% confidence intervals. The CE reduction from internal deferral is −0.025 [−0.181, 0.132] with the final affine exposed and 0.048 [−0.054, 0.149] with it deferred.

enwik8-to-TREC-6 transfer (F.2a). Each readout starts from its corresponding enwik8-pretrained 39.4M LayerNorm model. We fine-tune all parameters on TREC-6 using the same 4,938 training examples, sample order, and three-epoch schedule, with 6-bit forward/backward conversion and pulsed updates. Six existing vocabulary tokens represent the question classes, preserving the original readout. We evaluate the final checkpoints on 500 held-out validation examples; the official test set is not used.

Table F.2a. enwik8-to-TREC-6 analog fine-tuning.

ReadoutValidation CEAccuracy (%)
Exposed2.00116.0
Removed1.69019.4
Deferred1.72024.2

One seed.

For reference, fine-tuning the exposed checkpoint in FP32 gives 57.8% accuracy. The majority-class baseline is 23.0%.

CIFAR-100-to-TinyImageNet transfer (F.2b). We transfer a digitally pretrained CIFAR-100 compact ViT (four blocks, width 128) to TinyImageNet. Within each of five paired seeds, the three readouts start from the same pretrained backbone and the same randomly initialized 200-class head. We fine-tune all parameters for 5,000 steps on 32×32 images, using 6-bit forward/9-bit backward conversion and pulsed updates. Evaluation uses all 10,000 official validation images, held out as the final test set.

Table F.2b. CIFAR-100-to-TinyImageNet analog fine-tuning.

ReadoutCETop-1 accuracy (%)
Exposed4.1213 ± 0.029610.65 ± 0.69
Removed4.1535 ± 0.017911.03 ± 0.17
Deferred4.1455 ± 0.017710.97 ± 0.33

Mean ± SD over five seeds.

Digital and analog training comparisons (F.3–F.4). Both experiments train enwik8 d128-L4 for 5,000 steps. We report mean final validation CE over five paired seeds, with matched common-parameter initialization and training batches within each experiment. The readouts differ only in the final affine configuration.

Table F.3 changes the computation in all mapped linear layers. The STE conditions simulate 6-bit input/output conversion with floating-point weights and updates; the scale-gradient variant also differentiates through scaling. The backward-MVM condition uses 6-bit conversion during backpropagation with exact weight updates. Analog uses 6-bit forward/backward conversion and programming pulses. All converter conditions use ADC range 12.

Table F.3. Affine placement when computation rules vary across all mapped linear layers.

Training configurationExposed CERemoved CEDeferred CE
FP321.55651.55841.5574
Digital STE2.22482.20252.1921
Digital STE with scale gradients2.07602.08032.0827
6-bit backward MVMs, exact updates3.31322.95492.8798
Analog3.52703.12033.1471

Table F.4 changes only the readout computation, while the FP32 trunk continues training. W8A8/W6A6 quantizes the head's weights and input activations to 8 or 6 bits, keeping accumulation and outputs in FP32. The converter conditions use the same 6-bit forward conversion as F.3, with the backward computation specified in each row. Analog uses an AIHWKit head with pulsed updates. In the last two rows, the paired 95% intervals for the exposed-minus-removed CE gap include zero ([−0.053, 0.404] and [−0.036, 0.812]), so those comparisons are inconclusive.

Table F.4. Affine placement with an FP32 trunk and different readout computations.

Readout computationExposed CERemoved CE
FP321.54931.5514
BF161.54911.5514
Digital W8A81.54931.5514
Digital W6A61.54941.5512
Converter STE1.60261.5796
Converter STE with scale gradients1.57171.5678
Converter with FP32 backward MVM1.60311.5790
Converter with 6-bit backward MVM2.14181.9664
Analog2.40412.0161

Post-training quantization comparisons (F.5–F.6). We apply each PTQ method to frozen pretrained models, without further training. Within each readout and seed, methods use the same checkpoint and evaluation examples. The readouts are pretrained separately, so each has its own floating-point reference. Results are means over five seeds.

Digital W8A8/W8A6 quantizes weights to 8 bits and input activations to 8 or 6 bits, keeping accumulation and outputs in floating point. Converter PTQ keeps floating-point weights and quantizes MVM inputs and outputs to 6 bits. Its output range is selected by CE on eight training-data batches, without updating weights or fitting affine parameters. Evaluation uses 16 batches of 128 examples; the CIFAR-100 validation split is held out from training.

Initial calibration search (F.5). All checkpoints select output range 3.

Table F.5. Post-training quantization of the same frozen model within each readout.

enwik8 · CE

Evaluation methodExposedRemovedDeferred
FP source1.56381.57501.5677
Digital W8A81.56521.57641.5693
Digital W8A61.58321.59421.5872
Converter PTQ, calibrated2.51662.47982.4612

CIFAR-100 · CE / top-1 accuracy (%)

Evaluation methodExposedRemovedDeferred
FP source2.4058 / 47.592.2451 / 46.932.2901 / 47.76
Digital W8A82.4067 / 47.502.2462 / 46.892.2904 / 47.62
Digital W8A62.4108 / 47.392.2527 / 47.012.2982 / 47.62
Converter PTQ, calibrated2.5523 / 44.922.3703 / 44.392.4149 / 45.47

Expanded calibration search (F.6). We repeat converter PTQ with a wider search grid. The selected range is 1 for enwik8 and 2 for CIFAR-100, except one deferred seed selecting 1. F.6 uses the same checkpoints as F.5 but different evaluation batches, so results should be compared with the floating-point reference in the same table. Both grids will be specified in the appendix of the final version.

Table F.6. Calibrated PTQ with an expanded search.

enwik8 · CE

Evaluation methodExposedRemovedDeferred
FP source1.55081.56201.5547
Converter PTQ, calibrated2.02772.01262.0149

CIFAR-100 · CE / top-1 accuracy (%)

Evaluation methodExposedRemovedDeferred
FP source2.4265 / 47.562.2689 / 47.562.3247 / 47.55
Converter PTQ, calibrated2.5163 / 45.422.3623 / 45.632.4246 / 46.09
Reviewer_EkpmOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the impact of trainable normalization affine parameters before the final linear layer on model performance in finite-precision training for analog in-memory computing. The authors analyze the potential training harm caused by this setting, compare several methods for adjusting the affine position or readout structure, and report experiments on language models and image classification.

Critical strengths
  1. The research question is clear and practically valuable. The paper examines the impact of the position of normalization affine parameters relative to the DAC and ADC on analog training, and the conclusions can provide a reference for readout structure design.
  2. The mathematical analysis provides an explanation for the experimental phenomena. The extensive mathematical analysis provides strong support for the paper, and this mathematical analysis is meaningful rather than merely added on.
Critical weaknesses
  1. The image classification results lack accuracy metrics. Table 2 in the main text and the corresponding compact ViT experiments in the appendix report cross-entropy for CIFAR-100 and TinyImageNet-200, but do not report the classification accuracy of these models. I personally find the currently reported CE to be relatively high, and I am therefore concerned that the models may not have been trained sufficiently. Changes in cross-entropy also do not necessarily correspond to changes in accuracy; providing complete accuracy results would help assess the actual classification performance of these experiments.
Questions for authors
  1. Could the authors provide the classification accuracies for the compact ViT experiments on CIFAR-100 and TinyImageNet-200 in Table 2 of the main text, so that they can be compared with the reported cross-entropy results?
2 comments

Thank you for recognizing the practical value of the question and the mathematical analysis. We have uploaded a revised manuscript in which Table 2 reports top-1 accuracy alongside CE for CIFAR-100, TinyImageNet-200, and CIFAR-100 QAT. Tables E.1–E.4 appear in this response.

W1.1/Q1. The image classification results lack accuracy metrics.

We have added top-1 accuracy alongside CE in revised Table 2, using the same checkpoints, evaluation sets, and five seeds as the original experiments. Removal and deferral lower mean CE and improve mean accuracy in both from-scratch classification and CIFAR-100 QAT (Tables E.1–E.2). For example, deferral improves accuracy over exposed by 2.02 percentage points on CIFAR-100 and 2.49 points on TinyImageNet-200. Reporting both metrics makes the task performance clearer, as you suggest.

W1.2. The reported CE is relatively high, raising concerns about insufficient training.

The readout comparison persists when training is extended beyond the original 15,000 steps. At 45,000 steps on both datasets and 90,000 steps on CIFAR-100, removal and deferral still have lower mean CE and higher mean accuracy than exposed (Table E.3). All three TinyImageNet readouts meet the prespecified validation-stability criterion. CIFAR-100 performs worse at 90,000 than at 45,000 steps, with decreasing training loss and rising validation CE, consistent with overfitting; the readout comparison remains unchanged.

We also tested additional data augmentation and regularization on CIFAR-100 (Table E.4). In this single-seed experiment, removal and deferral improve analog-training accuracy and CE, while exposed remains slightly better in FP32. This provides additional evidence that the analog-training benefit is not confined to the original recipe.

Supporting experiments (Tables E.1–E.4)

Tables E.1–E.3 report mean ± SD over five seeds; E.4 uses one seed. All results use final checkpoints, evaluated on the full 10,000-image CIFAR-100 official test set or TinyImageNet-200 official validation set.

Original classification experiments (E.1). The original submission reports the compact-ViT CE results in Appendix H.4 (revised Appendix H.2). E.1 adds accuracy from the same checkpoints after 15,000 steps of training from scratch, with 6-bit forward/9-bit backward conversion.

Table E.1. CE and classification accuracy of the original compact-ViT experiments.

CIFAR-100

ReadoutCETop-1 accuracy (%)
Exposed2.7748 ± 0.052428.67 ± 1.10
Removed2.7503 ± 0.045629.70 ± 0.84
Deferred2.7219 ± 0.042130.69 ± 0.97
NA-N\mathtt{NA}\text{-}\mathtt{N}2.8615 ± 0.026830.19 ± 0.60
Exposed + clamp2.7278 ± 0.051030.01 ± 1.43
NA-NA\mathtt{NA}\text{-}\mathtt{NA}2.7509 ± 0.060530.87 ± 1.25

TinyImageNet-200

ReadoutCETop-1 accuracy (%)
Exposed3.5276 ± 0.039219.61 ± 0.58
Removed3.4570 ± 0.028721.72 ± 0.65
Deferred3.4592 ± 0.013722.10 ± 0.23
NA-N\mathtt{NA}\text{-}\mathtt{N}3.6735 ± 0.019420.44 ± 0.30
Exposed + clamp3.4645 ± 0.018321.62 ± 0.29
NA-NA\mathtt{NA}\text{-}\mathtt{NA}3.5894 ± 0.030520.76 ± 0.47

Original QAT experiments (E.2). The original submission reports the CIFAR-100 QAT CE results in Appendix H.5 (revised Appendix H.3). E.2 adds accuracy from the same checkpoints. Each readout is digitally pretrained for 15,000 steps, then adapted for 5,000 steps with quantized forward computation and exact weight updates.

Table E.2. CE and classification accuracy after the original CIFAR-100 QAT procedure.

ReadoutCETop-1 accuracy (%)
Exposed2.2241 ± 0.024742.43 ± 0.31
Removed2.1408 ± 0.023744.18 ± 0.50
Deferred2.1710 ± 0.023743.87 ± 0.53
NA-N\mathtt{NA}\text{-}\mathtt{N}2.3544 ± 0.019040.58 ± 0.60

Longer training (E.3). We retain E.1's models, data splits, analog settings, and five seeds, and train from scratch for 45,000 steps. A prespecified validation-stability rule triggered additional 90,000-step CIFAR-100 runs, also from scratch. Within each budget, all readouts use the same learning-rate schedule, with cosine decay over the full training horizon.

We assess stability by comparing validation averages over the final two 5,000-step windows. Across five seeds, the absolute mean changes must not exceed 0.01 CE and 0.5 percentage points in accuracy; at least four seeds must also remain within 0.02 CE and 1 percentage point individually. We will include the validation curves in the final version.

Table E.3. CE and classification accuracy after longer training.

CIFAR-100 — 45,000 steps

ReadoutCETop-1 accuracy (%)
Exposed2.6852 ± 0.087437.39 ± 1.54
Removed2.4105 ± 0.072740.54 ± 1.40
Deferred2.4618 ± 0.054040.39 ± 1.11

CIFAR-100 — 90,000 steps

ReadoutCETop-1 accuracy (%)
Exposed4.1592 ± 0.274635.36 ± 1.66
Removed2.9770 ± 0.168239.98 ± 1.54
Deferred3.3934 ± 0.152139.32 ± 1.13

TinyImageNet-200 — 45,000 steps

ReadoutCETop-1 accuracy (%)
Exposed3.1596 ± 0.023927.70 ± 0.33
Removed2.9236 ± 0.026432.35 ± 0.63
Deferred2.9433 ± 0.018332.33 ± 0.50

Additional augmentation and regularization (E.4). We compare all three readouts under digital FP32 and analog training, retaining the compact-ViT architecture and analog configuration. We add random cropping, Mixup/CutMix, label smoothing, and dropout to the original recipe. All six configurations train from scratch for 45,000 steps with matched common-weight initialization and training data streams.

Table E.4. CIFAR-100 results with additional augmentation and regularization.

ReadoutDigital FP32 CE ↓Digital FP32 accuracy (%) ↑Analog CE ↓Analog accuracy (%) ↑
Exposed1.50761.642.42836.61
Removed1.51061.372.36939.78
Deferred1.50961.282.36741.73

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

Thank you for your time and consideration throughout the review process. As the discussion period draws to a close, we provide this final summary of the main questions raised by the reviewers, the evidence provided in our responses, and the corresponding manuscript revisions.

R1. Significance and model scale (Reviewer F2cy W2 and follow-up). Our contribution is the identification and explanation of a training failure caused by a standard normalization affine that remains digital and unquantized. Head-replay controls and gain-gradient interventions provide evidence for damage accumulated during training and the role of misleading gain feedback (Sections 3–5). At 39.4M parameters, removal and deferral reduce mean CE from 5.868 to 3.645 and 2.654 across ten seeds, without additional MVMs (Table 2). The importance of this error relative to other bottlenecks at larger scales remains open.

R2. Layerwise affine placement and device non-idealities (Reviewers F2cy W1 and ypq9 W1/Q1). We added controls that independently defer internal and final normalization affines. Deferring the final affine improves CE under either internal placement, supporting the focus on the readout (Appendix C.2, Table 11). Separate experiments show that removal and deferral remain beneficial after per-layer ADC gain/offset calibration. We also report training-time drift results, whose paired comparisons support different remedies under different uniform-drift conditions and remain inconclusive under heterogeneous drift (Table 3; Appendix H.5). Physical-hardware validation remains open.

R3. Classification accuracy and training adequacy (Reviewer Ekpm W1/Q1). Revised Table 2 reports top-1 accuracy alongside CE from the original checkpoints. We also extended training to 45,000 steps on CIFAR-100 and TinyImageNet, and to 90,000 steps on CIFAR-100. Both repairs retain lower mean CE and higher mean accuracy than exposed under these budgets (response Table E.3).

R4. Alternative scaling and digital comparisons (Reviewers ypq9 W2/Q2 and F2cy Q1). Average-max scaling reproduces the degradation across five paired seeds; the static-range comparison remains inconclusive (Section 7; Appendix D.3). Digital controls show that the effect can also occur with digitally implemented converter quantization and depends on the quantization and gradient rules. Matched-checkpoint PTQ comparisons are reported separately (response Tables F.3–F.6). These controls do not establish generality across different AIMC architectures.

R5. Downstream evaluation (Reviewer F2cy W3 and follow-up). We added visual transfer and TREC-6 evaluations. We agree that the visual gains are inconclusive and the TREC-6 accuracy is too low to establish practical downstream gains. We retain these results and their limitations in our response; our mechanistic conclusions rest on the controlled localization and intervention experiments.

R6. Main-text experimental coverage and presentation (Reviewers ypq9 follow-up and F2cy W3). We expanded Section 7 with alternative-scaling results and a new Table 3 covering ADC non-uniformity and device drift. Table 2 now presents CE and classification accuracy together. These revisions preserve the mechanism analysis while keeping the main text within ten pages.

Taken together, the theoretical analysis and controlled experiments identify an avoidable source of analog-training error, explain how it arises, and support remedies that require no additional MVMs. We respectfully ask that your final assessment consider this mechanistic contribution and the supporting evidence alongside the stated limits on downstream utility and larger-scale hardware validation.

Thank you again for your time and consideration.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.