- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses a drifted analog accelerator after deployment with forward-only optimization recovery, that costs queries to update its parameters and to choose those updates. Specifically, it proposes a budget-aware step-size selection for forward-only recovery of degraded analog accelerators due to costly forward evaluations, including configuration search, calibration, validation, and recovery updates. The targeted research question here is to answer what accuracy-cost trade-off the shared calibration can achieve. To do this, this paper first investigates the forward-only recovery and its costs by training a network offline, simulating degradation, and adapting the parameters. Then, it employs a recovery step that can provide meaningful progress without accumulating excessive noise over the run. It leverages a two-point zeroth-order (ZO) optimization to estimate a gradient from forward evaluations, derives a finite-horizon bound on parameter displacement, and converts the scale into a donor-calibrated rule. In each run, the final parameter state replaces the drifted baseline only if it improves mean accuracy over five independent noisy validations; otherwise, the policy returns the baseline. All evaluations are based on simulations on an MNIST classification task and analog devices (e.g., ReRAM).
Critical strengths
The configuration cost is treated as part of the optimization problem, by considering forward evaluations used for calibration, candidate search, recovery, monitoring, and validation. It is useful as it shows the overhead of recovery algorithms for identifying the appropriate configuration.
Critical weaknesses
• The problem statement, motivation, and scope of study (e.g., the types of analog devices) are not well established in the beginning of the introduction section. Instead, the paper directly mentions the research question on accuracy-cost trade-off while skipping essential definitions of several techniques/terminologies (e.g., recovery, shared calibration, donor, etc.), and why they matter in a chain of arguments. This writing style makes it difficult to follow the discussion in the early part of the paper. • The paper studies devices that expose writable parameters and forward scalar-loss evaluations, but no backward pass for computing gradients. However, the scope of analog accelerators should be defined more explicitly. For instance, the accelerators are targeted for neural network workloads and the analog devices technologies such as non-volatile memories (NVMs). • All evaluations are based on simulations on a small MNIST classification task, limited types of analog devices (e.g., ReRAM) and analog crossbar architectures, and non-practical network model architectures. This condition is insufficient to draw conclusions about the effectiveness of the proposed techniques. Moreover, the details of simulators, analog crossbar architectures, analog device models, and device drift models are not clearly described and characterized. These are important to understand the experimental setup from hardware perspective, and how it is connected with the overall targeted evaluations.
Questions for authors
Besides addressing the above mentioned weaknesses, address the following: • It is suggested to provide evaluations based on bigger/practical datasets (e.g., ImageNet-1K), several analog devices (e.g., ReRAM, PCM, MRAM, and optical), practical analog crossbar architectures, and practical network model architectures (e.g., VGG-16, ResNets, etc.). • It is suggested to clearly describe and characterize the details of simulators, analog device models, and device drift models, because they are important to understand the experimental setup, especially from hardware perspective, and how it is connected with the overall evaluations. • It is also suggested to clarify the generality and scalability of the proposed solution across different design choices, including crossbar architectures, analog devices technologies (e.g., ReRAM, PCM, MRAM, and optical), network models, and larger datasets (e.g., ImageNet-1K and CIFAR-100). • The costs should also be presented in terms of latency, memory, and power/energy overheads to clearly show the practical requirements to conduct the proposed solution.