acceptodds
Under review as a conference paper at ICLR 2027

AAAE: Application Accuracy-Aware Evaluation Framework for Approximate Computing

Abstract

Approximate computing offers an effective approach to reducing the hardware cost of neural inference, but efficiently characterizing model-specific error tolerance across large approximate multiplier design spaces remains challenging. We present AAAE, an application accuracy-aware evaluation framework that guides approximate multiplier design under explicit accuracy constraints. AAAE combines multi-scale and model-conditioned error features, augmented with lightweight inference responses for Transformers, to predict the accuracy degradation of previously unseen multiplier designs. In parallel, a graph neural network estimates hardware costs from multiplier structures. These predictions, together with empirically calibrated accuracy bounds, guide design optimization, with selected designs verified through inference and high-fidelity hardware evaluation. Experiments on three vision Transformers and two CNNs demonstrate Top-1 accuracy-drop prediction mean absolute errors of 0.43–0.59 percentage points. The hardware surrogate achieves a power–delay product prediction mean relative error of 0.241%. In the reported timing setting, surrogate evaluation provides a 35.1× per-design speedup over 1,000-image accuracy evaluation plus high-fidelity hardware evaluation. Under a two-percentage-point accuracy-loss budget, the calibrated bound A1 achieves an empirical false-discovery proportion of at most 1% across all five models. Our implementation is publicly available at https://anonymous.4open.science/r/AAAE-7CB0/.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Approximate Computing, Error Tolerance, Hardware-Efficient Neural Inference, Accuracy Prediction, Design Space Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qBAuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents AAAE, an application accuracy-aware evaluation framework for exploring the design space of approximate multipliers for energy-efficient Transformer and CNN inference. The framework is mainly composed of four parts: (1) approximate multiplier generation and error characterization; (2) application-level accuracy prediction; (3) hardware cost prediction; and (4) design optimization. For approximate multiplier generation, the work focuses on a compressor-based approximate 8-bit multiplier design in which the least significant nine 4:2 compressors are approximated using nine different approximate compressors proposed in the literature. For accuracy prediction, the work proposed a predictor that is based on three different types of features: multi-scale error features computed from error maps of approximate multipliers, model-conditioned error features computed from the error map and profiled input distribution at different locations in the network, and inference response features computed by running approximate inference using a small set of input images. The hardware cost predictor uses a GNN that leverages compressor hardware characteristics to predict the hardware characteristics of an approximate multiplier design. The optimization stage then searches for Pareto-optimal configurations.

Critical strengths

• A novel accuracy predictor design based on error properties of the multiplier, characteristics of input data, and inference response with approximations in place using a small set of inputs. • Targets the emerging area of using approximate hardware components for improving the energy-efficiency of Transformer and CNN inference

Critical weaknesses

• The method uses the same approximate multiplier across the entire network, so layer-wise approximation selection is not possible. • The work targets compressor-based 8-bit approximate multipliers despite mentioning that "the design space of approximate multipliers is extremely large, involving diverse architectural choices such as bit-width selection, truncation schemes, compressor structures, partial-product reduction, sign processing, and error compensation." • The approximation is limited to the least significant nine 4:2 compressors. Additionally, there is no detailed and convincing justification provided for this constraint specifically when the core purpose of the search is to find Preto optimal configurations from the complete design space. • The distribution profiles generated using accurate simulations are used for model-conditioned error estimation. The distribution drifts that occur due to approximations in the previous stages are not considered at all. Specifically, this becomes an issue when the design space includes aggressive approximations as well, which this work seems to exclude by limiting approximations to the least significant nine 4:2 compressors. • The literature has already widely explored GNN-based hardware characteristics estimation. • Considering that the level of approximation has been significantly constrained, the error metrics presented are insufficient to quantify the performance of the predictors because of the lack of diversity in the overall design space.

Questions for authors

Besides addressing the above mentioned weaknesses, address the following: • Can the proposed technique be used for aggressive approximations? The current approximation space considered is quite limited and does not cover aggressive approximations. • Is the high accuracy of the proposed predictors related to the lack of diversity and aggressiveness in the approximation space? • How generalizable is the proposed technique to a broader design space of approximate multipliers, beyond a small set of compressor-based approximate multipliers? • For aggressive approximations, where intermediate distributions are significantly affected by approximations in earlier stages, how does the overall performance of the proposed framework change? Additionally, how would the contribution of each feature set change for generating an accurate accuracy prediction?

1 comment
  1. High-Order Approximation Restrictions and Design Space Selection AAAE currently employs 8-bit approximate multipliers based on 4:2 compressors, generating designs with different arithmetic error characteristics and hardware costs by configuring different compressor types. To evaluate the impact of high-order approximation, we further include the eight previously exact high-order 4:2 compressors in the approximate design space. Combined with the nine low-order compressor positions, we generate 1,000 distinct configurations and evaluate their application-level accuracy on five INT8 vision models. Experimental results show that over 98% of the configurations cause severe accuracy degradation, with the approximate models' Top-1 accuracy dropping to 0.17%–1.11%. Since high-order partial-product columns carry larger numerical weights, introducing approximation at these positions generally has a more significant impact on the final computation results. Therefore, we keep the high-order compressors exact and restrict structural exploration to the nine low-order compressor positions, focusing the design space on configurations that offer meaningful trade-offs between application accuracy and hardware cost.
  2. Uniform Approximate Multiplier Across Different Network Layers AAAE currently employs the same approximate multiplier at the specified computation sites of a target neural network, while its application-level accuracy prediction accounts for the error sensitivities of different computation sites. Specifically, we construct model-conditioned error features using multiplication input distributions, local decision sensitivities, network depth, and computation roles. For Transformers, we separately consider the Query, Key, and Value projections, the attention output projection, and the linear layers in the feed-forward network to characterize how approximation errors at different computation sites affect the final classification results. Thus, AAAE does not assume uniform error tolerance across network layers; instead, it incorporates site-specific error sensitivity information into accuracy prediction. The current framework does not yet assign different approximate multipliers to individual network layers. Supporting layer-wise heterogeneous designs would require extending the design variables from a single multiplier configuration to joint configurations across multiple layers, together with a prediction model capable of capturing accumulated errors and interactions among different approximation configurations. This remains a limitation of the current framework and a direction for future work.
  3. Applicability to Aggressive Approximation and Different Multiplier Architectures AAAE's application-level accuracy analysis is not inherently restricted to a specific 4:2 compressor topology. Instead, it constructs an error surface from the complete input-output mapping of each approximate multiplier. Based on this representation, multi-scale error features characterize error magnitudes, signed biases, and input-dependent error distributions, while model-conditioned features incorporate the computational characteristics of the target neural network to predict their impact on application-level accuracy. Therefore, for other approximate multipliers with the same INT8 input-output definitions, including designs based on different compressor structures, truncation, or other approximation strategies, the same error-characterization pipeline can be applied as long as their complete input-output mappings are available. This allows the error-characterization approach to extend beyond the current nine low-order compressor configurations.
Reviewer_BPxLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an Application Accuracy-Aware Evaluation Framework AAAE (Application Accuracy-aware Evaluation Framework) for approximate computing It aims to solve the problems of large design space for approximate multipliers, unpredictable accuracy loss and high hardware evaluation cost in neural network reasoning. This framework first generates 8-bit approximate multipliers of different structures and analyzes their complete computational errors to extract multi-scale error features. At the same time, it combines the input data distribution of the target neural network, computational position sensitivity, and a small amount of actual reasoning responses. Use LightGBM to predict the influence of different multipliers on the Top-1 Accuracy of the model. Furthermore, the author represents the multiplier circuit structure as a Graph and uses Graph Neural Network (GNN) to predict its Power consumption, Delay, area and power-delay Product (PDP), thereby avoiding time-consuming hardware synthesis and simulation for all candidate designs. On this basis, AAAE combines empirical accuracy boundary calibration, Bayesian Optimization and NSGA-II to automatically search for multiplier designs with higher hardware efficiency under the condition of meeting the specified accuracy loss constraints. The experiment covered three Vision Transformers and two CNN models. The results showed that the average absolute error of the accuracy loss prediction of this framework was only 0.43-0.59 percentage points, and the average relative error of PDP prediction was 0.241%. The overall single design evaluation speed has increased by 35.1 times compared to the complete reasoning and hardware evaluation.

Critical strengths

Strength 1: This paper addresses the problems of large hardware design space and high cost of accuracy evaluation at the application layer in approximate computing, and proposes a unified evaluation and optimization framework, which has clear application value in resource-constrained AI reasoning systems. Strength 2: The error modeling based on application perception has a certain degree of innovation. AAAE not only utilizes the multi-scale error characteristics of approximate multipliers, but also combines the input distribution of the target neural network, computational position sensitivity, and lightweight reasoning response, enabling a more comprehensive description of the impact of arithmetic errors on the final accuracy of the model. Strength 3: The joint optimization framework combining software and hardware is relatively complete. This paper integrates the generation of approximate multipliers, Accuracy Drop Prediction, GNN-based PPA Prediction, accuracy constraint optimization and final hardware verification, forming a relatively complete process of automated design space exploration.

Critical weaknesses

Weakness 1: The innovation of the method is relatively limited. This paper mainly integrates the existing technologies such as Multi-scale Error Features, LightGBM, GraphSAGE, Bayesian Optimization and NSGA-II. Although the overall framework is relatively complete, it lacks breakthrough algorithmic innovations. In addition, the unique contributions of the paper to each module compared with the existing approximate computational design exploration methods still need to be further clarified. Weakness 2: The space for hardware design is rather limited. The experiment only considers the 8-bit approximate multiplier of the fixed topology and generates candidate designs by replacing nine low-bit Compressor positions. It has not yet been verified whether this framework can be extended to other bit widths, different multiplier architectures or other approximate computing units, so its universality remains questionable. Weakness 3: The scale of accuracy assessment is limited. The author only used a fixed 1,000 images to evaluate the accuracy loss of the candidate multipliers. Although this setting reduces the experimental cost, it may not fully reflect the actual performance on the complete test set, especially when the accuracy loss budget is only 0.5 to 2 percentage points. The limited samples may affect the reliability of the design selection. Weakness 4: There is a lack of system-level assessment in hardware efficiency verification. Although the PPA prediction accuracy of GNN is relatively high, the hardware verification is mainly based on the synthesis of individual multipliers, gator-level simulation and timing analysis, lacking a system-level evaluation of a complete AI Accelerator. Therefore, the paper has not yet demonstrated that the PDP improvement at the multiplier level can be directly translated into power consumption, latency or energy efficiency benefits of actual neural network accelerators.

Questions for authors

AAAE integrates several established techniques, including LightGBM, GraphSAGE, Bayesian optimization, and NSGA-II. Could the authors clarify which components constitute the primary methodological innovations beyond their integration? In particular, how does the proposed framework fundamentally differ from existing learning-based approximate computing and hardware design-space exploration approaches?

1 comment
  1. Design Space Scope and Applicability Across Multiplier Architectures This work uses 8-bit multipliers as the experimental target and constructs the design space using nine configurable low-order 4:2 compressor positions, each with eight approximate compressor options, resulting in (8^9) nominal structural configurations. This setup generates diverse error distributions and hardware costs within a consistent multiplier architecture, providing a large yet controlled design space for investigating model-specific accuracy prediction. Different components of AAAE have different scopes of applicability. Its application-level error characterization is based on the complete input-output lookup table (LUT) of each approximate multiplier and is therefore not restricted to a specific 4:2 compressor topology. For other approximate multipliers with the same INT8 input-output definitions, the corresponding error surfaces can be constructed and multi-scale error features extracted.
  2. Reliability of Accuracy Evaluation Using 1,000 Images Top-1 accuracy measured on 1,000 images has a statistical resolution of 0.1%. For designs close to strict accuracy-constraint boundaries, the limited number of evaluation images may introduce uncertainty in feasibility assessment. For applications requiring strict accuracy guarantees, a larger or independent image set can be used to verify the final designs.
  3. System-Level Hardware Benefits We further developed an on-chip energy model of a 64×64 INT8 systolic array and combined it with execution statistics from SCALE-Sim v3 to evaluate the system-level benefits of the optimized approximate multipliers. Across five models, the approximate multipliers selected under a 5% Top-1 accuracy-drop budget were verified through full-model inference and RTL-based hardware evaluation. Compared with the exact INT8 multiplier, the selected designs achieve 41.2%–42.9% lower delay, 17.4%–20.9% lower area, 16.2%–24.0% lower power, and 51.2%–55.8% lower power–delay product (PDP). With the systolic array architecture and dataflow unchanged, the resulting on-chip energy reductions are 8.6%, 10.4%, 10.8%, 10.6%, and 9.0% for Swin-Tiny, ViT-Base, DeiT-Base, ResNet-50, and MobileNetV3, respectively.
Reviewer_PLadOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a technique to make the design space exploration of approximate hardware multipliers more efficient. Instead of performing end-to-end inference to measure accuracy and hardware synthesis runs to obtain the PPA (Performance, Power, and Area) metrics for each approximate design variant, the paper proposes using a model that could predict the accuracy drop and PPA for different designs, and validate only the selected design. The results are presented on CNN and Transformer models.

Critical strengths

This paper use the approximate computing paradigm to trade-off accuracy for energy efficiency. It proposes a practical solution by using trained models to predict the inference accuracy drop and PPA metrics for different approximate multiplier designs. This can be useful if the design space is large and we need the approximate design that provides the best trade-off between accuracy and energy efficiency.

Critical weaknesses
  • Lack of experiments that motivate the problem

    • Any special- and general-purpose machine learning hardware has computational units, multiple levels of on-chip cache or scratchpad, cache controller or DMA, and off-chip DRAM. Hence, the total energy consumption depends on the energy incurred by all these hardware modules. The paper optimizes the energy of the compute cores by using approximate multiplier design. Before presenting the solution, there needs to be clear experiments demonstrating how much of end-to-end energy efficiency can be obtained by the approximate multipliers for the CNN and Transformer models reported in this work.
    • In fact, even with the compute cores, there are local registers, buffers, and large bit-width accumulators, which consume substantial power compared to the multiplier. Hence, a clear pie-chart indicating the power consumed by the individual components of the compute cores is essential to understand the benefits of the approximate multiplier design.
  • Lack of positioning for a machine learning conference

    • Works of this nature need a clear description of the underlying hardware under consideration. Typical work in this area would build a hardware architectural simulator to estimate the end-to-end energy and latency/throughput metrics for different design optimizations across a suite of benchmark models. Presenting only the design space exploration methodology without clear motivational experiments and hardware architecture makes it extremely hard to evaluate the utility of the work. Hence, empirical works of this nature need to have substantial systems component and is better suited for systems conferences such as MLSys, DATE, DAC, MICRO, etc.
  • Insufficient system-level metrics to justify the approximate design

    • PPA metrics for the approximate multiplier alone is insufficient to justify the design choice. Rather, end-to-end energy numbers for the compute cores need to be reported for different benchmark models.
  • Unclear presentation of the experimental results

    • No explanation or insights are provides for results reported in Tables 1, 2, and 3. It is a good practice to present the key take-aways for results reported in the Tables rather than expecting the readers to interpret the data.
    • Absolute accuracy results are not reported. Accuracy drop metrics are meaningful only if the baseline accuracy is closer to the published state-of-the-art.
    • It is also unclear which dataset was used for the various models reported in this work.
  • Accuracy drop mentioned as acceptable in the paper is not practical ("the efficient identification of designs that simultaneously satisfy application-level accuracy constraints, such as a 5% accuracy-loss budget for image classification (Ponzina et al., 2022) or a 2% relative perplexity-increase budget for language modeling")

    • Accuracy degradation of 5% is unacceptable. Most works on model or system-level optimizations target 0.5-1% accuracy drop.
  • Certain text in the manuscript is not understandable

    • "Under a two-percentage-point accuracy-loss budget, the calibrated bound A1 achieves an empirical false-discovery proportion of at most 1% across all five models." What is calibrated bound A1?
    • "The Response-only regressor shares AAAE’s 50 response images and response features, so it covers the Transformers only;"
    • "ViT-Base and DeiT-Base form sealed evaluations, Swin-Tiny a postinspection promotion result, and ResNet-50 and MobileNetV3-Large post-failure audits."
    • "Appendix E illustrates post hoc candidate pairs that conventional metrics match closely yet whose measured drops diverge."
Questions for authors
  • Instead of using approximate INT8 multipliers, how does the accuracy-energy trade-off change if one were to use exact INT4 or hybrid INT4/INT8 multipliers?
  • The reviewer believes that state-of-the-art NAS frameworks can be adopted for identifying the optimal approximate designs. What is the benefit of the proposed method compared to hardware-aware NAS?
1 comment
  1. Hardware Design Motivation and System-Level Benefits To quantify the impact of approximate multiplier optimization on the overall computing system, we conducted an additional energy analysis of a 64×64 INT8 systolic array using SCALE-Sim v3. The architecture comprises 4,096 processing elements (PEs), and the component-level energy model accounts for multipliers, accumulators, registers, and on-chip SRAM. Under the Swin-Tiny workload, 8-bit multiplier cores account for 45.61% of the modeled on-chip energy consumption, indicating that multipliers offer substantial opportunities for hardware optimization. The on-chip energy analysis includes the PE array and on-chip SRAM, while excluding off-chip DRAM.
  2. Application-Level Accuracy Evaluation and Experimental Results We evaluate five fixed-weight INT8 vision models: Swin-Tiny, ViT-Base, DeiT-Base, ResNet-50, and MobileNetV3-Large. All accuracy drops are calculated as the differences in Top-1 accuracy between the exact INT8 models and their approximate counterparts, evaluated on the same ImageNet subset. The baseline Top-1 accuracies of the five models are 80.60%, 84.70%, 81.30%, 80.40%, and 72.20%, respectively. We evaluate 20,000 approximate multiplier configurations, divided into 14,000 training, 3,000 validation, and 3,000 testing configurations. AAAE achieves mean absolute errors (MAEs) of 0.432%–0.586% in accuracy-drop prediction across the five models. The MAE values in the tables represent absolute prediction errors on the Top-1 accuracy percentage scale rather than relative percentage errors. We additionally report the coefficient of determination (R²), the 95th-percentile absolute error, and the maximum absolute error to characterize prediction quality and error distributions.
  3. Accuracy Constraint Settings AAAE treats the allowable Top-1 accuracy drop as a configurable application-level constraint, since different application scenarios have different tolerances for accuracy degradation. Existing approximate computing studies have adopted accuracy-loss budgets ranging from 0.5%–1% to approximately 5%.
  4. Calibrated Bounds and Terminology in the Original Submission In the original submission, A1 denotes an accuracy-drop upper-bound rule constructed using independent calibration data. It is designed to reduce the risk of incorrectly admitting infeasible designs during accuracy-constrained screening. The corresponding false-discovery proportion measures the fraction of designs predicted to satisfy the accuracy constraint that actually violate it. This metric evaluates a different aspect of prediction performance from MAE. Regarding the terms Response-only and post-failure audits in the original submission, Response-only is a Transformer prediction baseline that uses only lightweight inference responses. It employs the same 50 response images and response features as the corresponding component of AAAE. CNNs follow a separate prediction path and therefore do not use this baseline. The design pairs with similar arithmetic errors but different application-level accuracy drops, presented in the original appendix, illustrate why arithmetic error metrics alone are insufficient to characterize model-level accuracy degradation.
  5. Differences from INT4, Mixed-Precision Quantization, and Hardware-Aware NAS INT4 and mixed-precision quantization primarily reduce computational costs by adjusting numerical representation precision. In contrast, AAAE optimizes the internal circuit structures and error characteristics of approximate multipliers while keeping the target models' INT8 quantization configurations and weights unchanged. Hardware-aware neural architecture search (NAS) primarily explores neural network architectures and associated hardware configurations, whereas AAAE explores approximate multiplier circuit structures for a given neural network and predicts the resulting model-level accuracy degradation. These approaches operate at different levels of the design process and can potentially be combined. The primary contribution of this work is to incorporate application-level accuracy constraints directly into circuit design space exploration, rather than relying solely on arithmetic error metrics such as MRED and NMED to evaluate approximate designs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.