Convolutional Neural Network Can Satisfy Representation and Interpretability without Nonlinear Activation Functions Simultaneously

Nonlinear activation functions enable the selection of features and achieve strong representation when combined with various feature mappings. However, in this paper, we find that for general visual recognition tasks, such as classification, nonlinear activation functions may actually constrain the interpretability of the model's intermediate and final outputs. We observe that nonlinear activations, particularly ReLU, discard negative responses, thereby disrupting the model’s semantic attribution and potentially leading to unreasonable output attributions. Moreover, models whose feature mappings can be represented by univariate polynomials, such as conventional convolutional neural networks, inherently possess limited feature discriminability and thus lower representational capacity. This observation motivates a rethinking of effective feature representation and selection methods without relying on nonlinear activation functions. We demonstrate that simple element-wise multiplication of feature responses can achieve complete semantic attribution and enhance feature discriminability. Building on this, we further optimize the approach by extending element-wise multiplication from single-head homogeneous inputs to a multi-head heterogeneous inputs scheme, which improves the model’s representational capacity and reduces inference latency. Experiments on ImageNet and visualizations of feature outputs show that NANet is the first to surpass the representational capability of activation-based models while also improving the interpretability of model.