acceptodds
Withdrawn from ICLR 2027

Convolutional Neural Network Can Satisfy Representation and Interpretability without Nonlinear Activation Functions Simultaneously

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Nonlinear activation functions enable the selection of features and achieve strong representation when combined with various feature mappings. However, in this paper, we find that for general visual recognition tasks, such as classification, nonlinear activation functions may actually constrain the interpretability of the model's intermediate and final outputs. We observe that nonlinear activations, particularly ReLU, discard negative responses, thereby disrupting the model’s semantic attribution and potentially leading to unreasonable output attributions. Moreover, models whose feature mappings can be represented by univariate polynomials, such as conventional convolutional neural networks, inherently possess limited feature discriminability and thus lower representational capacity. This observation motivates a rethinking of effective feature representation and selection methods without relying on nonlinear activation functions. We demonstrate that simple element-wise multiplication of feature responses can achieve complete semantic attribution and enhance feature discriminability. Building on this, we further optimize the approach by extending element-wise multiplication from single-head homogeneous inputs to a multi-head heterogeneous inputs scheme, which improves the model’s representational capacity and reduces inference latency. Experiments on ImageNet and visualizations of feature outputs show that NANet is the first to surpass the representational capability of activation-based models while also improving the interpretability of model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
nonlinear activation functions, effective activation-free representation, complete semantic attribution, improved interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Gi6pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors observes that ReLU activation functions discard negative responses in the neural network, making the semantic attribution of input features complicated and incomplete. This paper propose to replace nonlinear activation functions with element-wise multiplication. With experiments, they demonstrate that simple element-wise multiplication of feature responses can achieve competitive performance while maintaining good semantic attribution and hence the interpretability of the model.

Critical strengths
  • The paper shows that a ConvNet without any pointwise nonlinear activation can reach competitive ImageNet-1K accuracy from 3M to 50M parameters. Its only nonlinearities are element-wise multiplication and normalization layers.
  • The paper documents the full trajectory from ConvNeXt to NANet in a step-by-step roadmap. Each step is trained independently, averaged over three seeds, and evaluated on both accuracy and real iPhone 13 latency.
Critical weaknesses
  • The paper has severe factual incorrectness. Not all the nonlinear activation functions suppress negative responses, for example, leaky ReLU and tanh. The claim in the paper, that nonlinear activations discard negative responses, is not correct. The paper also critisizes ReLU for complicating model intepretation. In fact, ReLU activatino actually plays an important role in interpretable machine learning area. The foundamental paper "Layer-Wise Relevance Propagation" (LRP) highly relies on the interpretable property of ReLU activation. [1]
  • The novelty of the paper is limited. Almost the same idea has already been proposed in NAFNet [2], while the authors failed to clearly explain their difference compared with this prior work. Increamental contributino is the claimed improved interpretability from element-wise multiplication. However, improved interpretability is only supported by qualitative visual comparison on feature maps, which is quite subjective, and personally I feel FocalNet actually provides more clear contour. Also, the visualization of interpretability is only limited to a few samples, some samples in Appendix are also repeated in the main paper. More diversed sample visualizations are needed to support the claim.
  • The use of "univariate polynomial" for ConvNets and "bivariate polynomial" for ConvNets with element-wise multiplication is inaccurate and misleading. A convolution is a multivariate degree-1 polynomial, and the product of two branches is a multivariate degree-2 polynomial (a quadratic form) in the input. ReLU/GELU ConvNets are not polynomials at all. The relevant distinction is polynomial degree, not the number of variables, so the argument that "univariate polynomial" models have limited discriminability is built on a misnomer.

Reference

[1] Bach, Sebastian, et al. "On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation." PloS one 10.7 (2015): e0130140.

[2] Chen, Liangyu, et al. "Simple baselines for image restoration." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

Questions for authors
  • missing related works: Bilinear MLPs [3]
  • In Figure 4, how does multi-head further reduce the latency? Intuitively, more heads will cost more computation.
  • The FocalNet heatmaps in Figures 5 and 7 exhibit inverted polarity (dark objects, bright backgrounds), which contradicts the model's high accuracy. This likely occurs because FocalNet's final features have arbitrary signs (my hypothesis) and strongly negative activations may actually encode critical object evidence that the classifier weights map to positive logits. Consequently, the current direct visualization misrepresents feature importance, making the qualitative comparison favoring NANet unreliable. The authors should: (1) Specify the reduction method: clarify exactly how the C×H×WC \times H \times W tensor is reduced to a 2D map (e.g., mean, norm, or max). (2) Use sign-aware mapping: implement a class-specific visualization method, such as weighting feature channels by the classifier weights of the predicted class. (3) Provide quantitative evidence: support the interpretability claims with quantitative metrics rather than relying solely on subjective visual inspection.
  • In Section 5.0.1, the authors mentioned that adding ReLU after element-wise multiplication provides almost no performance improvement, which seems quite unsurprising. In the end, the element-wise multiplication is also a kind of nonlinear operation, so the ultimate effects of traditional non-linear activation functions and element-wise multiplications are the same: increasing the expressive capacity of the neural network.
  • The paper inaccurately frames nonlinear activation functions (e.g., ReLU, GELU) as causing information loss by "discarding negative responses". In reality, variants like Leaky ReLU, ELU, and GELU are primarily designed for optimization and regularization, such as maintaining gradient flow. The authors should revise their motivation to analyze training dynamics and gradient flow rather than attributing differences solely to information loss. Furthermore, the paper's own ablation (Table 4) is consistent with this optimization perspective, but inconsistent with the author's explanation. Adding ReLU after multiplication gives no gain (74.80%), while GELU improves accuracy (75.56%). Yet ReLU suppresses negative responses even more strongly than GELU, so the authors' explanation that GELU " improves feature discrimination by suppressesing strongly conflicting negative responses" (L421-L423) cannot account for the difference.

Reference

[3] Pearce, Michael T., et al. "Bilinear MLPs enable weight-based mechanistic interpretability." arXiv preprint arXiv:2410.08417 (2024).

Reviewer_tVi6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a cross-multiplication architecture as an alternative to ReLU activation. The resulting model is named NaNet. Experiments focus exclusively with a rework of CovNext, reworking the Conv/ReLU blocks to implement the cross-multiplication functions. Notes claiming improved latency, performance, and interpretability are made with some experiments and figures provided. Justifications are given that the improved interpretability is due to not discarding negative values in ReLU.

Critical strengths

Experimental results comparing their model to previous models appear strong.

Critical weaknesses
  • A combination of features necessary for the ReLu replacement (call them type 1 features) and not necessary for it (type 2 features) are made. Figure 4 indicates that type 1 features are a cause of extreme decrease of inference latency. Information in the appendix reveals that type 2 features are the cause of decreased latency. This is misleading, attributing to novel features what are actually due to common architectural changes.
  • A ConvNext model with only type-2 changes is never made as far as the reviewer can tell, so determining if the type 1 features really caused an increase in performance is not possible.
  • Claims of improved interpretability are not convincing to this reviewer. The chosen internal activation maps do not appear much more interpretable or preferable for NaNet.
Questions for authors

Please respond to the above, as well as the following more minor critiques:

  • The claims about improved interpretability and discrimination sound like heuristic-based hypotheses, but are claimed as a thesis without rigorous support. The experiments do not provide the evidence needed to substantiate these specific hypotheses.
  • Much is made of ReLU throwing away negative values, but the paper focuses on ConvNext, which used GELU, which does not zero-out negative values. The message seems cross-purpose.
  • Exact implementation of the cross-multiplation block is a bit inexact in exhibition. I would recommend making it even more explicit.
  • Not sure why figure 3 has the arrow reversed from down to up.
  • It is written "FLOPS", not "FLOPs".
Reviewer_jCXHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to maintain feature representation capacity and improve interpretability in convolutional networks for visual recognition after removing explicit nonlinear activation functions such as ReLU and GELU. Starting from ConvNeXt, the authors replace activation functions with element-wise multiplication and progressively introduce self-multiplication within a branch, multiplication between distinct branches, and multi-head channel partitioning to construct NANet, together with adjustments to the convolutional structure and normalization scheme. The authors evaluate models of different sizes on ImageNet-1K and report inference latency on an iPhone 13. In addition, the paper uses intermediate and final feature maps to show responses in target regions and analyzes the roles of multiplication, activation functions, and RMSNorm through ablation studies. The results support the feasibility of element-wise multiplication as an alternative to explicit activation functions. However, more comprehensive controlled experiments are needed to establish its distinct advantages over activation functions that retain negative values and to identify the sources of the final performance gains.

Critical strengths
  1. The network achieves competitive classification performance without explicit activation functions. In Table 3, NANet-L achieves the same accuracy of 83.1% as ConvNeXt-S, with similar parameter counts and computational costs. These results indicate that the approach scales to different model sizes and that its value extends beyond a proof of concept on small networks.
  2. The architectural evolution and component ablations provide valuable empirical analysis. Table 4 further compares multiplication, addition, RMSNorm, and BN. When multiplication is replaced with addition without introducing an activation function, accuracy drops from 74.76% to 61.09%, indicating that multiplicative interactions play a substantial role in this architecture. Together with the latency evaluation on an actual mobile device, these experiments provide a useful reference for understanding and implementing this class of models.
Critical weaknesses
  1. The paper presents complete semantic attribution as one of its core contributions, but the supporting evidence mainly consists of feature visualizations for a few examples in Section 4 and Appendix B. These results can show correspondence between feature responses and target regions, but they are insufficient to verify the completeness of semantic attribution or the actual contributions of these responses to the final class prediction. The authors need to define complete semantic attribution and its evaluation criteria clearly and provide more systematic validation to support this central claim.
  2. The motivation for removing explicit activation functions and the distinct advantages of multiplication over other activation schemes have not been sufficiently established. Section 2 primarily argues for replacing activation functions with multiplication on the grounds that negative responses are zeroed out or suppressed. However, the fact that ReLU zeroes out negative values cannot be directly generalized to all activation functions. Leaky ReLU and PReLU retain negative values, and GELU also retains some negative responses. Moreover, this paper starts from ConvNeXt, which uses GELU. The authors need to distinguish more clearly among the suppression of negative responses, information loss, and incomplete attribution, and explain what additional benefits multiplication offers over activation schemes that retain negative values.
Questions for authors

Please clarify the definition of complete semantic attribution and the criteria by which it can be verified. How do the current feature visualizations demonstrate attribution completeness, rather than merely spatial correspondence between feature responses and target regions? Could you provide a systematic evaluation related to the final class prediction, for example, perturbation or deletion experiments to verify the actual contribution of high-response regions to the prediction? Why does suppressing negative responses lead to incomplete attribution, and why would allowing negative responses to participate in multiplication avoid this problem? Please distinguish the effects of zeroing out negative responses with ReLU from those of suppressing them with GELU, and compare against schemes that retain negative values, such as Leaky ReLU and PReLU, using the same backbone and training settings. Could you explicitly list the differences in configuration and training between NANet and the ConvNeXt baseline, and provide controlled comparisons in which only the implementation of nonlinearity is changed while all other configurations are held fixed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.