- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors observes that ReLU activation functions discard negative responses in the neural network, making the semantic attribution of input features complicated and incomplete. This paper propose to replace nonlinear activation functions with element-wise multiplication. With experiments, they demonstrate that simple element-wise multiplication of feature responses can achieve competitive performance while maintaining good semantic attribution and hence the interpretability of the model.
Critical strengths
- The paper shows that a ConvNet without any pointwise nonlinear activation can reach competitive ImageNet-1K accuracy from 3M to 50M parameters. Its only nonlinearities are element-wise multiplication and normalization layers.
- The paper documents the full trajectory from ConvNeXt to NANet in a step-by-step roadmap. Each step is trained independently, averaged over three seeds, and evaluated on both accuracy and real iPhone 13 latency.
Critical weaknesses
- The paper has severe factual incorrectness. Not all the nonlinear activation functions suppress negative responses, for example, leaky ReLU and tanh. The claim in the paper, that nonlinear activations discard negative responses, is not correct. The paper also critisizes ReLU for complicating model intepretation. In fact, ReLU activatino actually plays an important role in interpretable machine learning area. The foundamental paper "Layer-Wise Relevance Propagation" (LRP) highly relies on the interpretable property of ReLU activation. [1]
- The novelty of the paper is limited. Almost the same idea has already been proposed in NAFNet [2], while the authors failed to clearly explain their difference compared with this prior work. Increamental contributino is the claimed improved interpretability from element-wise multiplication. However, improved interpretability is only supported by qualitative visual comparison on feature maps, which is quite subjective, and personally I feel FocalNet actually provides more clear contour. Also, the visualization of interpretability is only limited to a few samples, some samples in Appendix are also repeated in the main paper. More diversed sample visualizations are needed to support the claim.
- The use of "univariate polynomial" for ConvNets and "bivariate polynomial" for ConvNets with element-wise multiplication is inaccurate and misleading. A convolution is a multivariate degree-1 polynomial, and the product of two branches is a multivariate degree-2 polynomial (a quadratic form) in the input. ReLU/GELU ConvNets are not polynomials at all. The relevant distinction is polynomial degree, not the number of variables, so the argument that "univariate polynomial" models have limited discriminability is built on a misnomer.
Reference
[1] Bach, Sebastian, et al. "On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation." PloS one 10.7 (2015): e0130140.
[2] Chen, Liangyu, et al. "Simple baselines for image restoration." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.
Questions for authors
- missing related works: Bilinear MLPs [3]
- In Figure 4, how does multi-head further reduce the latency? Intuitively, more heads will cost more computation.
- The FocalNet heatmaps in Figures 5 and 7 exhibit inverted polarity (dark objects, bright backgrounds), which contradicts the model's high accuracy. This likely occurs because FocalNet's final features have arbitrary signs (my hypothesis) and strongly negative activations may actually encode critical object evidence that the classifier weights map to positive logits. Consequently, the current direct visualization misrepresents feature importance, making the qualitative comparison favoring NANet unreliable. The authors should: (1) Specify the reduction method: clarify exactly how the tensor is reduced to a 2D map (e.g., mean, norm, or max). (2) Use sign-aware mapping: implement a class-specific visualization method, such as weighting feature channels by the classifier weights of the predicted class. (3) Provide quantitative evidence: support the interpretability claims with quantitative metrics rather than relying solely on subjective visual inspection.
- In Section 5.0.1, the authors mentioned that adding ReLU after element-wise multiplication provides almost no performance improvement, which seems quite unsurprising. In the end, the element-wise multiplication is also a kind of nonlinear operation, so the ultimate effects of traditional non-linear activation functions and element-wise multiplications are the same: increasing the expressive capacity of the neural network.
- The paper inaccurately frames nonlinear activation functions (e.g., ReLU, GELU) as causing information loss by "discarding negative responses". In reality, variants like Leaky ReLU, ELU, and GELU are primarily designed for optimization and regularization, such as maintaining gradient flow. The authors should revise their motivation to analyze training dynamics and gradient flow rather than attributing differences solely to information loss. Furthermore, the paper's own ablation (Table 4) is consistent with this optimization perspective, but inconsistent with the author's explanation. Adding ReLU after multiplication gives no gain (74.80%), while GELU improves accuracy (75.56%). Yet ReLU suppresses negative responses even more strongly than GELU, so the authors' explanation that GELU " improves feature discrimination by suppressesing strongly conflicting negative responses" (L421-L423) cannot account for the difference.
Reference
[3] Pearce, Michael T., et al. "Bilinear MLPs enable weight-based mechanistic interpretability." arXiv preprint arXiv:2410.08417 (2024).