- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to learn spherical polynomials using a finite-width ReLU network with channel attention. The training has two stages. In the first stage, the algorithm takes one gradient step on the attention parameter. In the second stage, it resets the output layer and trains only the output coefficients.
The paper shows that the first step decreases the loss of a temporary predictor. It also shows that the attention step adds new features while keeping the original random features. Finally, the paper proves that the returned predictor has vanishing regression risk under suitable sample-size and width conditions.
Critical strengths
- The topic is interesting. The expected-risk theorem only requires finite noise variance, while an additional high-probability result is established under sub-Gaussian noise.
- The guarantee covers the full bounded class of spherical polynomials of degree at most . It does not require the target to depend on only one or a few directions.
- The paper shows that the attention network keeps the original random features and adds their products with a learned gate. The corresponding kernel identity gives a simple interpretation of how the representation changes after the attention step. This part of the analysis is clear.
Critical weaknesses
-
The paper does not prove that attention improves the final prediction result. It proves that the attention step adds new features. Since the original features are still kept, the new feature space is larger. Therefore, the best approximation over the new space cannot be worse than the best approximation over the old space. However, this only shows that the new model has more available features. It does not show that the learned features are useful for the target function. It also does not show that the final prediction risk is smaller. The final risk bound mainly uses the approximation ability of the original random features. Therefore, it seems that a standard random-feature model without attention can achieve the same rate.
-
The risk reduction in Proposition 4.2 is not connected to the final predictor. The proposition considers a temporary output layer, which is reset to zero before the second stage. The attention parameter is kept, but the paper does not show that the first-stage loss decrease improves the final estimator. It proves that one temporary predictor improves and that another predictor is consistent, without connecting the two results.
-
The network and training method are specially designed to obtain the main conclusions. The method uses paired directions, duplicated features, complementary gates, two output initializations, and a special learning rate. These choices make sure that the gradient is nonzero, the loss decreases, and the feature space becomes larger. Moreover, the ReLU directions are always fixed. The method is therefore closer to adding data-dependent features to a random-feature model and then performing linear regression than to standard neural-network feature learning.
Questions for authors
-
Can the same final risk rate be obtained if the attention step is removed?
-
Is it possible to prove that the final attention model has smaller risk than the model without attention? If not, why?
-
Can the authors show that the learned features contain useful information about the target function?
8 comments
Our Response, Part 2
Response: Thank you for raising the point that the proposed ReLU attention network is "closer to adding data-dependent features to a random-feature model". As a summary of our above responses and our explanation in Feature Learning by Strict Final Risk Reduction by the New Channel Attention, the new channel attention in the revised paper and its associated designs are among the first in the theoretical deep learning literature that rigorously prove the benefit of channel attention in reducing the final regression risk compared to the network without attention or with frozen attention, for the task of learning spherical polynomials with nearly optimal budgets for the sample size and network width . The feature learning capability of our ReLU attention network is mainly reflected in the learnable channel attention in stage one, and then the learned features are used in the output GD fitting in stage two.
We remark staged training, particularly two-stage training, has been used to study feature learning in neural networks with specific architecture designs [1,2]. In our network, the ReLU directions remain fixed, while stage one learns a target-dependent channel gate and stage two fits the resulting features. The architectural design preserves the original predictors while making products with the learned gate available to output fitting. Theorems 4.1 and J.1 establish that these learned features strictly reduce the final regression risk relative to the frozen-attention network and the non-attention network under their stated conditions, while preserving the consistency-rate order. As a result, the proposed ReLU attention network with the proposed careful designs and its two-stage training process extend the existing staged feature-learning research and reveal the benefit of channel attention.
References
[1] Alexandru Damian, Jason D. Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. COLT 2022.
[2] Yingzhen Yang. Shallow neural networks learn low-degree spherical polynomials with feature learning by learnable channel attention. ALT 2026.
Our Response, Part 3 (Feature Learning by Strict Final Risk Reduction by the New Channel Attention)
Feature Learning by Strict Final Risk Reduction by the New Channel Attention
We thank the reviewer for asking whether learned attention improves the predictor returned by training by reducing the final regression risk, beyond preserving or enlarging the available prediction space. The revised paper answers this question with Theorem 4.1, which proves strict final regression risk reduction relative to the frozen-attention network, and Theorem J.1 in Appendix J, which proves strict reduction relative to the non-attention network. These comparisons concern the returned predictors after all output GD steps, including the reset of the output coefficients before stage two.
The strict comparisons concern the new gate in channel attention in equation (9) of the revised paper, which uses harmonic inputs and normalized degree-specific functions.
Let be the uniform probability measure on , and take all norms below with respect to . Let be a predetermined spherical polynomial with bounded norm whose highest nonzero harmonic degree is the fixed integer . For a predictor , its population regression risk is . Write for that component, for its norm, and for the dimension of its harmonic space. The observations have independent, identically distributed centered noise, independent of the inputs and initialization, with variance at most .
Let be the sample size, the number of ReLU channels, the identity matrix, and the attention parameter. Write for the learned-attention predictor returned by Algorithm 1. The frozen-attention network keeps , so its attention matrix is . The non-attention network instead uses in the same network formula. Each comparison uses the same sample directions, second training sample and responses, zero output initialization, prescribed learning rate , and number of output steps .
The following statements use positive constants with the parameter dependence specified in the theorems.
- Theorem 4.1: Under its stated target, sample, width, and dimension conditions, the final decrease is at least with the explicit probability in equation (17).
- Theorem J.1: Under the same hypotheses, with a possibly larger dimension threshold, the final decrease has the same lower-bound order. Compared with Theorem 4.1, its failure bound includes an additional term of order . This theorem uses Algorithm 1's output learning rate and number of steps unchanged.
In particular, put . At and the default width , each theorem guarantees a decrease of at least a positive constant times with probability tending to one when is bounded away from zero and the target norm and noise variance remain bounded. The target class includes pure harmonics and mixtures of harmonic degrees, without a factorization assumption or a requirement for nonzero lower-degree components. These are guarantees for each predetermined target under the stated resource conditions, which make the algorithm's active harmonic degrees exactly . Constants are excluded from strict improvement, and a small highest-degree energy can make the probability bound uninformative. Training is not given the target degree or its coefficients.
Theorem 4.2 preserves the regression-risk order: the same learned-attention predictor has expected risk at these budgets over the full bounded- polynomial class of fixed degree, including constants. We agree that output training on the fixed biased ReLU features already achieves this consistency-rate order, as stated immediately after Theorem 4.2. The additional contribution is a quantitative strict comparison of the final trained predictors at this same rate. The proof connects the learned harmonic estimate to output approximation, finite GD error, and the effect of fitting the same response noise.
The following explanation is taken from Section 4.1 of the revised paper. All equation, lemma, and theorem numbers refer to that paper.
Our Response, Part 1
We appreciate the review and the suggestions in this review. The raised issues are addressed below.
In this response, we use the notation and detailed explanation in "Feature Learning by Strict Final Risk Reduction by the New Channel Attention", which is Part 3 of this response. It shows that the new efficient channel attention mechanism in the revised paper achieves a strict reduction in the final regression risk while preserving the original regression-risk order. The key reason for this strict reduction is that channel attention enables the ReLU attention network to approximate the highest-degree component fℓ0∗ of the target function f∗ better than the frozen-attention network, which only uses the ReLU features in the first layer. Learning fℓ0∗ is challenging because it lies in the degree-ℓ0 spherical harmonic space Hℓ0, whose dimension is ≍dℓ0, of the same order as the total dimension of all spherical harmonic spaces of degrees up to ℓ0. The new channel attention is efficient so that each forward and backward pass of the ReLU attention network has only linear time complexity of O(md) for each input (please refer to lines 283-285 and Appendix B.1 of the revised paper for details).
Weakness (1): "The paper does not prove that attention improves the final prediction result...."
Weakness (2): "The final risk bound mainly uses the approximation ability of the original random features. Therefore, it seems that a standard random-feature model without attention can achieve the same rate."
Question (2): “Is it possible to prove that the final attention model has smaller risk than the model without attention? If not, why?”
Response: Yes, we have proved the final attention model has smaller final regression risk than the model without attention (referred to as the non-attention network) and the model with frozen attention (referred to as the frozen-attention network). Theorems 4.1 and J.1 prove final population-risk decreases of at least cA∗2N(d,ℓ0)/m relative to the frozen-attention network and non-attention network, respectively, with high probability bounds (17) and (200). Please refer to Feature Learning by Strict Final Risk Reduction by the New Channel Attention for the details about the technical reason why the new attention in the revised paper achieves strict final risk reduction with high probability.
Question (3): “Can the authors show that the learned features contain useful information about the target function?”
Response: Yes, in the revised paper, the new gate gτ^ contains useful information about the target function, where τ^ is the learned attention parameter in training stage one. The products agϕj, where ag=gτ^−1/2, lead to an estimation of the highest-degree component fℓ0∗ in the output GD fitting in training stage two, while the retained features correct its residual with controlled coefficient cost (Lemmas G.1 and G.4).
Question (1): “Can the same final risk rate be obtained if the attention step is removed?”
Response: Yes. As stated immediately after Theorem 4.2, output-only GD on ψ+=ϕ/m, which are the biased ReLU features before channel attention, with the same second training sample and output schedule, achieves expected-risk bound (19) and rate O((logd)−2) at the stated sample n and network width m budgets. The components of ψ+ are features of the frozen-attention network and the non-attention network. Attention is unnecessary for this rate. On the other hand, as a key novel contribution of the revised paper, Theorems 4.1 and Appendix J.1 establish smaller final regression risk by learnable channel attention, compared to frozen-attention network (Theorems 4.1) and non-attention network (Appendix J.1).
Weakness (3): "The network and training method are specially designed to obtain the main conclusions. The method uses paired directions, duplicated features, complementary gates, two output initializations, and a special learning rate. These choices make sure that the gradient is nonzero, the loss decreases, and the feature space becomes larger. Moreover, the ReLU directions are always fixed. The method is therefore closer to adding data-dependent features to a random-feature model and then performing linear regression than to standard neural-network feature learning.