acceptodds
Under review as a conference paper at ICLR 2027

Learning Spherical Polynomials by Learnable Channel Attention

Abstract

We study the problem of learning a spherical polynomial of an unknown degree by training a ReLU network with channel attention by gradient descent (GD) with finite training sample and finite network width guarantee. The target functions are general spherical polynomials with bounded norm and a fixed, unknown degree bound . The training data are i.i.d. uniformly distributed inputs on the unit sphere in with i.i.d. centered noise of finite variance in their responses. The network width is the number of ReLU channels and the size of the diagonal channel-attention matrix. The training process of the network has two stages. In stage one, one ordinary GD step learns channel attention parameters and strictly enlarges the prediction space on the first half of the training data. In stage two, finitely many output GD steps on the second half of the training data achieve expected regression risk as . The feature kernel of the ReLU attention network evolves during training and the learnable channel attention provably reduceds the regression risk, demonstrating the provable benefit of feature learning. With and network width , the sample size and network width are only of logarithmic factors away from their corresponding lower bounds for learning degree- spherical polynomials. Under sub-Gaussian noise, the same algorithm also satisfies a high-probability risk bound with logarithmic confidence dependence. To the best of our knowledge, this is the first guarantee of vanishing regression risk for a finite ReLU network trained by GD with learnable channel attention, uniformly over bounded- spherical polynomials of degree at most , under these sample-size and width budgets.

Primary area:
learning theory
Keywords:
Spherical Polynomials, Feature Learning, Learnable Channel Attention, Gradient Descent, Nonparametric Regression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X8dnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to learn spherical polynomials using a finite-width ReLU network with channel attention. The training has two stages. In the first stage, the algorithm takes one gradient step on the attention parameter. In the second stage, it resets the output layer and trains only the output coefficients.

The paper shows that the first step decreases the loss of a temporary predictor. It also shows that the attention step adds new features while keeping the original random features. Finally, the paper proves that the returned predictor has vanishing regression risk under suitable sample-size and width conditions.

Critical strengths
  1. The topic is interesting. The expected-risk theorem only requires finite noise variance, while an additional high-probability result is established under sub-Gaussian noise.
  2. The guarantee covers the full bounded L2L^2 class of spherical polynomials of degree at most ℓ0\ell_0. It does not require the target to depend on only one or a few directions.
  3. The paper shows that the attention network keeps the original random features and adds their products with a learned gate. The corresponding kernel identity gives a simple interpretation of how the representation changes after the attention step. This part of the analysis is clear.
Critical weaknesses
  1. The paper does not prove that attention improves the final prediction result. It proves that the attention step adds new features. Since the original features are still kept, the new feature space is larger. Therefore, the best approximation over the new space cannot be worse than the best approximation over the old space. However, this only shows that the new model has more available features. It does not show that the learned features are useful for the target function. It also does not show that the final prediction risk is smaller. The final risk bound mainly uses the approximation ability of the original random features. Therefore, it seems that a standard random-feature model without attention can achieve the same rate.

  2. The risk reduction in Proposition 4.2 is not connected to the final predictor. The proposition considers a temporary output layer, which is reset to zero before the second stage. The attention parameter is kept, but the paper does not show that the first-stage loss decrease improves the final estimator. It proves that one temporary predictor improves and that another predictor is consistent, without connecting the two results.

  3. The network and training method are specially designed to obtain the main conclusions. The method uses paired directions, duplicated features, complementary gates, two output initializations, and a special learning rate. These choices make sure that the gradient is nonzero, the loss decreases, and the feature space becomes larger. Moreover, the ReLU directions are always fixed. The method is therefore closer to adding data-dependent features to a random-feature model and then performing linear regression than to standard neural-network feature learning.

Questions for authors
  1. Can the same final risk rate be obtained if the attention step is removed?

  2. Is it possible to prove that the final attention model has smaller risk than the model without attention? If not, why?

  3. Can the authors show that the learned features contain useful information about the target function?

8 comments
AuthorsOpenReview ↗

Our Response, Part 1

We appreciate the review and the suggestions in this review. The raised issues are addressed below.

In this response, we use the notation and detailed explanation in "Feature Learning by Strict Final Risk Reduction by the New Channel Attention", which is Part 3 of this response. It shows that the new efficient channel attention mechanism in the revised paper achieves a strict reduction in the final regression risk while preserving the original regression-risk order. The key reason for this strict reduction is that channel attention enables the ReLU attention network to approximate the highest-degree component fℓ0∗f_{\ell_0}^{\ast} of the target function f∗f^{\ast} better than the frozen-attention network, which only uses the ReLU features in the first layer. Learning fℓ0∗f_{\ell_0}^{\ast} is challenging because it lies in the degree-ℓ0\ell_0 spherical harmonic space Hℓ0\mathcal{H}_{\ell_0}, whose dimension is ≍dℓ0\asymp d^{\ell_0}, of the same order as the total dimension of all spherical harmonic spaces of degrees up to ℓ0\ell_0. The new channel attention is efficient so that each forward and backward pass of the ReLU attention network has only linear time complexity of O(md)\mathcal O(md) for each input (please refer to lines 283-285 and Appendix B.1 of the revised paper for details).

Weakness (1): "The paper does not prove that attention improves the final prediction result...."

Weakness (2): "The final risk bound mainly uses the approximation ability of the original random features. Therefore, it seems that a standard random-feature model without attention can achieve the same rate."

Question (2): “Is it possible to prove that the final attention model has smaller risk than the model without attention? If not, why?”

Response: Yes, we have proved the final attention model has smaller final regression risk than the model without attention (referred to as the non-attention network) and the model with frozen attention (referred to as the frozen-attention network). Theorems 4.1 and J.1 prove final population-risk decreases of at least cA∗2N(d,ℓ0)/mcA_*^2N(d,\ell_0)/m relative to the frozen-attention network and non-attention network, respectively, with high probability bounds (17) and (200). Please refer to Feature Learning by Strict Final Risk Reduction by the New Channel Attention for the details about the technical reason why the new attention in the revised paper achieves strict final risk reduction with high probability.

Question (3): “Can the authors show that the learned features contain useful information about the target function?”

Response: Yes, in the revised paper, the new gate gτ^g_{\hat {\mathbf \tau}} contains useful information about the target function, where τ^\hat {\mathbf \tau} is the learned attention parameter in training stage one. The products agϕja_g\phi_j, where ag=gτ^−1/2a_g = g_{\hat {\mathbf \tau}}-1/2, lead to an estimation of the highest-degree component fℓ0∗f_{\ell_0}^{\ast} in the output GD fitting in training stage two, while the retained features correct its residual with controlled coefficient cost (Lemmas G.1 and G.4).

Question (1): “Can the same final risk rate be obtained if the attention step is removed?”

Response: Yes. As stated immediately after Theorem 4.2, output-only GD on ψ+=ϕ/m\boldsymbol{\psi} _ +=\boldsymbol{\phi}/\sqrt m, which are the biased ReLU features before channel attention, with the same second training sample and output schedule, achieves expected-risk bound (19) and rate O((log⁡d)−2)O((\log d)^{-2}) at the stated sample nn and network width mm budgets. The components of ψ+\boldsymbol{\psi}_+ are features of the frozen-attention network and the non-attention network. Attention is unnecessary for this rate. On the other hand, as a key novel contribution of the revised paper, Theorems 4.1 and Appendix J.1 establish smaller final regression risk by learnable channel attention, compared to frozen-attention network (Theorems 4.1) and non-attention network (Appendix J.1).

Weakness (3): "The network and training method are specially designed to obtain the main conclusions. The method uses paired directions, duplicated features, complementary gates, two output initializations, and a special learning rate. These choices make sure that the gradient is nonzero, the loss decreases, and the feature space becomes larger. Moreover, the ReLU directions are always fixed. The method is therefore closer to adding data-dependent features to a random-feature model and then performing linear regression than to standard neural-network feature learning.

AuthorsOpenReview ↗

Our Response, Part 2

Response: Thank you for raising the point that the proposed ReLU attention network is "closer to adding data-dependent features to a random-feature model". As a summary of our above responses and our explanation in Feature Learning by Strict Final Risk Reduction by the New Channel Attention, the new channel attention in the revised paper and its associated designs are among the first in the theoretical deep learning literature that rigorously prove the benefit of channel attention in reducing the final regression risk compared to the network without attention or with frozen attention, for the task of learning spherical polynomials with nearly optimal budgets for the sample size nn and network width mm. The feature learning capability of our ReLU attention network is mainly reflected in the learnable channel attention in stage one, and then the learned features are used in the output GD fitting in stage two.

We remark staged training, particularly two-stage training, has been used to study feature learning in neural networks with specific architecture designs [1,2]. In our network, the ReLU directions remain fixed, while stage one learns a target-dependent channel gate and stage two fits the resulting features. The architectural design preserves the original predictors while making products with the learned gate available to output fitting. Theorems 4.1 and J.1 establish that these learned features strictly reduce the final regression risk relative to the frozen-attention network and the non-attention network under their stated conditions, while preserving the consistency-rate order. As a result, the proposed ReLU attention network with the proposed careful designs and its two-stage training process extend the existing staged feature-learning research and reveal the benefit of channel attention.

References

[1] Alexandru Damian, Jason D. Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. COLT 2022.

[2] Yingzhen Yang. Shallow neural networks learn low-degree spherical polynomials with feature learning by learnable channel attention. ALT 2026.

AuthorsOpenReview ↗

Our Response, Part 3 (Feature Learning by Strict Final Risk Reduction by the New Channel Attention)

Feature Learning by Strict Final Risk Reduction by the New Channel Attention

We thank the reviewer for asking whether learned attention improves the predictor returned by training by reducing the final regression risk, beyond preserving or enlarging the available prediction space. The revised paper answers this question with Theorem 4.1, which proves strict final regression risk reduction relative to the frozen-attention network, and Theorem J.1 in Appendix J, which proves strict reduction relative to the non-attention network. These comparisons concern the returned predictors after all output GD steps, including the reset of the output coefficients before stage two.

The strict comparisons concern the new gate in channel attention in equation (9) of the revised paper, which uses harmonic inputs and normalized degree-specific functions.

Let μ\mu be the uniform probability measure on X=Sd−1\mathcal{X}=\mathbb{S}^{d-1}, and take all L2L^2 norms below with respect to μ\mu. Let f∗f^{\ast} be a predetermined spherical polynomial with bounded L2L^2 norm whose highest nonzero harmonic degree is the fixed integer ℓ0≥1\ell_0\ge1. For a predictor f^\widehat f, its population regression risk is R(f^)=∥f^−f∗∥L22\mathcal{R}(\widehat f)=\lVert\widehat f-f^{\ast}\rVert_{L^2}^2. Write fℓ0∗f_{\ell_0}^{\ast} for that component, A∗=∥fℓ0∗∥L2>0A_{\ast}=\lVert f_{\ell_0}^{\ast}\rVert_{L^2}>0 for its norm, and N(d,ℓ0)≍dℓ0N(d,\ell_0)\asymp d^{\ell_0} for the dimension of its harmonic space. The observations have independent, identically distributed centered noise, independent of the inputs and initialization, with variance at most σ02<∞\sigma_0^2<\infty.

Let nn be the sample size, mm the number of ReLU channels, Im\mathbf I_m the identity matrix, and τ\boldsymbol{\tau} the attention parameter. Write f^A\widehat f_A for the learned-attention predictor returned by Algorithm 1. The frozen-attention network f^0\widehat f_0 keeps τ=0\boldsymbol{\tau}=0, so its attention matrix is Im/2\mathbf I_m/2. The non-attention network f^I\widehat f_I instead uses Im\mathbf I_m in the same network formula. Each comparison uses the same sample directions, second training sample and responses, zero output initialization, prescribed learning rate η=[4(d+1)]−1\eta=[4(d+1)]^{-1}, and number of output steps T=4(d+1)mT=4(d+1)m.

The following statements use positive constants c,Cc,C with the parameter dependence specified in the theorems.

  • Theorem 4.1: Under its stated target, sample, width, and dimension conditions, the final decrease R(f^0)−R(f^A)\mathcal{R}(\widehat f_0)-\mathcal{R}(\widehat f_A) is at least cA∗2N(d,ℓ0)/mcA_{\ast}^2N(d,\ell_0)/m with the explicit probability in equation (17).
  • Theorem J.1: Under the same hypotheses, with a possibly larger dimension threshold, the final decrease R(f^I)−R(f^A)\mathcal{R}(\widehat f_I)-\mathcal{R}(\widehat f_A) has the same lower-bound order. Compared with Theorem 4.1, its failure bound includes an additional term of order Cσ02m/(nA∗2)C\sigma_0^2m/(nA_{\ast}^2). This theorem uses Algorithm 1's output learning rate and number of steps unchanged.

In particular, put Hn=log⁡(en)H_n=\log(en). At n≍dℓ0(log⁡d)4n\asymp d^{\ell_0}(\log d)^4 and the default width m=4⌈n/(4Hn2)⌉≍dℓ0(log⁡d)2m=4\lceil n/(4H_n^2)\rceil\asymp d^{\ell_0}(\log d)^2, each theorem guarantees a decrease of at least a positive constant times A∗2/(log⁡d)2A_{\ast}^2/(\log d)^2 with probability tending to one when A∗A_{\ast} is bounded away from zero and the target norm and noise variance remain bounded. The target class includes pure harmonics and mixtures of harmonic degrees, without a factorization assumption or a requirement for nonzero lower-degree components. These are guarantees for each predetermined target under the stated resource conditions, which make the algorithm's active harmonic degrees exactly 1,…,ℓ01,\ldots,\ell_0. Constants are excluded from strict improvement, and a small highest-degree energy can make the probability bound uninformative. Training is not given the target degree or its coefficients.

Theorem 4.2 preserves the regression-risk order: the same learned-attention predictor has expected risk O((log⁡d)−2)O((\log d)^{-2}) at these budgets over the full bounded-L2L^2 polynomial class of fixed degree, including constants. We agree that output training on the fixed biased ReLU features already achieves this consistency-rate order, as stated immediately after Theorem 4.2. The additional contribution is a quantitative strict comparison of the final trained predictors at this same rate. The proof connects the learned harmonic estimate to output approximation, finite GD error, and the effect of fitting the same response noise.

The following explanation is taken from Section 4.1 of the revised paper. All equation, lemma, and theorem numbers refer to that paper.

Reviewer_cH7mOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper shows that finite width ReLU network with channel attention trained by two stage gradient descent can learn any bounded L2\mathcal{L}^2 spherical polynomial data of unknown degree. They showed a synthetic experiment on a cubic polynomial that ReLU with channel attention has better test MMSE than without attention.

Critical strengths

S1) The paper is well written, and the proof outline is very useful for understanding the machinery used to obtain the convergence results.

S2) The work provides strong guarantees under weak assumptions.

S3) The authors have provided mature literature review with comparison.

Critical weaknesses

W1) The authors have only presented one synthetic experiment on cubic polynomial with the observed test MMSE. However, they did not compare the theoretical bounds with the empirical ones.

W2) The data assumptions may not hold in practice, but experiments on real datasets would still be valuable for assessing whether the channel attention benefit extends beyond the synthetic setting.

Questions for authors

Q1) Can the authors verify their theoretical bounds in the setting of Figure 1?

Q2) Can the authors comment on how do their bounds for ReLU with channel attention compare with ReLU without channel attention? Does do they match with observations in the synthetic experiment?

3 comments
AuthorsOpenReview ↗

Our Response

We appreciate the review which acknowledges our contribution and the suggestions in this review. The raised issues are addressed below.

Weakness W1: "The authors have only presented one synthetic experiment on cubic polynomial with the observed test MMSE. However, they did not compare the theoretical bounds with the empirical ones."

Question Q1: "Can the authors verify their theoretical bounds in the setting of Figure 1?"

Response: The synthetic-data study in Appendix K.1 of the revised paper uses the normalized cubic f∗(x)=d(d+2)(d+4)x1x2x3f^*(\mathbf x)=\sqrt{d(d+2)(d+4)}x_1x_2x_3, whose squared L2L^2 norm and highest-degree energy are both one. Put N3=N(d,3)=d(d−1)(d+4)/6N_3=N(d,3)=d(d-1)(d+4)/6. Under its hypotheses, Theorem 4.2 bounds expected regression risk by C[N3/m+σ02m/n+(1+σ02)n−7/2]C[N_3/m+\sigma_0^2m/n+(1+\sigma_0^2)n^{-7/2}], where σ02\sigma_0^2 bounds the noise variance and CC is the theorem's constant. For bounded noise variance, this gives a regression risk bound of O((log⁡d)−2)O((\log d)^{-2}) at n≍d3(log⁡d)4n\asymp d^3(\log d)^4 and m≍d3(log⁡d)2m\asymp d^3(\log d)^2.

Figure 1 reports mean test MSE over ten initializations with 1,000 independent test inputs at each tested dimension. Appendix K.1 reports that the mean loss reductions relative to both baselines are at least the reference value (log⁡d)−2(\log d)^{-2} at d=10,20,…,100d=10,20,\ldots,100, matching the theoretical prediction made by Theorem 4.1.

Question Q2: "Can the authors comment on how do their bounds for ReLU with channel attention compare with ReLU without channel attention? Does do they match with observations in the synthetic experiment?"

Response: The paragraph after Theorem 4.2 of the revised paper explains that the frozen-attention network achieves the same expected-risk bound, Equation (19) of the revised paper, as the ReLU attention network with learnable channel attention. Learnable channel attention is unnecessary for this bound. The benefit of learnable channel attention is a strict reduction of the final regression risk over the frozen-attention network (the network with frozen attention parameter τ\mathbf \tau) and the non-attention network (the network without attention), as shown in Theorem 4.1, equation (17), and Theorem J.1, equation (200) of the revised paper. Such a risk reduction is at least cA∗2N(d,ℓ0)/mcA_*^2N(d,\ell_0)/m relative to the frozen-attention network and non-attention network, respectively, with their stated probability bounds under their hypotheses. Here ℓ0\ell_0 is the highest target degree, A∗=∥fℓ0∗∥L2A_*=\|f_{\ell_0}^*\|_{L^2}, and c>0c>0 is a constant. Both comparisons keep Algorithm 1's output learning rate and number of steps unchanged.

For the cubic target function $f ^ * $ in Appendix K.1, ℓ0=3\ell_0=3 and A∗=1A_*=1, the reduction of the test loss by the ReLU attention network over the non-attention network and the frozen-attention network in Figure 1 is lower bounded by (log⁡d)−2(\log d)^{-2} for d∈[10,100]d \in [10,100], which is consistent with the final regression risk reduction of the order Θ((log⁡d)−2)\Theta((\log d)^{-2}) when $| f ^ *_ {\ell _ 0} |_{L^2} = \Theta(1) $ and ∥f∗∥L2\| f ^ * \| _ {L^2} is bounded, according to Equation (17) in Theorem 4.1 of the revised paper.

Weakness W2: "The data assumptions may not hold in practice, but experiments on real datasets would still be valuable for assessing whether the channel attention benefit extends beyond the synthetic setting."

Response: Appendix K.2 and Figure 2 in the revised paper present a mini-ImageNet evaluation with input dimensions d=10,…,19d=10,\ldots,19. For each dd, we apply principal component analysis to the original data and retain the leading dd principal components. We perform this study to empirically demonstrate the test loss of the ReLU attention network when the input data distribution and target function may not satisfy the assumptions required in our analysis in Theorem 4.1 and Theorem 4.2. Figure 2 shows that the ReLU attention network has lower test loss than both baseline networks at each dimension.

AuthorsOpenReview ↗

Thank You and Author-Reviewer Discussion Reminder

Dear Reviewer cH7m,

We hope this message finds you well, and you just had a great Tuesday!

As a gentle reminder, tomorrow (11/18) is the final day of the Author-Reviewer Discussion period. We would like to thank you again for the time and effort you have devoted to reviewing our submission.

We believe we have addressed all of the issues and questions you raised during the review process in our rebuttal. We would be very grateful if you could take a moment to evaluate our responses before the discussion period concludes.

If there are any remaining questions, concerns, or points that you feel have not been adequately addressed, please let us know. We will respond as promptly as possible and are happy to provide any additional clarification needed.

Thank you again for your thoughtful feedback and consideration.

Regards,

The Authors

I thank the authors for clarifying all my questions. I maintain my positive recommendation for this submission.

Reviewer_JQwgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper analyzes the sample complexity and number of neurons to estimate a polynomial function on the sphere by a shallow neural networks. The authors consider a two layer network with ReLU activation and sigmoidal gating (neuron-wise attention) acting on each neuron's output. It is shown that sample complexity n=Θ(dℓ0(log⁡d)4)n = \Theta(d^{\ell_0}(\log d)^4) and width m=Θ(dℓ0(log⁡d)4)m =\Theta(d^{\ell_0}(\log d)^4) yields the test loss decrease O(log⁡(d)−2)O(\log(d)^{-2}). This results significantly improves existing bound, for example, Yang (2026) showed convergence with n=O(dL+2ℓ0log⁡dn = O(d^{L+2\ell_0} \log d and m=Ω(d2ℓ0)m = \Omega(d^{2\ell_0}). Their targets are assumed to satisfy a fixed weighted harmonic norm bound and a positive lower bound on every normalized coefficient, but this paper does not require such conditions. On this setting, the authors consider a two stage optimization method: (first step) one-step gradient descent to update the attention part, and (second step) gradient descent to optimize the linear coefficient in the second layer. Their analysis shows that the attention update produces additional basis function which reduces the residual while the required norm bound of parameters are preserved.

Critical strengths

Overall, this paper gives a solid and strong theoretical results on the literature. Estimating a polynomial defined on the sphere is one of the typical problem settings, and this paper adds an important result on the field.

  • Complexity improvement: The provided analysis actually improves existing bound from d2ℓ0d^{2 \ell_0} to dℓ0(log⁡d)4d^{\ell_0}(\log d)^4. This is a strong contribution, and it achieves almost the optimal rate up-to (log⁡d)4(\log d)^4. Moreover, it does not require bounded harmonic norm bound and positive lower bound on every normalized coefficient unlike existing work. This is a significant contribution.
  • Benefit of feature learning: They also showed that the feature space is indeed expanded by training the attention part which reduces the population risk. This also supports importance of feature learning.
  • Technical soundness: As far as I checked the proof, there is no technical concern in the proof.
  • Writing: The paper is overall well written. The main contribution of this paper is clearly written. On the other hand, I have a concern on this part as described in the weakness part.
Critical weaknesses

My biggest concern is the benefit of feature learning. This paper considers the attention network and the feature learning aspect is emphasized. On the other hand, it seems that the essential ingredient to improve the width stems from employment of the biased ReLU activation. The proof already establishes the bound for the conjugate kernel induced by the biased ReLU activation at the initialization without update of the attention layer. The feature learning part of the proof only guarantees the approximation error (and the RKHS norm) does not increase by the attention update. Although decrease of the population risk is quantitatively shown, it is not essential to achieve dℓ0(log⁡d)4d^{\ell_0}(\log d)^4 width bound.

For this reason, I was confused by the writing of this paper that emphasizes the feature learning aspect. In that sense, I think it is better to position this paper as a new analysis of the biased ReLU activation rather than showing benefit of feature learning. This point is not directly described. I think it should be clarified.

Minor point: There is a unresolved reference "refeq:summary-network" just below Equation (12).

Questions for authors

As I described in the weakness item, I think the improvement comes from the biased ReLU activation instead of the attention. Thus, the feature learning is not essential to achieve the optimal rate where the constant factor improvement is not significant. If this is not true, please correct it.

5 comments
AuthorsOpenReview ↗

Our Response

We appreciate the review which acknowledges our contribution and the suggestions in this review. The raised issues are addressed below.

Weakness (1): "My biggest concern is the benefit of feature learning. This paper considers the attention network and the feature learning aspect is emphasized. On the other hand, it seems that the essential ingredient to improve the width stems from employment of the biased ReLU activation. The proof already establishes the bound for the conjugate kernel induced by the biased ReLU activation at the initialization without update of the attention layer. The feature learning part of the proof only guarantees the approximation error (and the RKHS norm) does not increase by the attention update. Although decrease of the population risk is quantitatively shown, it is not essential to achieve dℓ0(log⁡d)4d^{\ell_0}(\log d)^4 width bound."

Question: "As I described in the weakness item, I think the improvement comes from the biased ReLU activation instead of the attention."

Response: We agree that the biased ReLU features suffice for this consistency-rate order. Lemmas F.1 and F.2 provide the biased ReLU harmonic coefficients and finite-width approximation with controlled coefficient norm. As stated after Theorem 4.2 of the revised paper, output-only GD on ψ+=ϕ/m\boldsymbol{\psi}_+=\boldsymbol{\phi}/\sqrt m, which are the biased ReLU features, satisfies expected-risk bound (19) and consistency rate (20). ψ+\boldsymbol{\psi} _ + are features of the frozen-attention network (the network with frozen attention parameter τ=0\mathbf \tau = 0) and the non-attention network (the network without attention).

We also agree with the reviewer that learnable channel attention does not contribute to the nearly optimal budget of sample size nn and the network width mm. To make this point clear, we have noted immediately after Theorem 4.2 of the revised paper that "...The learned-attention predictor and frozen-attention network both achieve expected risk O((log⁡d)−2)O((\log d)^{-2}), and attention is unnecessary for achieving this consistency rate with nearly optimal sample size n≍dℓ0(log⁡d)4n \asymp d^{\ell_0} (\log d)4 and network width m≍dℓ0(log⁡d)2m\asymp d^{\ell_0}(\log d)^2.

Weakness (2): "For this reason, I was confused by the writing of this paper that emphasizes the feature learning aspect. In that sense, I think it is better to position this paper as a new analysis of the biased ReLU activation rather than showing benefit of feature learning. This point is not directly described. I think it should be clarified."

Question: "Thus, the feature learning is not essential to achieve the optimal rate where the constant factor improvement is not significant. If this is not true, please correct it."

Response: As mentioned in the previous response, we agree with the reviewer that the feature learning is not essential to achieve the optimal rate.

On the other hand, we have proposed a new channel attention mechanism in the revised paper, which achieves strict final regression risk reduction over the frozen-attention network and the non-attention network. This new channel attention reflects an enhanced feature learning effect of the ReLU attention network.

As a result, the contributions of this paper are twofold.

First, we propose a new two-stage training algorithm on a novel ReLU attention network. For the nearly optimal sample size n≍dℓ0(log⁡d)4n\asymp d^{\ell_0}(\log d)^4 and network width m≍dℓ0(log⁡d)2m\asymp d^{\ell_0}(\log d)^2, Theorem 4.2 gives expected risk O((log⁡d)−2)→0\mathcal O((\log d)^{-2}) \to 0 as d→∞d\to\infty. Both resources exceed the polynomial dimension dℓ0d^{\ell_0} only by logarithmic factors in terms of log⁡d\log d. Regression with the biased ReLU features without the learnable channel attention can achieve the same regression risk under the same budgets of sample size nn and network width mm.

Second, the new learnable channel attention in the ReLU attention network achieves the feature learning effect, which strictly reduces the final regression risk compared to the frozen-attention network and the non-attention network, as stated in Theorem 4.1 of the revised paper.

The Role of Feature Learning. The role of feature learning by our learnable channel attention is to strict reduce the final regression risk compared to the frozen-attention network and the non-attention network. With fixed energy of the highest-degree component of the target function ∥fℓ0∗∥L2\| {f_ {\ell_0} ^ *}\| _ {L^2}, such a risk reduction is of Ω((log⁡d)−2))\Omega ( (\log d)^{-2})) by Theorem 4.1, the same order as the regression risk bound.

The above twofold contributions are reflected in the introduction section of the revised paper.

Minor point: "There is a unresolved reference "refeq:summary-network" just below Equation (12)."

Response: The unresolved reference has been fixed in the revised paper.

Please kindly let us know if you have any remaining concerns, and we will address them momentarily. Thank you!

AuthorsOpenReview ↗

Thank You, Further Update, and Author-Reviewer Discussion Reminder

Dear Reviewer JQwg,

We have further clarified the contributions of this paper in the paragraph titled The Role of Feature Learning in the introduction of the revised paper, which is copied below for your convenience.

The Role of Feature Learning. At the nearly optimal budgets n≍dℓ0(log⁡d)4n\asymp d^{\ell_0}(\log d)^4 and m≍dℓ0(log⁡d)2m\asymp d^{\ell_0}(\log d)^2, fixed biased ReLU features already attain expected risk O((log⁡d)−2)\mathcal O((\log d)^{-2}) without learning attention. Our new learnable channel attention preserves this rate over the full bounded-L2L^2 polynomial class, with the same order at fixed confidence under sub-Gaussian noise (Theorem 4.1). Its feature learning contribution is a strict final risk decrease of Ω(∥fℓ0∗∥L22/(log⁡d)2)\Omega(\| f_{\ell_0}^* \| _ {L^2}^2/(\log d)^2) over the frozen-attention network and the non-attention network under Theorems 4.1 and J.1, respectively. When the highest-degree energy is bounded away from zero and the target norm and noise variance are bounded, this decrease is Ω((log⁡d)−2)\Omega((\log d)^{-2}) with probability tending to one, on the same scale as the risk bound O((log⁡d)−2)\mathcal O((\log d)^{-2}) . Our novel channel attention adds O(m)\mathcal O(m) work per input to the O(md)\mathcal O(md) ReLU feature computation and O(m)\mathcal O(m) preparation after its update (Appendix B.1).

As a gentle reminder, tomorrow (11/18) is the final day of Author-Reviewer Discussion period. We would like to thank you again for the time and effort you have devoted to reviewing our submission.

We believe we have addressed all of the issues and questions you raised during the review process in our rebuttal. We would be very grateful if you could take a moment to evaluate our responses before the discussion period concludes.

If there are any remaining questions, concerns, or points that you feel have not been adequately addressed, please let us know. We will respond as promptly as possible and are happy to provide any additional clarification needed.

Thank you again for your thoughtful feedback and consideration.

Regards,

The Authors

AuthorsOpenReview ↗

Thank You and Author-Reviewer Discussion Reminder

Dear Reviewer JQwg,

We hope you are enjoying a great day!

As a gentle reminder, today is the final day of Author-Reviewer Discussion period. We would like to thank you again for the time and effort you have devoted to reviewing our submission.

We believe we have addressed all of the issues and questions you raised during the review process in our rebuttal. We would be very grateful if you could take a moment to evaluate our responses before the discussion period concludes.

If there are any remaining questions, concerns, or points that you feel have not been adequately addressed, please let us know. We will respond as promptly as possible and are happy to provide any additional clarification needed.

Thank you again for your thoughtful feedback and consideration.

Regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and Summary of Final Revisions

Dear Reviewers and Area Chair,

Thank you for your careful reviews and constructive discussion. We have addressed all the concerns from the three reviewers. We summarize the revisions and recorded reviewer responses below.

(1) Reviewer JQwg: We clarified that fixed biased ReLU features already achieve the regression risk of the order O((log⁡d)−2)\mathcal O((\log d)^{-2}), while learned channel attention provides an additional, quantitative final-risk reduction in Theorem 4.1. Under the stated conditions, this decrease is Ω((log⁡d)−2)\Omega((\log d)^{-2}), on the same scale as the risk bound. The reviewer confirmed that Theorem 4.1 explicitly quantifies the risk reduction, while maintaining the original score (2) because "the story of the paper's position should be re-organized to be published."

We remark that the revised introduction’s “The Role of Feature Learning” paragraph already explicitly separates the contributions of fixed biased ReLU features in achieving the regression risk bound and the risk reduction by the learnable channel attention. Furthermore, the explanation following Theorem 4.1 connects the limitation of frozen ReLU features to the information learned by the gate and then to the risk of the returned predictor. This positions the random-feature model as the foundation and the quantified benefit of learning channel attention as the central feature-learning contribution. As a result, the revised paper has already been organized in the way suggested by the reviewer. The reviewer further mentioned a factually ungrounded claim that "I guess any generic nonlinear transformation would achieve it and using attention in FNNs is not very common" in their follow-up response, which we clarified in our further response to this reviewer.

To this end, we believe the current evaluation is not grounded in actual evidence, and we sincerely hope that the area chair will make an informed assessment and decision.

(2) Reviewer X8dn: The new Theorems 4.1 and J.1 establish strict final-risk improvement over the frozen-attention network and the non-attention network, respectively, using the prescribed output-GD schedule. Theorem 4.2 preserves the regression-risk order. We explained how the learned gate captures useful target information, clarified the specialized architecture and its computational cost, discussed the comparison with direct harmonic regression, and specified experimental sample sizes and widths. The reviewer explicitly acknowledged that Theorem 4.1 addresses the missing final-prediction-risk guarantee, while maintaining the original score (2) because of the following three concerns/questions:

  1. "Why not perform regression directly using spherical harmonics? The proposed approach seems rather unnatural. Therefore, it is unclear whether the proposed neural network offers any advantage over a direct harmonic-based method, either statistically or computationally".

  2. "Furthermore, the revised architecture is even more specialized than the original one. The attention mechanism now incorporates harmonic kernels, degree-dependent weights, and carefully designed normalization factors. Hence, this mechanism appears to rely heavily on prior knowledge of the target function class."

  3. "sample-size and width conditions in section K appear incompatible with the mini-ImageNet experiment, and the actual values of n(d) and m(d) are not clearly specified."

We fully answered and solved (1) and (3). Regarding concern (2), we provided a detailed explanation in our "Further Response, Part 1" to this reviewer. In summary, it is a common practice in existing research in neural networks feature learning to design specialized architectures and training methods to learn particular classes of target functions. We remark that given that the target functions in this paper are assumed to be spherical polynomials, our careful and specialized network design achieves feature learning with reduced final regression risk over networks without attention or with frozen attention, which already achieves the goal of this paper. While we agree that gradient-based feature learning with a general neural network architecture to learn broader target functions is definitely interesting, this is beyond the scope of this paper.

(3) Reviewer cH7m: We addressed all the concerns from this reviewer and the reviewer maintained their positive rating (3) of this paper.

We will faithfully incorporate all promised revisions and would be happy to address any remaining questions or concerns.

Thank you again for your time and consideration.

Regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.