- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SUMMARY The article offers analysis of weight perturbation based gradient free training. In particular, it derives an expression for the cosine (square) angle between the perturbation based gradient and the backpropagation gradient, as a function of the dimensions oof hte weight matrix and the number of forward evaluation iterations per update. Then this expression used to evaluate different experimental scenarios including the large scale experiment of EGGROLL article. The authors also make some information theoretic claims regarding the updates, which are not rigorously based.
Critical strengths
Theorem 1 providing the cosine (square) alignment result, providing a clear expression in terms of weight matrix dimensions is valuable. The article also contains some numerical analysis based on this.
Critical weaknesses
Theorem 1 provides a useful result, however, it is not as major result to base a full article. The theorem itself and proof steps are relatively elementary. The rest of the numerical analysis and claims are not in a level to be offered as fundamental findings for a flagship ML conference article. In addition, the presentation is poor, obscured by financial analogies, convoluted narratives, undefined symbols, and imprecise assumptions.
COMMENTS ABOUT SECTIONS
Abstract
- the sentence about the cosine(square) angle with backrop G is ambigious.
- killing all four alternatives: ambigious and not precise statement.
- that pre-registered experiment: not clear
- not familiar with the use of prices and finances terms within this context.
Introduction
-
parallelizes trivially: weight perturbation, if handled in the same worker (by the claim: fits hardware smaller than backprop case), synchronous perturbations in multiple weights for paralllel updates cause performance issues? and if handled in multiple workers requires communication of update terms among workers and requires more hardware than backprop then. I am not sure if trivial is appropriate here.
-
The use of exchange-rate terminology (in title) and at the end of paragraph 1 is not precise?
-
the third paragraph: cos^2 is not clear, the parameters m and n are not defined yet. and what floor? the last sentence is also not clear (aspect-ratio chain...). what is folklore here, and what are natural alternatives (citations?)
-
Paragraph 4 is also not clear.
Section 2
This section could be clarified and potentially be placed in the Appendix.
Section 3
-
Before Theorem 1, subindex was used for perturbation scale, right after that it is used as forward evaluation index .
-
The expression for assumes an even value. In, Theorem 1, with that definition does not make sense. subindex of \hat{g} is not really the number of forward evaluations but antithetic index. whitened-> white. also zero mean i.i.d. unit variance.
-
Proof of Theorem 1: The expression is valid for kurtosis of is equal to , i.e. Gaussian. However, this assumption is not stated.
Overall, Theorem 1 appears very basic, but useful. Its proof can be expanded for better clarity.
-
Table 1: A column showing normalized error in predicting the would be informative.
-
Section 3.4 what are the source of these hypotheses (references?) why do you need to compare against them if H0 is theoretically grounded?
Section 4 In my opinion this whole section can be reduced to one or two paragraphs, which is looking at the large scale experiments of EGGROLL (Sarkar et al., 2026) from the lens of the Theorem 1 of the current article.
Section 5 I could not follow the discussion in this section, which put forwards some information theoretic statements such as mutual information, capacity without any rigorous treatment.
Section 6 What are these labels P-7.6x refering to? Regarding the discussion in this section: it is clear that gradient does not necessarily offer the best search direction, but it is informative to see alignment vs accuracy values together.
Section 7 I would expect better discussion here discussing the utility, contributions, conclusions and limitations of the article. Instead it contains specific information which is confusing and containing labels like R-7.55a which is not predefined.
Questions for authors
-
Regarding Theorem 1: Do you assume that a,b are Gaussian and and zero mean?
-
what are some symbols used in the article such as P7-6, R-7..
-
what are the hypotheses in Section 3, are they from any specific references?
-
Could you provide more rigorous statements about the information theoretic claims made in Section 5?