- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose two ZO method to introduce the online sample-wise second-moment estimate to scale the vanilla ZO perturbations to achieve updates with curvature information. It provides both theoretical analysis and empirical studies to validate their efficiency. For the modern LLM models, the fine-tuning performance can achieve better than several classical baselines.
Critical strengths
-
The paper writing and total organization is good. Each key factors and points are claimed clearly and can be easily understood in the detailed operations.
-
It brings several novel and interesting perspectives into the ZO community, enriching the insights for the algorithm designing.
Critical weaknesses
-
The motivation of adopting average of the squared per-sample gradients surrogate is unclear. The authors claimed that squaring the mini-batch averaged gradient discards the variation among individual sample gradients and may yield a small value. I do not believe this is a weakness which should be resolved in the training. Since in the FO method, this is a classical and widely adopted operation. Also, the empirical advantages are unclear between these two cases.
-
The calculation of terms adopt instead of . The authors claim term has included the information of . However, it is a loss gap, which means a scaler to determine the strength of perturbation. The true direction of ZO gradient is (or ). Thus simply ignoring the term in second order momenta does not make sense enough. More discussions should be added to claim the motivation.
-
Assumption 1 seems too strong that for the general Gaussian sampling its upper and lower bound may divergent. The proofs seems only consider them as constants in the final bounds.
-
Lack of key ablation studies. It's not clear the improvements come from which component primarily.
Questions for authors
-
Why and how the assumption 1 is satisfied under ?
-
We usually use batch-wise gradient in practice. Why is adopting sample-wise gradient important?
-
What is the theoretical impacts between these two cases, batch-wise and sample-wise? If we use the sample-wise gradient, can it converge faster or better?
-
Why the second-order momenta is calculated only on instead of or ? Can authors provide the detailed studies between these cases?
-
Can authors provide a detailed ablation on each component in the proposed method? As my current understanding, the method mainly adopts the curvature matrix to revise the perturbations and uses sample-wise gradient for second-order momenta. So which is the key contributor for the accuracy compared to MeZO or HiZOO? I am not quite clear the source of the improvements.
-
Can authors provide a detail comparison to illustrate the differences of calculation among the baselines and the proposed method? If possible, please survey more related baselines. Experiments are not important, while the difference and insight is.
-
What is the training profiles on memory and calculations?