- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies parameter free adaptation for linear minimization oracle on zero-order optimization. Motivated by the fact that ZO methods needs careful hyper parameter selection, the proposed method uses the local-smoothness stepsize idea and applies to ZO setting. Theoretical analysis are provided on convergence. Experimental results are provided on Opt-1.3B on SST-2 dataset with LoRA.
Critical strengths
The idea of reusing the same perturbation at adjacent updates is interesting and is a valid contribution. This helps with analysis and cancels some terms nicely.
Critical weaknesses
The experiment results are very thin and limited. Just one model over one dataset and test case, with LoRA rather than full parameter FT. There's no error bars and it's a single seed. More importantly, no training curve are shown for those methods. It's unclear how other methods work under same 35 wall clock hours. The training configurtions are also unclear for those baselines compared. There's also no empirically results and discussions on training stability and remaining hyper-parameters.
Questions for authors
-
More experiments and details need to be provided. See weakness.
-
What hyper parameters are eliminated, preserved, and introduced? How does the algorithm work with different hype-paramters? How sensitive is it?
-
Can you compare with MeZO? The algorithm seems requiring more compute (function evaluations) than MeZO. How does that affect memory usage and wall clock runtime?