- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper explores whether the momentum spectrum may be used to determine good rank parameters for low rank optimizers. The paper introduces a noise threshold diagnostic procedure to estimate the rank for which higher rank singular vectors in the momentum spectra consist of noise. The paper empirically finds that momentum spectra are approximately gapless and that model performance reliably improves with higher rank optimization. It also identifies a "feedback collapse" phenomenon that may occur with online used of the noise threshold diagnostic procedure and introduces a method to correct for it. Experiments are conducted on langauage models ranging from 60M to 350M parameters. The paper claims that its method provides a " lower bound on useful rank", but this claim requires further justification. The fact that validation loss improves beyond the selected rank is not evidence that the procedure finds a lower bound on "useful rank".
Critical strengths
The paper identifies the failure mechanism of feedback collapse and successfully introduces a method to correct for it, empirically preventing feedback collapse across the experimental model scales.
Critical weaknesses
The rank selection procedure is not practically justified by the paper's findings. The paper does not demonstrate that the rank selection method it introduces gives a procedure for finding favorable points on a cost/quality curve. The paper shows that its diagnostic is stable across model scales, but the case is not made that the paper's rank selection method is one that should actually be used. In particular the paper does not show why its rank selection method would be favorable to a simple fixed rank choice. The procedure should be evaluated against such alternatives under a specified objective, for example loss achieved for a fixed computational budget. In its current form, the paper does not make a clear connection between a stable spectral measurement and practically useful optimizer rank.
Questions for authors
Are there any ways in which the rank hyperparameter found by your procedure could be shown to select ranks that are favorable in some sense to a simple baseline?
Thank you for the response, and I will keep the current rating.