- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper analyzed the instability of EBTs by comparing the optimization quality of random truncation across model sizes and the changes in the retained gradient after truncation in existing algorithms. They utilized curvature score (defined using largest positive Hessian eigenvalue and learning factor) calculated during training and used it to monitor the correlation with instability and predict training failure and estimate performance. Their experiments involved using small, base, and large energy-based transformers for denoising.
Critical strengths
It is interesting that the paper consider using a curvature score during training to predict instability and training training failure to estimate performance.
Critical weaknesses
Although the analysis sounds interesting, however, I don't know what is the benefit from doing this. Gradient clipping has been commonly used to avoid instability, the paper did not discuss how clipping will affect their analysis, and I wonder if the maximum eigenvalue provided too generous bound here for the curvature score.
Meanwhile, the paper has many ambiguous statements, and imprecise terminologies. For example: "a gradient component along an eigenvector with curvature x"
Questions for authors
Is the maximum eigenvalue provided too generous bound here for the curvature score?
Response to All
We sincerely thank the reviewers for their time and their constructive feedback. We are glad that the reviewers find the research question practically relevant and the gradient comparison well controlled (yQET), and that they find the gradient-amplification analysis and the curvature score new and potentially useful for training EBTs reliably (o47G, qgTj).
After careful consideration, we have decided to withdraw the submission. The reviews raise points that we agree with and that require more work than the rebuttal period allows.
We will continue to work on the paper along these lines and submit it again once it is ready. Thank you again for your suggestions, which will make the next version stronger.