- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This work tries to back propagate prediction error back and examine how numerical errors come in. There is no clean conclusion (as expected in very general setups), and the takeaway is ambiguous "the adequacy of a numerical approximation must be assessed in relation to the current learning state and the quantity to be preserved; numerical accuracy should itself be treated as part of the learning objective."
Critical strengths
The high level question can be interesting: how compute error changes model performance. This can interesting when LLMs now are usually compressed / quantized after training.
The writing style is very clean, with perfect english, and clean structure.
Critical weaknesses
I think there are two main problems.
First, this work is not readable. The terms and setups are not put into previous works for readers to understand what each term means. Instead, it follows AI writing style, where random terms and notations come in. I will just give a few examples: a. "primitive error" refers to numerical error in each computation step. b. "learning state" means the computed values given a data and a model param in the compute graph of a learned model. c. "learning budget" means the final compute error
Second, there is no interesting technique or interesting conclusion. saying that 'it depends' is okay if surprising and interesting setups are shown to given different behavior. Simply showing that it depends on the logit distribution, for example, is too trivial.
Questions for authors
See weaknesses.