- Rating
- 1
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies minimum sufficient precision for LLMs. The authors propose two metrics: numerical MSP and learning MSP. The former measures numerical difference with different precision setting whereas the latter measures the effect on downstream tasks. Experiments are conducted over multiple models to compare how those two metrics relate with each other.
Critical strengths
-
The idea of isolating/comparing local changes (numerical) vs global effects (learning) is interesting. Precision selection based only on local output error can miss changes in real operations that affects training, e.g. gradient update, which have different tolerence on precision requirements.
-
Experiment results are candid and transparent on limitations and failures. These disclosures make the work valuable as an emprical study.
Critical weaknesses
-
It's unclear how this is useful in practice. The results seem pretty noisy. Models can have two to three selected MSP and the numerical MSP often disagrees with learning MSP. And this is only across four precision setting. In practice, the setting is exponentially much more. The measurement also seems expensive under large number of settings. The authors should discuss and show how this method is useful in practice. Otherwise, this work remains an interesting finding, but not complete.
-
The robustness of the threshold and method is unclear. First, the two metrics, numerical and learning, are not on the same scale and needs to be normalized in certain ways. Then it's unclear how the threshold are selected to decide MSP. This is crutial and seems to be manually picked without justification or analysis.
Questions for authors
-
how's this useful in practice, espeically for guiding training process?
-
how robustness are the threshold and normalization method? does it changes with models, sizes, data?
-
Does "operator execution" means both forward GEMM and backward GEMM, or only forward?
-
Which is more favorable, numerical or learning? Can you provide actual evidence on using MSP produces better training outcomes, e.g. training loss?