- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work tackles the problem of numerical divergence when different (faster) kernels are substituted for known good ones. This work proposes a contract which specifies the divergence behaviour and a concrete way to test if a specific kernel substitution fits within the tolerances specified.
Critical strengths
This work identifies an important problem which is the long term numerical (behavioural) stability of different kernel implementations. This is particularly relevant now since there is significant diversity in hardware specific implementations in all parts of the stack (framework, parallelism, device specific kernels, etc.).
Critical weaknesses
- This work does not fundamentally solve the soundness problem.
- Training dynamic problems stemming from numerical and algorithmic differences in kernels can emerge way down the line during training[1] . This work essentially defers the correctness problem from a single invocation of the different kernels (the baseline) to a 50-step screen, while section 2.3 concedes that full paired training still determines the outcome.
- This work also does not justify the 50 iteration prefix length for contract checking.
- Practically, if the screen states a kernel selection is sound but it in fact diverges, we would not be able to know other than to train with the sound kernel end to end, which the paper already requires. What does the screen save?
- The propagation bound is badly justified.
- The work does not show that values the lipschitz factor takes on, and how it would change as training goes on. We don't know what factor to use before comprehensive measurements, and we also don't know if the same value would persist throughout the training.
- Furthermore, even a small lipschitz factor, when multiplied like in equation 7 (and 22 in the appendix) would explode over many steps (e.g. 5000) even for very small values of L. In reality, much of the single-step differences could be corrected by later gradient updates.
Questions for authors
- Would this methodology scale to larger models?
- Suppose a divergence happens later in the training, how would the authors detect and perhaps defer to a more sound kernel?