- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper applies a know technique in glass physics directly into the training of a neural network. The set up looks simple, during training checkpoints at different training times are saved and from them a set of new trainings are spawned with the same parameters. The aim is to measure how much of the training is from that particular spawning point still left to chance. The authors claim that the contribution is this instrument. The authors separate the effect into three terms eq1 in the paper, where the observed variance is then assigned to chance (variability allowed still in the training procedure) and to a part that is not chance (further divided ni the observable part and the non observable part). Then using this instruments, the authors recover known effects in neural network training a provide 4 new observations.
Critical strengths
The main strength of the paper is the use of creativity to apply a physics instrument into the training of neural networks. It is based on previous work that sustains the paper and the literature seems to be well researched. The authors recover many known effects in network training and that provides credibility to the instrument. With the help of the instrument authors claim 4 new observations. I think the strongest could be the localisation of the committing point of a network. That, is when the training change variability is low, below 0.1% in the paper.
Critical weaknesses
The paper is somehow difficult to read at some parts, some concepts are used before they are defined in the paper. One of the claims marked as new in the introduction (number 3, Decide early) is marked as "recovered" in A.2. The instrument seems to be computationally expensive to use in practice, since many retrainings per saved checkpoint are needed.
For me is not clear if before the “decision” still something could be read by another instruments, since the presented one has a part not readable (beta) at any of the sampled steps.
Questions for authors
Could you clarify if the "decision" point is dependent on the sampling grid? And if before this could be in practice be even earlier due to the "non readable" part of the eq1 at every sample point?
Over the text it seems that the main bed used is the bed F, and some of the observations seems to be extrapolated from this runs. Could these extrapolations be particular of the F configuration?
How much computation is needed to run this instrument? Could you provide some numbers?