- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors observe, that, under convex optimization, the earlier gradient errors matter geometrically less than the final ones. Therefore, one should focus the compute on the final optimization steps. This is possible to implement directly for Neural ODEs, via scheduling the tolerance of the method.
Empirically, the authors demonstrate the superiority of their simpler scheduling approaches over fixed-tolerance ones -- on a cubic synthetic dataset. Then, the authors compare these approaches on real data -- Jena weather and MNIST. On real data, the results are mixed, with a fixed-tolerance approach (FixedGeom) taking the lead (training time being equal).
Critical strengths
This is a very interesting application niche for Neural ODEs. Traditional methods do not allow such explicit & simple regulation of a method's capacity, meaning that they cannot utilize the optimization theory insight into an error weight, that changes throughout training. Simple synthetic cubic data shows it well: in the absence of noise and the complexity of real data, scheduling mostly helps.
Critical weaknesses
Contribution. The tolerance scheduling idea already exists, as you state yourself in related work. Ott et al. (2021) already proposes to adaptively reduce tolerances throughout training, but they aim to achieve a different goal -- consistency with different solvers; to make the ODE an actual ODE -- not inspired by the "geometrically decaying loss weights" bound. That makes your bound-based motivation the main contribution and the novelty of this paper, which suffices, in my opinion. However, this bound motivation is only weakly supported by the presented evidence (theoretical and practical). Therefore, I believe that this work is incomplete without at least some confirmation of this mechanism, and some application of it. I would like to see some practical benefit which comes from this mechanism, turning it from just an interesting observation into a portable finding.
Then, some other observations:
- Your motivation does not explain, why the cosine schedule leads over the geometric one. For a schedule-proposing paper, I would like to see some schedule comparison and explanation there.
- Since tolerance is a hyperparameter, it should be selected & tuned per-dataset, so, at the moment, comparison to a fixed method which is not the leader (fixed-final, for instance) is not really fair: comparison should be against fixed-best, wherever you claim to outperform fixed methods.
Questions for authors
Contribution. I would like to provide an example of what I mean by "deeper analysis of the optimiser mechanism": measuring errors at different states during training, comparing that curve to what the theory predicts, making conclusions about the given datasets. Because, on real data, we observe that your scheduling loses to a carefully-tuned tolerance. Then, it would be interesting to see some additional modification of the integration procedure, inspired by that analysis.
A positive result here would make me raise my mark.
Besides, it would be preferable to fix the other observed weaknesses:
- Analyze why, how and where the schedules differ -- to produce some portable finding here as well.
- Compare your schedules to the fixed-best model to the text and abstract -- a fixed-final comparison is not fair.