- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper addresses the cost of choosing how to impose a physical law when learning the dynamics of systems that only approximately respect it. The architectural choice (hard constraint, soft penalty, or no constraint) is usually made experimentally. This paper argues this choice can instead be made before training: first, a scalar statistic measures, from the noisy training trajectories, how badly the law is violated, and two thresholds t1 and t2, computed experimentally, map which constraint we should use. The thresholds are fitted once per invariance class by grid search on a simple calibration system and then reused on other systems in the same class. A simple analysis gives a theoretical motivation for the three-regime structure as a bias–variance trade-off. Experiments on 12 synthetic PDEs with controlled law-breaking show that the selector reduces rollout error by about 12% relative to a fixed configuration while training a single model, but remains roughly 20% less accurate than exhaustive grid search.
Critical strengths
The first merit of the paper is proposing a solution to the aforementioned problem of choosing the right way of enforcing physical laws: instead of relying on a brute force scheme, the paper suggests a rule with a invariance class (per law) that is able to reduce the computational overhead (after calibration). The paper also tries to give a theoretical foundation and a better understanding of the technique presenting it as a bias variance tradeoff. Moreover, they design statistic that are experimentally able to predict which architectural choice we should use before the training itself.
Critical weaknesses
- the first weakness is that the central assumption of the paper is that the thresholds should generalize and be invariant but the paper does not justify this. No justification is given for why the thresholds should not be system dependent. The theoretical justification for the existence of 3 phases also shows that the thresholds, in that specific case, do depend on the system that is being analyzed. So what is proving the choice of the 3 phases regime has some fundamental rigor it is also suggesting that the 3 phases are not invariant within systems belonging to the same class.
- The evidence is confined to small synthetic experiments built to be measurable and also the generalization of the thresholds, given there is no theoretical proof of the invariance, is not extensively tested as this would need more cases for each class.
Questions for authors
It would be important to validate the paper showing more experimentally (given the absence of a theoretical understanding) how invariant are the classes. Moreover, given that the theory suggest the size of the problem has an impact on the thresholds, one should do a size study for 'all' the cases used in the paper. Moreover, it would also be important to validate this idea on larger experiments where this technique would be of major importance.