I have multiple concerns about this submission.
- Novelty limitations: Many existing works (including ODEFormer) pose the problem in two stages consisting of structural candidates (e.g., ODEFormer also samples candidate structures as proposal structures) and selection + coefficient learning (again, ODEFormer and previous works it cites, also optimizes coefficients via an L2 loss across different sampled candidates and selects the optimal one). Of course, the scope for ODEFormer is quite different as it is not limited to linear combinations of basis functions (see next point). The idea of using amortized pre-training to select which candidates among a final library are present in a given expression with a second stage to actually select and form the expression is extremely common. For example, [1] from 2016 does just that to compose programs.
I understand that this is not directly related, but my main claim is that conceptually as well as methodologically, there is little novel insight. There is also no principled/theoretical motivation for this approach, i.e., it is a plausible idea to try, but whether it is a good idea entirely hinges on empirical performance (see next point).
I would also disagree with some specific statements: l.131 "providing no reusable, composable notion of candidate quality: a generator that misses one term..." Other methods to provide multiple proposals that are sampled from their internal distribution of plausible candidates. Since they are often not restricted to linear combinations of a fixed library (e.g., ODEFormer can nest non-linearities and combine them non-additively, etc) they don't have the strict notion of "an omitted term". However, even in linear combinations, it is not clear that "missing a term" is the relevant failure mode. In many ways, sampling entire compositions of terms according to model likelihoods is possibly more relevant than mere term selection.
[1] Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2016). Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989.
-
Scope limitations: The chosen scope on which the approach is tested and evaluated is too narrow. After some searching in the appendix, my current understanding is that the pre-trained support predictor is trained only on 2D systems with up to 4-th degree polynomials as basis functions. Within the scope of the existing literature, this is way too narrow. It also does not cover most of the chosen evaluation datasets such as ODEBench and MDBench (non-polynomial, many 3D equations, etc), which makes me wonder how this evaluation was even performed. There is a variety of other concerns in terms of too narrow training/evaluation choices such as the fixed regularization parameters in the loss (eq. 6), a single additive noise model with a single fixed variance (multiplicative and various variances are standard), not varying the number of 32 observations (most of this without justification, selection protocols, or ablations).
-
Evaluation: This is one of the main points. In "4.1 Corpora", the submission claims three public benchmarks.
- Strogatz: cites the Strogatz book, not the Strogatz dataset; which one/version is used here
- ODEBench: The cited work (Gilpin, 2021) does not introduce a dataset and does not contain the word ODEBench. I assume the authors meant to cite ODEFormer (d'Ascoli, 2023) which actually introduced the ODEBench dataset (this work was in ICLR 2024 however; in general it would be good to update the citations to where they actually appeared instead of the arxiv references)? ODEBench contains most of the systems from the Strogatz dataset as well (different initial conditions, but same systems).
- MDBench: The MDBench paper presents a benchmark consisting of PDEs and ODEs. Since PriSIFT only works for ODEs I assume that only the ODE part of MDBench is used? However, MDBench literally just copies the entire ODEBench as their ODE part.
Moreover, a substantial fraction of the systems in ODEBench are 3 or 4 dimensional, which is not covered by the pre-trained prior network. Is the evaluation restricted to 2D systems? How big is the overlap between Strogratz and ODEBench? With all these missing details (are those also sub-sampled to 32 observations, what are chosen noise levels, etc, etc), I cannot judge the reported empirical performance.
Finally, I could not find any concrete information about the claimed "20-system literature corpus curated ..." What are these systems, how have they been curated, from what sources exactly (only Lüders, 2022 is mentioned without further details). Appendix A.2 also only says 3 sentences about this including merely stating "seven canonical literature systems" and, oddly, that the "evaluation seeds 421000-421009 were frozen independently of the development seeds."
- Clarity: Large parts of the paper are extremely hard to follow. There are many verbal constructions that I cannot map to precise mathematical/modeling concepts. Unnecessary LLM-style contrasts and odd implementation references while at the same time lacking mathematical descriptions of relevant contents make this hard to judge and read. Just examples: in eq. (1) what are these losses (some are in the Appendix, but it would still be good to understand it in the main). In section 3.3 how does the point embedding work? It is also unclear to me in this part whether trajectories refer to different components of a single multi-variate trajectory, or trajectories from the same system for different initial conditions, or trajectories from different ODEs. In the latter case, the trajectory pooling does not make sense to me? In the Baselines. paragraph in 4.1, what does it mean that baslines are "converted into proposal heuristics". Are they not used as intended for direct system inference, but somehow squeezed into the existing proposal/fitting split? How? Why?
Artificial unnecessary contrasts like the one in l.319 confuse more than they help.
Finally, the Appendix exposes really awkard LLMy unnecessary (and confusing) code/implementation references. I don't think the checkpoint naming convention (l.639), the used seeds, or the specific SHA256 hashes of the "untouched corpus manifest" or the "degree-fice specialist" are really relevant in the paper.