- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors establish universal approximation of autonomous dynamical systems with the Kuramoto system. The authors first establish that the broader lagged fast-slow system (LFS), which allows for a per oscillator phase lag parameter as well as a time scale separation between two sets of oscillators can approximated by the Kuramoto system. This is given in Lemma 4.1 and constructs a system of 3 non-phase lagged oscillators whose attracting equilibria in the final oscillator gives the phase lagged oscillator and a time scaling on the system establishes convergence to that equilibria, giving an error term.
The authors then show that the LFS “hidden state” system has an attracting equilibria given by a “Normalized Affine Vector” and whose convergence to this equilibria is governed by the LFS time scale . The “visible state” system is then shown to be a projection of the complex linear combination of NAVs onto the direction tangent to the unit circle at each visible state’s phase where the NAVs are determined by the hidden state system. Writing a target dynamical system’s velocity field as the projection of a velocity field into the direction tangent to the unit circle at each visible state’s phase reduces the problem to showing the NAVs converge to .
This convergence is established with density with Prop 4.1 and exploits density of complex exponentials (fourier series). The proof shows that the fourier basis can be approximated in the span of finite NAVs and gives a bound from the fourier series error and the finite span NAV error. The authors then apply an existing lemma relating vector field convergence to trajectory convergence and use that in combination with density of NAVs (prop. 4.1) and LFS convergence to the NAV equilibria (Lemma 4.2) in to give a universal approximation theorem for LFS systems in Prop. 4.2. Then finally a universal approximation theorem for the Kuramoto model (on trajectories) is established in Theorem 3.1 with the inclusion of the kuramoto’s convergence to equilibria LFS system of Lemma 4.1. The sine specific couplings of Theorem 3.1 and Prop. 4.2 are generalized to a general periodic coupling universal approximation theorem in Theorem 4.1. Section 5 then examines some experiments.
Critical strengths
-
Universal approximation results of Prop. 4.2, Theorem 3.1 and Theorem 4.1 seem to be very novel and quite interesting. They also seem quite relevant and fundamental if the kuramoto model can be embedded in hardware and offer a new computing framework.
-
The proof techniques are quite different from conventional universal approximation results as they require convergence to attracting equilibria.
-
The overall paper is well organized and the inclusion of proof sketches was useful.
Critical weaknesses
-
Expositional clarity for theoretical results needs improvement. The proofs, both in the appendix and the main text where the sketches appear could be written with more clarity and the appearance of certain terms/constructions could be given more clarity. For example in the proof of Lemma 4.1 it is not immediately obvious why specifically three additional oscillator need to be added per added phase lag oscillator. It becomes clear in the appendix when this gives contraction of to the phase lag through a trig identity. Additionally, should appear in the proof sketch. The inductive argument for the bounds is hard to follow and could be expanded to be made more clear in the appendix. The proof of Lemma 4.2 in B.2 should justify the form of Eq. 4.10 in the proof sketch. Lemma B.1 and its use to show every finite combination of complex exponentials belongs to was hard to follow. More exposition on details such as these could be useful to a reader.
-
No dependence on the oscillator network size discussed. The final convergence bound in the main text does not sufficiently discuss its dependence on the Kuramoto model’s system dimension. From my reading determines the number of terms in the NAV which determines the number of number of hidden variable in the LFS system which determines the number of variables in the Kuramoto system (scaling the dimension of lagged variables by 3). I wouldn’t say that an exact dependence in the bound would be required, though that might be interesting, as it would potentially require further results, however, some additional discussion or hypotheses for how the bound of Theorem 3.1 depends on would be helpful.
Questions for authors
-
Is it ever the case that a target dynamical system’s velocity field can be written as the projection of a velocity field into the direction tangent to the unit circle at each visible state’s phase?
-
What are the specific training details for the dynamical systems fitting. I assume this is fitting via an loss on the trajectories, as the authors mention , Is the Kuramoto model integrated and gradients are computed through integration?
-
The proof strategy of Proposition 4.1 seems like it could potentially be tightened, Is an appeal to density of complex exponentials necessary? Could the result be determined directly? I ask this since it might simplify the resulting convergence term and possibly allow that term to be parameterized by the number of NAVs.
-
What are realistic time scale separations? Are the time scale separations that appear in the kuramoto model and LFS approximation bounds realistic?
-
I have already mentioned this but what do the authors suspect the dependence of the bound on is?
2 comments
Response to rebuttal comment
Thank you for the clarifications.
For question 1, I actually had a typo in my original question and is self-resolved. My main concerns involved the results regarding the tightness of prop 4.1 and the dependence of the number of oscillators, outside of exposition. I maintain my score.
Thank you for your detailed comments. We will improve the clarity of the proofs in the next revision. We also consider the dependence of approximation rates on model size an important direction for future work. We answer your questions below:
Could you please provide more details on the specific aspect you are interested in? From my current understanding, your description matches how we construct the target field. Set Ti(v)=−ieiviFi(v). Then Im(eiviTi(v))=Fi(v) exactly, and no extra assumption on the target field is needed at this step.
We train on vector field MSE, ∥F(vr)−F(vr)∥22, rather than a trajectory loss. We sample states vr uniformly from the domain and fit the NAV reduced field to the target velocities F(vr). The horizon t1=0.5 is used only after training to simulate and evaluate the fitted system. Training therefore does not require gradients computed through integration. Our theoretical analysis provides a potential advantage over existing ML work using the Kuramoto model: They use BPTT to learn couplings (Wang et al., Unconventional AI) while we remove this training overhead in the cost of adding time-scale separation in hardware. Balancing these training and hardware costs is future work.
& 5. We currently do not have a tighter construction to claim. We expect approximation rate arguments to require a different route than the universal approximation theorem and require restricting the target function to specific classes, similar to those in neural networks, e.g., Barron.
The practical separation is task-dependent and can be potentially reduced by better training and model architecture. For example, we ran a follow-up experiment on the same distributional fitting tasks and reduced the required time-scale separation by 1000$\times$ with a two-layer NAV architecture. Please refer to CQ1 for practical hardware consideration and CQ2 for time-scale separation experiment details.