Overcoming Stiffness Barriers in Neural ODE Training

Vanishing sensitivities to fast decay rates can hinder shooting based training of stiff neural ODEs. We investigate how established representation and training techniques can be combined to address this difficulty. Specifically, we avoid differentiation through an initial value solve by combining a production rate parameterization with a derivative matching initialization and simultaneous collocation. Analytically, we show for a scalar decay model that transformed derivative matching retains a restoring gradient for overestimated rates and further that a joint collocation step maintains a nonvanishing log rate correction while the corresponding shooting step vanishes. Experimentally, we observe on six stiff benchmarks benefits from pairing the architecture and training procedure, including improved spectral recovery on Van der Pol and the Oregonator, although recovery remains mixed across systems. We also show via counterexample a separate identification limit: exact trajectory and derivative fits don't imply identification of fast Jacobian eigenvalues. Together, these results present an avenue for addressing vanishing sensitivities through training formulation, while showing why improvements in optimization must be evaluated separately from exact recovery of the underlying dynamics.