Online Newton Learning for Multi-Output Nonlinear Models under Non-i.i.d. Data

We study online learning for a class of multi-output nonlinear stochastic regression models under non-i.i.d. data and general convex losses. This problem arises in several representative settings, including multi-output linear regression, learning from component-wise saturated observations, and multiclass softmax classification under the KL-divergence loss. We develop an adaptive Newton-type estimation algorithm that combines observation-driven gradients, recursively accumulated second-order information, and local curvature of the nonlinear function, allowing its updates to adapt to the evolving geometry of sequential data. We then establish parameter convergence without imposing i.i.d. data assumptions, requiring only an information-growth condition that matches the minimal requirement for least- squares estimation in linear stochastic regression. Moreover, we show that the average prediction regret decays at a logarithmic-over-linear rate without requiring additional excitation on the data. Experiments on synthetic and real-world datasets corroborate the theoretical results and demonstrate the effectiveness of the proposed method.