FlashRLS: Exact Differentiable Second Order Online Learning

Second-order online methods adapt to streaming data through sequential matrix updates that are costly on GPUs. Existing chunked implementations handle squared loss, where update weights are known in advance. Logistic and robust losses intro- duce response-dependent weights, making both batching and exact differentiation more difficult. We present FlashRLS, a block formulation that runs local update rules on a small matrix of observation interactions and batches the dense work into matrix products. It covers recursive least squares (RLS), online Newton, recursive Laplace, and robust updates, and we prove that it reproduces sequential execution and its exact gradients. With the forward rule held fixed, exact differentiation improves associative recall by 9.4 points and reduces outlier-regression MSE from 0.085 to 0.057. FlashRLS matches sequential estimators with 38.8×, 42.3×, and 53.3× speedups on Criteo, echo cancellation, and chaotic-system readouts.