BOLT: Blockwise Residual Ridge Solving for Frozen-Feature Classification

Frozen-feature classification repeatedly fits a prediction head on representations produced by a fixed encoder. Although closed-form ridge regression avoids iterative optimization, jointly fitting a wide random-feature head incurs expensive matrix factorizations and large temporary solver memory, while iterative classifier training requires repeated parameter updates. We introduce BOLT (Blockwise Optimizer-free Learning with cyclic refinemenT), a blockwise residual solver for this refitting task. BOLT first decomposes the wide solve into smaller blockwise problems through a sequential residual cascade, enabling a fast one-pass fit. Cyclic refinement then revisits these blocks to correct approximation errors and recover the joint ridge solution, while augmented QR improves numerical stability without forming normal equations. Across five frozen-feature vision benchmarks, BOLT maintains competitive predictive accuracy while substantially reducing peak solver memory compared with the matched joint solve.