BOLT: Budget-aware Ordered Low-rank Training

Deep neural networks (DNNs) are a key driving force behind many modern artificial intelligence (AI) systems due to their powerful learning capabilities. However, the growing scale and complexity of DNNs demand substantial memory, computation, and infrastructure. This limits deployment on edge devices and restricts experimentation to well-resourced settings. Existing compression methods commonly use sparsity, reduced precision, or low-rank factorization, but often lack direct control over the final model size under a prescribed global parameter budget. We address this limitation by formulating model compression as a budget-constrained optimization problem and presenting Budget-aware Ordered Low-rank Training (BOLT), a training-time compression method that learns data-driven ordered low-rank parameterizations under an explicit parameter budget. BOLT factorizes eligible layers into ordered low-rank factors and uses supervised learning and soft structural self-transfer to concentrate information learned from the data into leading modes. It then applies a scale-invariant mode-strength criterion for ordering and estimates the loss impact of complete suffix removal using a first-order Taylor approximation and directional Gauss–Newton (GN) curvature. Sparse Pareto dynamic programming (DP) then selects layer specific prefix ranks under the global budget followed by replay buffer based output distribution drift validation. This allows BOLT to determine suitable ranks for each layer without predefined rank schedules or manual pruning thresholds. Experiments across convolutional and Transformer architectures show that BOLT performs competitively against seven baselines, including the dense model. BOLT achieves near-dense performance with only 10% of the original parameters on select architectures, enabling up to 90% model reduction for resource-constrained deployment.