acceptodds
Under review as a conference paper at ICLR 2027

BOLT: Budget-aware Ordered Low-rank Training

Abstract

Deep neural networks (DNNs) are a key driving force behind many modern artificial intelligence (AI) systems due to their powerful learning capabilities. However, the growing scale and complexity of DNNs demand substantial memory, computation, and infrastructure. This limits deployment on edge devices and restricts experimentation to well-resourced settings. Existing compression methods commonly use sparsity, reduced precision, or low-rank factorization, but often lack direct control over the final model size under a prescribed global parameter budget. We address this limitation by formulating model compression as a budget-constrained optimization problem and presenting Budget-aware Ordered Low-rank Training (BOLT), a training-time compression method that learns data-driven ordered low-rank parameterizations under an explicit parameter budget. BOLT factorizes eligible layers into ordered low-rank factors and uses supervised learning and soft structural self-transfer to concentrate information learned from the data into leading modes. It then applies a scale-invariant mode-strength criterion for ordering and estimates the loss impact of complete suffix removal using a first-order Taylor approximation and directional Gauss–Newton (GN) curvature. Sparse Pareto dynamic programming (DP) then selects layer specific prefix ranks under the global budget followed by replay buffer based output distribution drift validation. This allows BOLT to determine suitable ranks for each layer without predefined rank schedules or manual pruning thresholds. Experiments across convolutional and Transformer architectures show that BOLT performs competitively against seven baselines, including the dense model. BOLT achieves near-dense performance with only 10% of the original parameters on select architectures, enabling up to 90% model reduction for resource-constrained deployment.

Primary area:
optimization
Keywords:
Model compression, neural network compression, data driven compression, ordered low rank learning, budget constrained rank selection, parameter efficient learning, model compaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XJVBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BOLT learns ordered low-rank representations and compresses models by retaining layer-wise prefixes under a global parameter budget. It combines prefix training and self-distillation with curvature-based estimates of compression damage. Dynamic programming allocates ranks across layers, followed by replay-based checks on loss and output drift before accepting a compressed configuration.

Critical strengths
  1. Incorporating a global parameter budget directly into layer-wise rank allocation is more systematic than manually assigning compression ratios and makes model-size requirements explicit.
  2. The training and compression procedures are well connected. Prefix training and self-distillation encourage leading components to retain predictive information, providing a reasonable basis for subsequent truncation.
  3. Loss and curvature information guide global rank allocation, while the CE/KL checks provide an additional empirical safeguard.
Critical weaknesses
  1. Algorithm 1 initializes the final configuration at full rank and updates it only when a candidate satisfies both the budget and the CE/KL criteria. If no candidate passes, the returned model may exceed the budget. Assumption 1 in Appendix C assumes that an acceptable candidate exists, but the algorithm does not establish this or specify a fallback.

  2. BOLT builds substantially on prior ordered low-rank training, and its additional complexity should be justified against competitive alternatives. Table 1 mainly compares BOLT with dense models, which does not establish an advantage over other low-rank methods at larger scales.

  3. Figure 4 shows lower FLOPs and peak memory but higher P95 inference latency than the dense baseline. Prefix training, curvature estimation, dynamic programming, and candidate validation also introduce additional training costs that are not fully quantified.

  4. The large degradation in the ablations requires further explanation. In Figure 5, removing self-distillation reduces accuracy by 71.2 and 80.4 percentage points. This shows that the components are tightly coupled, but does not by itself establish an advantage over simpler alternatives. Retuned, budget-matched ablations would make these results more informative.

  5. The text in Table 1 and Figures 4 and 5 is quite small and difficult to read. For Table 1 in particular, I am also concerned that the font size may not comply with the ICLR 2027 formatting guidelines.

Questions for authors

Does BOLT retain a clear advantage over competitive low-rank baselines when both the final parameter count and total training compute are matched?

1 comment
AuthorsOpenReview ↗

Solution Candidate, Novelty, Costs, Degradation Clarification

We thank the reviewer. New experiments addressing the concerns are included in the revision.

Solution Candidate and Fallback. The reviewer is right that BOLT could end above budget if no candidate passed the replay checks. In our new three-seed sensitivity study, this occurred only with strict KL limits (ϵKL≤0.005\epsilon_{KL}\le0.005). With the paper's limits, every run met its budget, including ten new runs targeting ratios 0.022–0.30. We address this gap with a conditional step after the candidate loop in Algorithm 1. If at epoch EfbE_{\mathrm{fb}} (E−10E-10 here) the configuration still exceeds BB, BOLT accepts the budget-feasible candidate with the largest PphysP_{\mathrm{phys}} from the same Pareto DP set (ties broken by lowest surrogate damage), compacts it, and continues substructure training. This set is non-empty whenever Pfixed+∑ℓcℓkℓmin⁡≤BP_{\mathrm{fixed}}+\sum_\ell c_\ell k_\ell^{\min}\le B (Assumption 1). Existing steps, hyperparameters and replay acceptance rules remain unchanged; the fallback is skipped once the budget is met. Revised Algorithm 1 (Appendix A) and the new runs (Appendix I.5) documents this.

Advantage over low-rank methods. BOLT jointly addresses three challenges: preserving accuracy under truncation, scoring components despite scale non-identifiability, and meeting a prescribed model-size budget. It combines prefix consistency, scale-invariant mode strength (Fig. 1), and exact budget-constrained rank allocation without threshold tuning or monotonic sensitivity assumptions, giving direct control over the final footprint. New budget-matched comparisons with MAESTRO on ResNet-18 show higher accuracy on CIFAR-100 and CIFAR-10 and comparable accuracy on three other datasets (mean ±\pm std).

DatasetRatioBOLTMAESTRO
CIFAR-1000.06369.83 ±\pm 0.2365.48 ±\pm 0.18
CIFAR-100.04191.11 ±\pm 0.1790.39 ±\pm 0.24
F-MNIST0.02894.08 ±\pm 0.1593.93 ±\pm 0.22
MNIST0.02299.59 ±\pm 0.0299.61 ±\pm 0.05
SVHN0.03096.48 ±\pm 0.0696.41 ±\pm 0.08

On VGG-19, MAESTRO pushed toward ratio 0.10 reaches 87.70% at 0.169, while BOLT reaches 90.44% at 0.099 (response to hn2S). Table 1 demonstrates large-scale applicability rather than superiority over low-rank alternatives at that scale and we clarified this distinction in the revision.

Cost. Fig. 4 shows higher P95 latency despite lower FLOPs and peak memory. As discussed in lines 408–413, replacing one dense convolution with two factorized convolutions reduces arithmetic but adds kernel launches, intermediate activations and memory traffic. These overheads can outweigh the savings, particularly at smaller batch sizes. This trade-off is shared by factorized methods but remains a practical limitation of BOLT (Sec. 6).

BOLT takes 0.5–0.8 L40S GPU hours per run on ResNet-18/CIFAR-10, compared with 0.2–0.3 for MAESTRO. However, budget targeting may require multiple runs. MAESTRO’s six runs consumed 1.84 GPU hours but missed the target ratio of 0.10 whereas BOLT met every tested target in one run per seed (response to hn2S). BOLT therefore trades higher cost per run for lower cumulative cost when targeting a budget. We added a comparative training-cost results to Appendix I.2.

Degradation Clarification. We agree. The Fig. 5 soft-transfer arm removed the prefix-consistency loss and positional ordered dropout together, since the sampler acts only inside that loss, and compared a fixed 10% cut of every layer against the full-BOLT accuracy. The bar therefore reports the coupled mechanism as in Eq. 6 (we have updated Fig.5 accordingly). To separate the roles (also requested by JFJa, see our response there), we ran ablations targeting ρB=0.10\rho_B=0.10. After warm-up, retaining 50% of the modes in every layer gives 85.3% with BOLT's positional sampler, 60.4% with uniform-prefix sampling and 45.0% without soft transfer. Replacing the ordered sampler thus costs 24.9 pp, and removing soft transfer costs 40.3 pp, which is why disrupting the coupled mechanism is so damaging.

Formatting. We have enlarged the text in Table 1 and Figs. 4 and 5 in the revision.

Q (Matched parameters and compute). All baselines used equal epochs and identical data splits. At matched parameters, BOLT outperforms MAESTRO on CIFAR-100 and CIFAR-10 and is comparable on three other datasets (response to hn2S). Despite higher per-run cost, BOLT met every tested target in one run; MAESTRO’s six-run sweep missed the target and used more total GPU-hours. This shows BOLT’s practical advantage in budget control.

Code for all new experiments, including the fallback step, has been released with the revision. In light of these results and clarifications, we hope the reviewer would reconsider the assessment.

Reviewer_JFJaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper worked on low rank decomposition of DNNs to achieve faster inference. It decomposed parameter arrays by SVD, and then added ordered rank dropout to enforce learning towards clustering more info to low ranks in a order. A self distillation like approach (named as "self-transfer") is also used during low rank learning. Taylor approximation and directional Gauss–Newton (GN) curvature are used finally to get optimal ranks. These are old techniques and experiments are done at small scale. The interesting part is if the ordered rank dropout can help it to achieve lower rank in SVD, which is not well ablated

Critical strengths

None

Critical weaknesses
  • ablation on how the ordered rank dropout/masking can help it to achieve lower rank in SVD
Questions for authors

How does the method generalize to larger scale data and models

3 comments
AuthorsOpenReview ↗

Component Ablation, Contributions, Clarifications on Large Scale Data

We thank the reviewer. New experiments addressing each concerns are included in the revision.

Ablation of ordered rank dropout. The soft-transfer objective (Eq. 6) combines a prefix-consistency loss and ordered dropout, which selects the prefixes trained. We ablated each separately (ResNet-18, CIFAR-10, ρB=0.1\rho_B=0.1), changing only substructure training: no soft transfer (no prefix loss, no ordered dropout), uniform-prefix ordered dropout (nested, prefix length uniform), non-nested dropout (independent per-mode masks, same keep rates), and BOLT's positional ordered dropout.

(1) What the ordering puts in the leading modes. After warm-up, before rank selection, we retain the first fraction of modes in every layer:

Modes keptPositional (BOLT)Non-nestedUniform-prefixNo soft transfer
50%85.384.560.445.0
30%75.460.318.912.9
20%50.322.810.411.1

Without soft transfer, information is spread across all modes and accuracy collapses as modes are removed. With it, the full model distils its predictions into sampled substructures, helping leading modes preserve the function and making suffixes removable. Positional nesting gives the highest leading-prefix accuracy.

(2) Whether the model then reaches a low rank. Every variant uses BOLT's unchanged rank selection (progressive projection with GN damage estimates and CE/KL checks, final allocation) targeting ρB=0.1\rho_B=0.1. BOLT reaches the budget with accuracy 92.13 ±\pm 0.14% (three seeds), and non-nested dropout reaches it with 91.76%, about 0.4 pp lower. Uniform-prefix ordered dropout and no soft transfer fail to reach it through accepted projections, since replay checks reject most candidates at every epoch, and end at ratios 0.108 and 0.166. Forcing the no-soft-transfer model to the budget with the fallback step gives 87.23% (three seeds), 4.9 ±\pm 0.4 pp below full BOLT. In table (1), at half the modes replacing positional ordered dropout costs 24.9 pp (uniform-prefix) and removing soft transfer 40.3 pp; at 20% of modes the gaps are 39.9 and 39.2 pp, respectively.

The prefix-consistency loss helps reach the budget through accepted projections, and positional nesting gives the strongest ordering of leading modes. Non-nested dropout is comparable after selection, so the loss is the primary driver. We added this ablation as Appendix I.3.

Contributions. We agree that ordered dropout, Taylor and Gauss Newton damage estimates, and dynamic programming exist individually in prior work. As reviewers XJVB and hn2S noted, our contribution lies in systematically integrating these components into a practical solution for model compression. BOLT addresses three challenges in making ordered low-rank training reliably meet a specified budget. First, reliable component scoring during training: raw σ~\tilde{\sigma} is scale non-identifiable (Fig. 1). BOLT uses scale-invariant mode strength, which achieves the highest mean accuracy in our new scorer comparison (86.9% vs. 85.1% for GN at ρB=0.1\rho_B=0.1; response to hn2S). Second, direct control of final model size: in the training-time baselines we compare, size emerges from regularization and threshold choices rather than an explicit budget. MAESTRO's authors acknowledge that these hyperparameters do not guarantee the final footprint, as our budget-targeting sweep illustrates (response to hn2S). BOLT scores each candidate suffix as a whole, requiring no monotonic loss sensitivity assumption, and jointly selects layer-specific ranks through exact allocation under a global budget, meeting feasible targets in a single run. Third, preserving predictive information under truncation: BOLT trains sampled substructures through prefix consistency. Without this supervision, projections accepted by the replay checks miss the budget (ablation above). Together, these mechanisms enable low-rank training that takes a feasible budget and returns a model meeting it across CNNs and Transformers with only a loss specification. New budget-matched experiments (response to hn2S) show that at MAESTRO's own compression ratios, BOLT is more accurate on CIFAR-100 and CIFAR-10 and comparable on the other three datasets. We have also released the code for reproducibility.

Q (Scale) Table 1 includes ImageNet-1K (1.28M training images, 1000 classes) with ViT-Huge/14 (632M parameters, compressed to 190M, 74.22% Top-1) and DINOv2-Giant (1.14B parameters, compressed to 695M, 80.26% Top-1), and M2M-100 (1.2B parameters) for machine translation. Fig. 6 shows the compressed models transfer well to new tasks.

Code for all new experiments has been released. We hope these ablation experiments and scale clarification address the concerns and would be grateful if the reviewer would reconsider their assessment.

Thanks for the feedback. I would keep the original rating as my concerns haven't been fully resolved.

We thank the reviewer for the response. Could you clarify which aspect of the original concern remains unresolved? Appendix I.3 shows that retaining 30% of modes gives 75.4% accuracy with positional dropout, compared with 60.3% for non-nested and 18.9% for uniform-prefix dropout. Thus, positional dropout helps leading modes preserve predictions, enabling subsequent rank reduction with less damage. It does not directly reduce rank. Non-nested dropout also reaches the budget, so our claim is stronger prefix preservation, not that positional nesting is necessary.

Reviewer_hn2SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Budget-aware Ordered Low-rank Training (BOLT), a training-time compression framework that selects layer-wise ranks under a global parameter budget. BOLT represents eligible weight matrices as ordered sums of learnable rank-one components and trains randomly sampled prefix subnetworks to match the full model’s predictions through self-distillation. After warm-up, it reorders components by a scale-invariant mode-strength measure, estimates the loss change from suffix removal using first-order Taylor terms and within-layer directional Gauss–Newton curvature, and selects prefix ranks by solving a multiple-choice knapsack problem with sparse Pareto dynamic programming at each epoch. Candidates that satisfy the budget and pass replay-based cross-entropy and KL-divergence checks are physically compacted during training. Experiments are performed on CNN and Transformer architectures across vision and language tasks and show comparable accuracy with dense models with reduced FLOPs and peak inference memory but higher inference latency.

Critical strengths
  1. Explicit parameter-budget formulation. BOLT accounts for both fixed parameters and rank-dependent costs in selecting layer-wise prefix ranks to introduce explicit model-size constraint. It combines suffix-level damage estimates with global budget allocation and checks the actual loss and prediction drift on replay data before physical compaction. This provides a practical mechanism for targeting a specified model size while assessing the immediate impact of compression.

  2. Promising compression–accuracy trade-offs on CNNs. Relative to dense models trained under the same protocol, BOLT preserves near-dense accuracy with about 10% of the parameters in several settings. For example, it reaches 92.37% accuracy versus 92.69% for the dense model with ResNet-18 on CIFAR-10. The main ResNet-18 and VGG-19 results are averaged over five seeds.

Critical weaknesses
  1. Insufficient comparison against adaptive rank-allocation methods. Adaptive rank allocation has been studied in low-rank compression, including SVD-LLM V2 [R1], Dobi-SVD [R2], and ZS-SVD [R3]. ZS-SVD is particularly relevant because it uses first-order calibration-loss estimates for global singular-component selection. Although these methods differ from BOLT’s training-time setting, their relationship to BOLT’s budget formulation and loss-guided rank selection is not adequately addressed. The paper should clarify the additional contribution of suffix-level directional Gauss–Newton estimation and its coupling with ordered-prefix training. The benefit over existing compression approaches is not yet established, considering that there is no baseline for comparison in the experiments on Transformers in Table 1.

  2. Mismatched budgets and unclear evaluation protocols. The CNN comparisons largely report individual operating points at substantially different parameter budgets. For example, on ResNet-18/CIFAR-10, BOLT retains 10% of the parameters while MAESTRO retains 4.1%. This comparison does not identify their relative accuracy at a common model size. For MAESTRO, the regularization and pruning controls should be tuned to obtain comparable operating points, or budget–accuracy curves should be reported. In addition, assigning IMP and XNOR a compression ratio of 1.0 based on dense parameter counts does not capture their sparse or low-bit storage benefits; storage costs and execution assumptions should be reported separately. Figure 8 also reports test accuracy alongside the discussion of hyperparameter selection, requiring clarification of whether selection was performed exclusively on validation data.

  3. Limited validation of the damage estimator. The “GN OFF” ablation suggests that curvature information is useful, but does not isolate the benefit of interactions among removed modes. A comparison among full within-layer GN, diagonal GN, and a simple normalized mode-strength score, holding the learned representation, parameter budget, solver, and replay checks fixed, would test this design choice more directly. The paper should also report prediction errors and ranking correlations between estimated and observed loss changes for joint multi-layer truncations. This is particularly important at aggressive compression ratios, where deleting entire suffixes constitutes a large perturbation and omitted cross-layer interactions may matter. The theoretical results do not resolve this uncertainty. Exact DP optimization concerns the surrogate objective, while the true-loss regret bound depends on approximation-error terms that are not evaluated.

[R1] Xin Wang et al. “SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression.” NAACL, 2025.

[R2] Qinsi Wang et al. “Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives.” ICLR, 2025.

[R3] Ali Abbasi et al. “Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression.” arXiv:2602.02848, 2026.

Questions for authors
  1. Figure 2(b) shows intermediate parameter ratios, such as 0.527 and 0.333, before reaching the target. How are these configurations produced, given that Algorithm 1 only accepts candidates satisfying the final budget (B)?

  2. How were MAESTRO’s regularization strength and pruning threshold selected? Could you report its performance at a parameter ratio close to BOLT’s 0.10 on ResNet-18/CIFAR-10, with hyperparameters selected using validation data?

1 comment
AuthorsOpenReview ↗

Budget-matched comparisons, Damage estimator validation, Clarifications

We thank the reviewer for the feedback. New results are includeded in the revision.

Related methods. SVD-LLM V2, Dobi-SVD and ZS-SVD are post-training methods that truncate the SVD of fixed pretrained weights, where singular components are well defined. BOLT learns low-rank structure during training, where factors are scale non-identifiable and raw σ~\tilde{\sigma} is unreliable (Fig. 1), motivating our scale-invariant mode strength and soft-transfer objective. Each suffix is scored as a whole, requiring no monotonic sensitivity assumption or threshold sweep; budgeted allocation is solved exactly (Thm. 1). Our baselines span 3 compression families with reproducible training-time methods, including MAESTRO, the closest ordered low-rank method.

Budget mismatch. MAESTRO's size depends on regularization and threshold, while BOLT takes a budget, so we matched budgets in both directions. (i) BOLT at MAESTRO's achieved ratios on ResNet-18 (mean ±\pm std).

DatasetRatioBOLTMAESTRO
CIFAR-1000.06369.83 ±\pm 0.2365.48 ±\pm 0.18
CIFAR-100.04191.11 ±\pm 0.1790.39 ±\pm 0.24
F-MNIST0.02894.08 ±\pm 0.1593.93 ±\pm 0.22
MNIST0.02299.59 ±\pm 0.0299.61 ±\pm 0.05
SVHN0.03096.48 ±\pm 0.0696.41 ±\pm 0.08
BOLT met all the matched target in one run.

(ii) We swept MAESTRO toward ratio 0.10 on CIFAR-10 (six group-lasso values, selected by ratio only). On ResNet-18 both methods trace similar curves. MAESTRO's closest run (ratio 0.122) reaches 92.69% and BOLT at ratio 0.098 reaches 92.42%. The 0.27 pp gap is within seed variation, and MAESTRO is 24% larger. On VGG-19, MAESTRO in Fig. 3 retained 75.8% of the parameters. Pushed toward 0.10, its closest run reached 87.70% at ratio 0.169, below BOLT's 90.44% at 0.099, and stronger regularization collapsed it to chance. Adjacent MAESTRO settings jumped from 0.122 to 0.064 (ResNet-18) and 0.169 to 0.022 (VGG-19), skipping 0.10, whereas all BOLT runs with targets from 0.022 to 0.30 met their target in one run without exceeding it. These results are given in Appendix I.2.

For IMP and XNOR we report dense-shape parameter counts and added bit-width storage columns to Table 6. All experiments use one fixed setting (λsoft=1\lambda_{\mathrm{soft}}=1 and the CE/KL limits of Table 7), chosen beforehand without test data; Fig. 8 only probes sensitivity around it. We re-ran it with three seeds on a held-out validation split. Among settings meeting the budget, mean validation accuracy varies by at most 0.2 pp, and the paper setting is within one standard deviation of the best. Revised Fig. 8 and I.1 reports this.

Damage estimator. With representation, budget, calibration and checks fixed, we compared four scorers over three seeds at ρB=0.1\rho_B=0.1, measuring Spearman between predicted and observed loss change on 200 random joint truncations.

ScorerSpearman at ratio ≈ 0.5ratio ≈ 0.1Acc. at ρB=0.1\rho_B=0.1
Full GN0.80 ±\pm 0.130.05 ±\pm 0.0685.13 ±\pm 1.53
Diagonal GN0.74 ±\pm 0.140.07 ±\pm 0.1083.34 ±\pm 2.89
First-order0.36 ±\pm 0.310.09 ±\pm 0.2983.44 ±\pm 3.03
Mode strength0.80 ±\pm 0.100.09 ±\pm 0.0186.87 ±\pm 0.08

At moderate compression, curvature scorers rank far better than first-order (0.80 vs. 0.36), with mode strength on par. Compression toward ρB=0.1\rho_B=0.1 passes through ratio 0.5, where full GN leads first-order by 0.5 pp, a gap gone by 0.3. In a single selection at ρB=0.1\rho_B=0.1, full GN gains 1.69 pp over first-order. As the reviewer anticipated, no scorer ranks reliably under aggressive truncation, and scale-invariant mode strength gives the highest mean accuracy and lowest variability. We present the findings in Appendix I.4.

Q1 (Fig. 2b). Fig. 2a shows the per-layer ranks BOLT selects during training by each layer's estimated contribution: deeper layers (layer4.0.conv2, layer4.1.conv1) are compressed heavily, conv1 is preserved, layer2 and layer3 shrink gradually. Fig. 2b shows the resulting total ratio. After warm-up, ranks are reduced progressively with each accepted configuration until the target is reached. The initial 1.133 arises because full-rank factors (U,σ,V)(U,\sigma,V) exceed the dense parameter count.

Q2 (MAESTRO tuning). Its default group-lasso strength (1.6×10−51.6\times10^{-5}) gave no compression (ratio 1.009), so we used 5×10−45\times10^{-4} (ResNet-18) and 3.2×10−53.2\times10^{-5} (VGG-19) with threshold 10−510^{-5} for both, since the profile value 5×10−65\times10^{-6} also gave none (we added the sweep in Appendix I.2).

Code for all new experiments has been provided (inside separate rebuttal_experiments folder). We hope these results address the concerns and would appreciate reconsideration of the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.