- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
BOLT learns ordered low-rank representations and compresses models by retaining layer-wise prefixes under a global parameter budget. It combines prefix training and self-distillation with curvature-based estimates of compression damage. Dynamic programming allocates ranks across layers, followed by replay-based checks on loss and output drift before accepting a compressed configuration.
Critical strengths
- Incorporating a global parameter budget directly into layer-wise rank allocation is more systematic than manually assigning compression ratios and makes model-size requirements explicit.
- The training and compression procedures are well connected. Prefix training and self-distillation encourage leading components to retain predictive information, providing a reasonable basis for subsequent truncation.
- Loss and curvature information guide global rank allocation, while the CE/KL checks provide an additional empirical safeguard.
Critical weaknesses
-
Algorithm 1 initializes the final configuration at full rank and updates it only when a candidate satisfies both the budget and the CE/KL criteria. If no candidate passes, the returned model may exceed the budget. Assumption 1 in Appendix C assumes that an acceptable candidate exists, but the algorithm does not establish this or specify a fallback.
-
BOLT builds substantially on prior ordered low-rank training, and its additional complexity should be justified against competitive alternatives. Table 1 mainly compares BOLT with dense models, which does not establish an advantage over other low-rank methods at larger scales.
-
Figure 4 shows lower FLOPs and peak memory but higher P95 inference latency than the dense baseline. Prefix training, curvature estimation, dynamic programming, and candidate validation also introduce additional training costs that are not fully quantified.
-
The large degradation in the ablations requires further explanation. In Figure 5, removing self-distillation reduces accuracy by 71.2 and 80.4 percentage points. This shows that the components are tightly coupled, but does not by itself establish an advantage over simpler alternatives. Retuned, budget-matched ablations would make these results more informative.
-
The text in Table 1 and Figures 4 and 5 is quite small and difficult to read. For Table 1 in particular, I am also concerned that the font size may not comply with the ICLR 2027 formatting guidelines.
Questions for authors
Does BOLT retain a clear advantage over competitive low-rank baselines when both the final parameter count and total training compute are matched?
Solution Candidate, Novelty, Costs, Degradation Clarification
We thank the reviewer. New experiments addressing the concerns are included in the revision.
Solution Candidate and Fallback. The reviewer is right that BOLT could end above budget if no candidate passed the replay checks. In our new three-seed sensitivity study, this occurred only with strict KL limits (ϵKL≤0.005). With the paper's limits, every run met its budget, including ten new runs targeting ratios 0.022–0.30. We address this gap with a conditional step after the candidate loop in Algorithm 1. If at epoch Efb (E−10 here) the configuration still exceeds B, BOLT accepts the budget-feasible candidate with the largest Pphys from the same Pareto DP set (ties broken by lowest surrogate damage), compacts it, and continues substructure training. This set is non-empty whenever Pfixed+∑ℓcℓkℓmin≤B (Assumption 1). Existing steps, hyperparameters and replay acceptance rules remain unchanged; the fallback is skipped once the budget is met. Revised Algorithm 1 (Appendix A) and the new runs (Appendix I.5) documents this.
Advantage over low-rank methods. BOLT jointly addresses three challenges: preserving accuracy under truncation, scoring components despite scale non-identifiability, and meeting a prescribed model-size budget. It combines prefix consistency, scale-invariant mode strength (Fig. 1), and exact budget-constrained rank allocation without threshold tuning or monotonic sensitivity assumptions, giving direct control over the final footprint. New budget-matched comparisons with MAESTRO on ResNet-18 show higher accuracy on CIFAR-100 and CIFAR-10 and comparable accuracy on three other datasets (mean ± std).
On VGG-19, MAESTRO pushed toward ratio 0.10 reaches 87.70% at 0.169, while BOLT reaches 90.44% at 0.099 (response to hn2S). Table 1 demonstrates large-scale applicability rather than superiority over low-rank alternatives at that scale and we clarified this distinction in the revision.
Cost. Fig. 4 shows higher P95 latency despite lower FLOPs and peak memory. As discussed in lines 408–413, replacing one dense convolution with two factorized convolutions reduces arithmetic but adds kernel launches, intermediate activations and memory traffic. These overheads can outweigh the savings, particularly at smaller batch sizes. This trade-off is shared by factorized methods but remains a practical limitation of BOLT (Sec. 6).
BOLT takes 0.5–0.8 L40S GPU hours per run on ResNet-18/CIFAR-10, compared with 0.2–0.3 for MAESTRO. However, budget targeting may require multiple runs. MAESTRO’s six runs consumed 1.84 GPU hours but missed the target ratio of 0.10 whereas BOLT met every tested target in one run per seed (response to hn2S). BOLT therefore trades higher cost per run for lower cumulative cost when targeting a budget. We added a comparative training-cost results to Appendix I.2.
Degradation Clarification. We agree. The Fig. 5 soft-transfer arm removed the prefix-consistency loss and positional ordered dropout together, since the sampler acts only inside that loss, and compared a fixed 10% cut of every layer against the full-BOLT accuracy. The bar therefore reports the coupled mechanism as in Eq. 6 (we have updated Fig.5 accordingly). To separate the roles (also requested by JFJa, see our response there), we ran ablations targeting ρB=0.10. After warm-up, retaining 50% of the modes in every layer gives 85.3% with BOLT's positional sampler, 60.4% with uniform-prefix sampling and 45.0% without soft transfer. Replacing the ordered sampler thus costs 24.9 pp, and removing soft transfer costs 40.3 pp, which is why disrupting the coupled mechanism is so damaging.
Formatting. We have enlarged the text in Table 1 and Figs. 4 and 5 in the revision.
Q (Matched parameters and compute). All baselines used equal epochs and identical data splits. At matched parameters, BOLT outperforms MAESTRO on CIFAR-100 and CIFAR-10 and is comparable on three other datasets (response to hn2S). Despite higher per-run cost, BOLT met every tested target in one run; MAESTRO’s six-run sweep missed the target and used more total GPU-hours. This shows BOLT’s practical advantage in budget control.
Code for all new experiments, including the fallback step, has been released with the revision. In light of these results and clarifications, we hope the reviewer would reconsider the assessment.