- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents BOLT, a blockwise ridge solver for classification on cached, frozen representations. It fits fixed feature blocks sequentially to residual targets, optionally performs damped cyclic refinement, and solves block subproblems using augmented QR. The experiments assess predictive accuracy, fitting time, and peak GPU memory relative to joint ridge and other classifier baselines.
Critical strengths
-
Substantial memory savings under a matched comparison. Table 1 reports approximately 64–72% reductions in peak allocated GPU memory relative to a joint solver using the same feature bank and regularization. Preserving feature capacity while reducing temporary working memory provides meaningful practical value for repeatedly fitting classification heads.
-
A clear connection between approximation and refinement. The paper explicitly distinguishes the single-sweep approximation from the limiting joint ridge solution and provides a convergence argument under stated assumptions. Table 4 shows that refinement recovers most of the measured test-accuracy gap. This makes the method’s approximation behavior and computational trade-offs interpretable.
Critical weaknesses
-
Limited additional methodological contribution. As the manuscript acknowledges, the forward cascade is the first Gauss–Seidel/BCD cycle from zero initialization, and refinement continues the block updates with damping. lgorithm 3 in "Large Scale Kernel Learning using Block Coordinate Descent"(https://arxiv.org/pdf/1602.05310) already describe the corresponding random-feature ridge block minimization, up to regularization normalization and choices of block visitation. The convergence proof also specializes standard BCD results. The potentially distinctive contribution therefore lies primarily in the implementation and empirical trade-offs for frozen-feature refitting. I remain unconvinced that the current evidence establishes sufficient additional practical insight or benefit over established blockwise ridge solving.
-
Incomplete evidence for the practical trade-off relative to the existing BCD baseline. Table 2 reports higher mean accuracy for BCD-Classic than either BOLT variant, but provides no corresponding fitting-time or peak-memory measurements. Its sweep budget, visitation order, damping, stopping rule, and numerical solver are also insufficiently specified. Table 4 shows that increasing refinement from one to twenty sweeps improves accuracy from 74.69% to 76.72% while increasing fitting time from approximately 0.95 s to 15.95 s. This characterizes BOLT’s internal trade-off, but leaves its comparative value unclear. Resource measurements for the existing BCD baseline under the same cached-input protocol are important for assessing the practical contribution.
Questions for authors
-
How does BCD-Classic differ from BOLT-Sweep in sweep budget, visitation order, initialization, damping, and numerical solver? Which differences explain the accuracy results in Table 2?
-
Can you report fitting time and peak memory for BCD-Classic at comparable optimization error or predictive accuracy, specifying whether feature matrices and block factorizations are cached or recomputed? Appendix C.4 demonstrates a synthetic stability benefit for augmented QR; how does this choice affect the practical time–accuracy–memory trade-off?
Thank you for recognizing BOLT's memory savings and refinement mechanism. We tested solver settings, caching, and numerical accuracy.
Contribution and related work. BOLT combines a full-bank residual pass, damped refinement, and augmented QR for refitting classifiers on cached frozen representations. It shares the cited algorithm's BCD foundation. QR, damping, and block order affect numerical behavior and finite-sweep iterates. We will add a direct technical comparison with Algorithm 3 of the cited work.
Q1. How does BCD-Classif differ from BOLT-Sweep in sweep budget, visitation order, initialization, damping, and numerical solver? Which differences explain the accuracy results in Table 2?
For consistency with our tables, we refer to this method as BCD-Classic. The core implementation starts from zero, visits blocks in forward order, and uses alpha=1 and augmented QR. It runs for at most five cycles, with a relative coefficient-change threshold of 1e-3. It caches block features but recomputes QR factors. BOLT-Sweep uses three alternating sweeps with alpha=0.8, also starting from zero with QR block solves. We apply the damped update to the stored prediction coefficients:
Wknew=(1−α)Wkold+αWkexact.
Our new controls fix the feature bank, targets, regularization, initialization, and block solver. On CIFAR-100/ViT (2,048 RBF features, block width 256, ridge=0.1), three paired seeds gave:
The results show how damping, order, and sweep budget affect accuracy under the same ridge objective.
We also checked Figure 3(b)'s ReLU setting: 6 blocks of 192, ridge=0.01, alpha=0.8, and alternating sweeps. Across ten paired seeds on CIFAR-100/ViT, refinement reduced the mean accuracy gap to joint QR from 1.977 percentage points after one sweep to 0.011 after twenty sweeps. The mean absolute gap was 0.065 points; 8 of 10 seeds were within 0.1 points. The main and audit implementations matched in test accuracy across 40 checks. With the same feature bank partitioned into four, six, or eight blocks, the mean gaps after twenty sweeps were 0.012, 0.011, and 0.043 points, respectively.
Q2. Can you report fitting time and peak memory for BCD-Classic at comparable optimization error or predictive accuracy, specifying whether feature matrices and block factors are cached or recomputed? Appendix C.4 demonstrates a synthetic stability benefit for augmented QR; how does this choice affect the practical time–accuracy–memory trade-off?
First, we compared three caching strategies using the same block updates. For 20 forward BCD sweeps with alpha=1 in the RBF setting above, all three gave identical objective values and 78.82% accuracy:
All three had a normalized objective gap of 0.04495 to joint QR. Because objective values and accuracy matched, this comparison isolates the cost of caching. The cache policies change computational cost without changing the measured solution quality. Reusing QR factors reduced fitting time by about 3.5x. Timings include preparation, feature construction, factorization, and updates, but exclude diagnostic evaluation. The five-dataset comparisons use three paired seeds and three timing runs per seed.
Second, for each pre-specified objective-gap threshold, we recorded the first checkpoint that met it, using FP64 augmented QR as the reference. Both methods could run up to 1,000 sweeps or 120 s. On CIFAR-100/ViT with ReLU features, cached BOLT and BCD reached a normalized gap below 1e-3 in 0.420 s and 0.422 s. Both had the same measured peak GPU allocation of 507.9 MiB under the matched implementation and caching policy. This shows comparable refinement costs at the same tolerance. Streamed QR required 0.146 s and 439.5 MiB. All three seeds met the threshold. Reported times are averaged over three runs for each of three seeds.
Third, checks on real feature matrices found both low-precision factorization failures and loss of positive definiteness during accumulation. In one R18 case at width 8,192, the minimum eigenvalue was about -0.0089 after FP32 accumulation versus 0.1 after FP64 accumulation. This shows a practical failure mode of forming normal equations in low precision. We include streamed QR as a numerically stable joint baseline.
Within each comparison, software, hardware, cached inputs, and feature settings were matched. Memory figures are peak PyTorch-allocated GPU memory.