- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tackles the problem of neural network storage compression by jointly optimizing parameter sparsity and quantization symbol distribution during training. The authors propose a novel method called PRESTO, which introduces a joint formulation by coupling sparsity and symbol counts within a proxy of code length. Experimentally, PRESTO showcases gains against compared baselines across five out of six different networks from different families : ResNet, ViT, DeiT, EfficientNet on ImageNet (falling behind only on AlexNet).
Critical strengths
The paper introduces a joint formulation by coupling sparsity and symbol counts within a proxy of code length.
The proposed method showcases good empirical compression: against the LSQ baseline, trained under the same recipe and budget, with sizes measured on the whole checkpoint after serialization, the packet is roughly halved on ResNet-18 and ResNet-50, and is reduced by about 58% for a 0.44 point accuracy cost on ViT-B/16.
I believe the paper provides a detailed and transparent empirical reporting: there is an extensive supplementary material providing dose-transfer guidelines, wall-clock training times, and even an honest reporting on negative results and algorithmic edge cases.
Code (along with instructions, environment details and options to run it) is provided.
Critical weaknesses
The added value of the perspective structure over standard regularization is unclear: when dual variables are inactive, the perspective formulation collapses analytically to standard regularization in the regime where the interior minimizer of Eq. (24) is active, yet no baseline is provided to isolate its contribution. (See question 1.)
PRESTO induces a heavy training overhead: the dual solver introduces an epoch training slowdown of to compared to the LSQ baseline, limiting scalability. Moreover, the proposed method requires empirically calibrating three coupled hyperparameters which can not be derived analytically.
I believe the paper provides an untested hypothesis framed as an established finding: the performance deficit on AlexNet is attributed to layer-wise sparsity allocation across the Abstract, Introduction, and Conclusion without empirical verification under the paper's final recipe.
Despite the breadth of the experimental part, the deployment cost of the stored artifact is never reported. Sizes are measured after zstd level 22 on the support mask and the value stream, but nothing is said about decoding. There is no decompression time, no cost of unpacking the mask and indices, and no cost of dequantization or model loading. There is also no peak memory at load time and no accuracy measured on the model reconstructed from the packet.
Questions for authors
- On the real added value of the perspective formulation
When the entropy term is disabled () or when the dual count multipliers remain inactive (), the scalar per-weight subproblem of Eq. (23) reduces to:
Its unconstrained interior minimizer from Eq. (24) is:
Substituting this minimizer back into yields:
Summing across all weights within tensor , and wherever this interior minimizer is feasible, the regularizer collapses analytically to an penalty on the clipped weights:
Since the perspective formulation without the dual solver acts as standard regularization in this regime, could the authors isolate whether the compression gains arise from the perspective envelope formulation or simply from an -induced sparsity effect ? Given that the dual solver increases training time per epoch by a factor of to , could the authors provide a direct comparison against a baseline combining LSQ with standard regularization (as well as iterative magnitude pruning) compressed with zstd?
-
On the AlexNet deficit and the layer-wise allocation hypothesis In the Abstract (lines 29-30) and the Conclusion (lines 481-483), the authors state that the performance deficit on AlexNet is attributed to sparsity allocation. In Section 4.4 and Appendix C.6, the authors hypothesize that PRESTO's deficit against prune-first methods on AlexNet stems from using a single global sparsity coefficient across all layers, arguing that sparing convolutional layers would bridge the gap. However, the authors explicitly concede that this is an untested hypothesis and that no controlled experiment with the final training recipe was conducted. The only run with layer-wise T₂ allocation (Table 11) is worse on both axes than the T₂-only arm of Table 8: 54.94% at 3.67% of the checkpoint, against 55.77% at 3.03%. I note that this run differs from the reference in several respects (per-channel instead of per-tensor step sizes, a 40-epoch dose ramp instead of a fixed dose, a higher learning-rate floor, and a final sparsity 7.6 points lower), so it is not a controlled comparison and I do not take it as evidence against the hypothesis. But it does mean that the hypothesis currently has no direct empirical support in the paper. Moreover, to my understanding, introducing layer-wise tuning would undermine the core value proposition of PRESTO, which relies on a unified, joint optimization with a minimal set of scalar hyperparameters. The experiment would therefore also establish what the proposed fix costs. Could the authors provide the controlled per-layer allocation experiment under the final recipe to substantiate this claim ? If not, I would recommend to tone down the Abstract, Introduction, and Conclusion to explicitly state that the underlying cause of this deficit remains an open question and an untested hypothesis.
-
On the deployment cost. Since the contribution is a better trade-off between accuracy and deployable file size, I believe the cost of turning that file back into a usable network should be part of the evaluation, especially against NNCodec, designed for this purpose. Could the authors report the decoding cost of the PRESTO packet, and verify the round trip? Wall-clock time and peak memory to go from the serialized packet to a ready-to-run model (zstd decompression, mask and index unpacking, dequantization), on CPU/GPU, for ResNet-18 and ViT-B/16, and the same measure for the NNCodec bitstreams used as baselines. Even a single-machine measurement would be informative. Furthermore, could the authors provide the accuracy of the model reconstructed from the serialized packet? If I understood well, accuracies in the paper appear to be measured on the fake-quantized network under bfloat16 autocast, and the paper does not describe loading the packet back into a model. Could the authors please confirm that the decoded model reproduces the reported top-1 and size, so that the end-to-end claim is checked and not only the two ends?
Given the current state of the paper, I am leaning toward a weak rejection. However, if the points raised are sufficiently addressed during the rebuttal, I am willing to increase my score.
Update during the rebuttal: I thank the authors for their detailed response and additional experiments, which addressed most of my concerns. I have raised my score accordingly (see my reply to the authors for the remaining reservations).
3 comments
Response to Rebuttal
I would like to thank the authors for the substantial effort made during the rebutal to adress my concerns, particularly for providing the requested L1 and magnitude pruning baselines, running the controlled layer-wise allocation experiment on AlexNet, and benchmarking end-to-end decoding against NNCodec. I believe these additions significantly strengthen the paper, and I have raised my score accordingly.
My remaining reservation is that the gain over magnitude pruning (0.2 to 0.4 points) rests on single runs on ResNet-18 at 20 epochs, which is close to the 0.2-point run-to-run variation the authors mention, and the gain over plain L1 is also from single runs. For the camera-ready, I would ask the authors to report mean ± std over multiple seeds for these comparisons (L1, pruning, PRESTO), along with the remaining ± entries of Table 1 as stated in the general response.
Thanks and camera-ready commitment
Thank you very much for the update. We will report mean ± std over several seeds for these comparisons (L1, pruning, PRESTO), together with the remaining ± entries of Table 1, in the camera-ready version.
Response to JsCX
Dear Reviewer, thank you for the detailed review and for the precise questions. The revised PDF (changes in blue) and the updated supplementary material are uploaded.
Q3 (deployment cost). We implemented it. The packet is written to a file and read back; on ImageNet validation under bfloat16, the rebuilt network has the same top-1 as the fake-quantized one and exactly zero logit difference on the first batch, on five checkpoints of four architectures (ResNet-18 70.100% and 69.906%, DeiT-Small 79.430%, EfficientNet-B0 76.756%, ViT-B/16 80.306%). Decoding to a ready network takes 0.20 s (ResNet-18), 0.40 s (DeiT-Small), 0.12 s (EfficientNet-B0) and 1.14 s (ViT-B/16) on an 8-core CPU, and 0.13 to 1.12 s on an A100 with 20 to 330 MiB of GPU memory; decompression is a small part (ViT-B/16: zstd 0.03 s, unpacking 0.47 s). NNCodec's streams are 4.1% smaller on the same symbols but decode about five times slower. The file is slightly larger than our parameter-based size (batch-norm statistics and a header): by 0.01 to 0.10 points, and by 0.91 on EfficientNet-B0, whose factor becomes 9.1x instead of 9.9x (Section 4.1, Appendix C.13). The output of the round trip for each checkpoint (both top-1 values, the logit difference, the size of each component and the decoding times) is in
logs/rebuttal/verify/of the supplementary material.Q2 (AlexNet). You are right that the hypothesis had no direct support. We ran the controlled test (the recipe of the sparsity-only arm, with its single T2 replaced by the per-layer allocation of Table 11): at equal top-1 (55.76% against 55.77%) it needs 47% more packet (4.46% against 3.03%). It tests one allocation at two doses, but gives the hypothesis no support, so the Abstract, Introduction and Conclusions now say that we cannot attribute the deficit (Appendices C.6 and C.11).
W2 (overhead; hyperparameters). The overhead comes from the dual solver of the entropy term; T1 and T2 add no time. Calling it every 16 steps instead of 4 brings ResNet-18 from 3.8x to 1.5x LSQ with the same result (ResNet-18, one run; not verified on other networks). T1 can be set to zero without loss at 20 epochs on ResNet-18, leaving two doses; the doses are still calibrated empirically (Appendices B.4 and B.5).
Q1 (what the perspective adds over L1). Your reduction is exact where it applies, and we tested it directly. At the dose of the paper (T1 = 1e-5, T2 = 1e-7) it gives an L1 of slope λ = 2√(T1·T2) = 2e-6. A plain L1 on the latent weights with exactly that λ leaves the packet at the LSQ size (9.37% of the checkpoint, against 9.42%), while the same T1, T2 inside PRESTO give 7.47%. The difference comes from where your interior minimizer is infeasible: it needs V ≥ √(T2/T1) = 0.1, with V the largest level of the grid, and in the LSQ baseline the tensors that hold about 90% of the weights stand below that (the four largest, 71% of the weights, at 0.044 to 0.060; the median over tensors is 0.097). There a weight sits on its representability floor, its penalty is (T2/V + T1·V)·|w̄|, never below your λ, and it pushes the step size up until V reaches √(T2/T1): with the ridge those four tensors grow to 0.07 to 0.08, without it to 0.10 to 0.14 (
lsq_diagin the logs; the medians of Table 13 weigh all 21 tensors equally). With T1 = 0 that cap disappears and the term is exactly T2·|w̄|/V, an L1 on the weights normalized by their grid. On ResNet-18, 20 epochs, one recipe and one packet (LSQ: 9.4%), top-1 at size:So what the formulation adds over L1 is the coupling of the sparsity price to the learned grid: 0.8 points of top-1 over a plain L1 at the same size, and 0.2 to 0.4 over pruning at the two sizes we ran, which single runs with a 0.2 spread do not separate. The perspective ridge does not add to it: at the dose of the paper it caps the growth of the step sizes, which is how the sparsity term creates zeros, and removing it made the packet smaller in both pairs we ran. The entropy term lowers the size by a further 7% without the ridge, but at this budget we cannot separate that from a move along the normalized-L1 frontier (69.57% at 5.43%, within the spread of identical runs). Our reading, not a tested mechanism: growing steps widen the zero interval and lower the entropy of the surviving symbols (1.85 bits, against 2.9 to 3.1 for plain L1 and pruning), so the concentration the entropy term aims at is largely already there. Single runs on one network at 20 epochs: we make no claim at the budgets of Table 1 or on the other networks (Section 4.3, Appendices B.4 and C.10).