acceptodds
Under review as a conference paper at ICLR 2027

PRESTO: PeRspective-based Entropy-Sparsity Training Optimization

13 views

Abstract

Pruning and quantization make a neural network smaller, and when the goal is storage size, a lossless compressor is eventually applied (Han et al., 2016) to a model that is usually not trained to be compressible. We introduce PRESTO, a quantization-aware training method that adds to the loss a proxy for the compressed size of the weights, leaving the activations in floating point. This proxy is the value of a convex inner problem over the quantization grid learned by LSQ (Esser et al., 2020), written in perspective form, i.e. the tightest convex relaxation of the decision to keep a weight. A Lagrangian relaxation makes that problem separable, and an envelope argument gives a (sub)gradient of its value. To our knowledge, PRESTO is the first method to jointly balance sparsity and the codebook symbol distribution arising from quantization within a single convex inner problem, rather than through separate terms that act independently, as in HEMP+LOBSTER (Tartaglione et al., 2021) and LilNetX (Girish et al., 2023). On ImageNet, with sizes measured on the whole checkpoint after lossless coding, PRESTO reaches on ResNet-18, on ResNet-50, on ViT-B/16, on DeiT-Small and on EfficientNet-B0, above full-precision accuracy on the two ResNets and less than accuracy points below it on the other three; on five of the six networks, we hold an operating point that no published method we compare against betters on both axes. On AlexNet, prune-first methods remain ahead, a deficit we attribute to sparsity allocation.

Primary area:
optimization
Keywords:
Neural Network Compression, Lagrangian Relaxation, Quantization-Aware Training, Convex Optimization, Pruning, Entropy Coding, Perspective Reformulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_JsCXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles the problem of neural network storage compression by jointly optimizing parameter sparsity and quantization symbol distribution during training. The authors propose a novel method called PRESTO, which introduces a joint formulation by coupling sparsity and symbol counts within a proxy of code length. Experimentally, PRESTO showcases gains against compared baselines across five out of six different networks from different families : ResNet, ViT, DeiT, EfficientNet on ImageNet (falling behind only on AlexNet).

Critical strengths

The paper introduces a joint formulation by coupling sparsity and symbol counts within a proxy of code length.

The proposed method showcases good empirical compression: against the LSQ baseline, trained under the same recipe and budget, with sizes measured on the whole checkpoint after serialization, the packet is roughly halved on ResNet-18 and ResNet-50, and is reduced by about 58% for a 0.44 point accuracy cost on ViT-B/16.

I believe the paper provides a detailed and transparent empirical reporting: there is an extensive supplementary material providing dose-transfer guidelines, wall-clock training times, and even an honest reporting on negative results and algorithmic edge cases.

Code (along with instructions, environment details and options to run it) is provided.

Critical weaknesses

The added value of the perspective structure over standard L1L_1 regularization is unclear: when dual variables are inactive, the perspective formulation collapses analytically to standard L1L_1 regularization in the regime where the interior minimizer of Eq. (24) is active, yet no L1L_1 baseline is provided to isolate its contribution. (See question 1.)

PRESTO induces a heavy training overhead: the dual solver introduces an epoch training slowdown of 3.3×3.3\times to 12.6×12.6\times compared to the LSQ baseline, limiting scalability. Moreover, the proposed method requires empirically calibrating three coupled hyperparameters which can not be derived analytically.

I believe the paper provides an untested hypothesis framed as an established finding: the performance deficit on AlexNet is attributed to layer-wise sparsity allocation across the Abstract, Introduction, and Conclusion without empirical verification under the paper's final recipe.

Despite the breadth of the experimental part, the deployment cost of the stored artifact is never reported. Sizes are measured after zstd level 22 on the support mask and the value stream, but nothing is said about decoding. There is no decompression time, no cost of unpacking the mask and indices, and no cost of dequantization or model loading. There is also no peak memory at load time and no accuracy measured on the model reconstructed from the packet.

Questions for authors
  1. On the real added value of the perspective formulation
    When the entropy term is disabled (T3=0T_3 = 0) or when the dual count multipliers remain inactive (ξ=0  ⟹  μi=0,si=0\xi = 0 \implies \mu_i = 0, s_i = 0), the scalar per-weight subproblem of Eq. (23) reduces to:

Gi(yi)=T1(wˉi)2yi+T2yiG_i(y_i) = T_1 \frac{(\bar{w}_i)^2}{y_i} + T_2 y_i

Its unconstrained interior minimizer from Eq. (24) is:

yi∗=∣wˉi∣T1T2y_i^\ast = |\bar{w}_i| \sqrt{\frac{T_1}{T_2}}

Substituting this minimizer back into GiG_i yields:

Gi(yi∗)=T1(wˉi)2∣wˉi∣T1/T2+T2∣wˉi∣T1T2=2T1T2 ∣wˉi∣G_i(y_i^\ast) = T_1 \frac{(\bar{w}_i)^2}{|\bar{w}_i| \sqrt{T_1 / T_2}} + T_2 |\bar{w}_i| \sqrt{\frac{T_1}{T_2}} = 2\sqrt{T_1 T_2} \ |\bar{w}_i|

Summing across all weights within tensor ll, and wherever this interior minimizer is feasible, the regularizer collapses analytically to an L1L_1 penalty on the clipped weights:

∑i∈IlGi(yi∗)=λ∥wˉ(l)∥1withλ=2T1T2\sum_{i \in \mathcal{I}_l} G_i(y_i^\ast) = \lambda \|\bar{w}^{(l)}\|_1 \quad \text{with} \quad \lambda = 2\sqrt{T_1 T_2} Since the perspective formulation without the dual solver acts as standard L1L_1 regularization in this regime, could the authors isolate whether the compression gains arise from the perspective envelope formulation or simply from an L1L_1-induced sparsity effect ? Given that the dual solver increases training time per epoch by a factor of 3.3×3.3\times to 12.6×12.6\times, could the authors provide a direct comparison against a baseline combining LSQ with standard L1L_1 regularization (as well as iterative magnitude pruning) compressed with zstd?

  1. On the AlexNet deficit and the layer-wise allocation hypothesis In the Abstract (lines 29-30) and the Conclusion (lines 481-483), the authors state that the performance deficit on AlexNet is attributed to sparsity allocation. In Section 4.4 and Appendix C.6, the authors hypothesize that PRESTO's deficit against prune-first methods on AlexNet stems from using a single global sparsity coefficient T2T_2 across all layers, arguing that sparing convolutional layers would bridge the gap. However, the authors explicitly concede that this is an untested hypothesis and that no controlled experiment with the final training recipe was conducted. The only run with layer-wise T₂ allocation (Table 11) is worse on both axes than the T₂-only arm of Table 8: 54.94% at 3.67% of the checkpoint, against 55.77% at 3.03%. I note that this run differs from the reference in several respects (per-channel instead of per-tensor step sizes, a 40-epoch dose ramp instead of a fixed dose, a higher learning-rate floor, and a final sparsity 7.6 points lower), so it is not a controlled comparison and I do not take it as evidence against the hypothesis. But it does mean that the hypothesis currently has no direct empirical support in the paper. Moreover, to my understanding, introducing layer-wise tuning would undermine the core value proposition of PRESTO, which relies on a unified, joint optimization with a minimal set of scalar hyperparameters. The experiment would therefore also establish what the proposed fix costs. Could the authors provide the controlled per-layer allocation experiment under the final recipe to substantiate this claim ? If not, I would recommend to tone down the Abstract, Introduction, and Conclusion to explicitly state that the underlying cause of this deficit remains an open question and an untested hypothesis.

  2. On the deployment cost. Since the contribution is a better trade-off between accuracy and deployable file size, I believe the cost of turning that file back into a usable network should be part of the evaluation, especially against NNCodec, designed for this purpose. Could the authors report the decoding cost of the PRESTO packet, and verify the round trip? Wall-clock time and peak memory to go from the serialized packet to a ready-to-run model (zstd decompression, mask and index unpacking, dequantization), on CPU/GPU, for ResNet-18 and ViT-B/16, and the same measure for the NNCodec bitstreams used as baselines. Even a single-machine measurement would be informative. Furthermore, could the authors provide the accuracy of the model reconstructed from the serialized packet? If I understood well, accuracies in the paper appear to be measured on the fake-quantized network under bfloat16 autocast, and the paper does not describe loading the packet back into a model. Could the authors please confirm that the decoded model reproduces the reported top-1 and size, so that the end-to-end claim is checked and not only the two ends?

Given the current state of the paper, I am leaning toward a weak rejection. However, if the points raised are sufficiently addressed during the rebuttal, I am willing to increase my score.

Update during the rebuttal: I thank the authors for their detailed response and additional experiments, which addressed most of my concerns. I have raised my score accordingly (see my reply to the authors for the remaining reservations).

3 comments
AuthorsOpenReview ↗

Response to JsCX

Dear Reviewer, thank you for the detailed review and for the precise questions. The revised PDF (changes in blue) and the updated supplementary material are uploaded.

Q3 (deployment cost). We implemented it. The packet is written to a file and read back; on ImageNet validation under bfloat16, the rebuilt network has the same top-1 as the fake-quantized one and exactly zero logit difference on the first batch, on five checkpoints of four architectures (ResNet-18 70.100% and 69.906%, DeiT-Small 79.430%, EfficientNet-B0 76.756%, ViT-B/16 80.306%). Decoding to a ready network takes 0.20 s (ResNet-18), 0.40 s (DeiT-Small), 0.12 s (EfficientNet-B0) and 1.14 s (ViT-B/16) on an 8-core CPU, and 0.13 to 1.12 s on an A100 with 20 to 330 MiB of GPU memory; decompression is a small part (ViT-B/16: zstd 0.03 s, unpacking 0.47 s). NNCodec's streams are 4.1% smaller on the same symbols but decode about five times slower. The file is slightly larger than our parameter-based size (batch-norm statistics and a header): by 0.01 to 0.10 points, and by 0.91 on EfficientNet-B0, whose factor becomes 9.1x instead of 9.9x (Section 4.1, Appendix C.13). The output of the round trip for each checkpoint (both top-1 values, the logit difference, the size of each component and the decoding times) is in logs/rebuttal/verify/ of the supplementary material.

Q2 (AlexNet). You are right that the hypothesis had no direct support. We ran the controlled test (the recipe of the sparsity-only arm, with its single T2 replaced by the per-layer allocation of Table 11): at equal top-1 (55.76% against 55.77%) it needs 47% more packet (4.46% against 3.03%). It tests one allocation at two doses, but gives the hypothesis no support, so the Abstract, Introduction and Conclusions now say that we cannot attribute the deficit (Appendices C.6 and C.11).

W2 (overhead; hyperparameters). The overhead comes from the dual solver of the entropy term; T1 and T2 add no time. Calling it every 16 steps instead of 4 brings ResNet-18 from 3.8x to 1.5x LSQ with the same result (ResNet-18, one run; not verified on other networks). T1 can be set to zero without loss at 20 epochs on ResNet-18, leaving two doses; the doses are still calibrated empirically (Appendices B.4 and B.5).

Q1 (what the perspective adds over L1). Your reduction is exact where it applies, and we tested it directly. At the dose of the paper (T1 = 1e-5, T2 = 1e-7) it gives an L1 of slope λ = 2√(T1·T2) = 2e-6. A plain L1 on the latent weights with exactly that λ leaves the packet at the LSQ size (9.37% of the checkpoint, against 9.42%), while the same T1, T2 inside PRESTO give 7.47%. The difference comes from where your interior minimizer is infeasible: it needs V ≥ √(T2/T1) = 0.1, with V the largest level of the grid, and in the LSQ baseline the tensors that hold about 90% of the weights stand below that (the four largest, 71% of the weights, at 0.044 to 0.060; the median over tensors is 0.097). There a weight sits on its representability floor, its penalty is (T2/V + T1·V)·|w̄|, never below your λ, and it pushes the step size up until V reaches √(T2/T1): with the ridge those four tensors grow to 0.07 to 0.08, without it to 0.10 to 0.14 (lsq_diag in the logs; the medians of Table 13 weigh all 21 tensors equally). With T1 = 0 that cap disappears and the term is exactly T2·|w̄|/V, an L1 on the weights normalized by their grid. On ResNet-18, 20 epochs, one recipe and one packet (LSQ: 9.4%), top-1 at size:

  • plain L1, λ = 2e-6 (your reduction): 69.92% at 9.37%;
  • plain L1, λ = 2e-5: 68.88% at 5.64%;
  • magnitude pruning plus zstd: 69.72% at 7.52%; 69.58% at 5.85%;
  • PRESTO, T1 = 1e-5, T3 = 0: 70.10% at 7.47%;
  • PRESTO, T1 = T3 = 0 (normalized L1): 69.80% at 5.86%;
  • full PRESTO: 69.67 to 69.91% at 5.83%; without the ridge, 69.67% at 5.43%.

So what the formulation adds over L1 is the coupling of the sparsity price to the learned grid: 0.8 points of top-1 over a plain L1 at the same size, and 0.2 to 0.4 over pruning at the two sizes we ran, which single runs with a 0.2 spread do not separate. The perspective ridge does not add to it: at the dose of the paper it caps the growth of the step sizes, which is how the sparsity term creates zeros, and removing it made the packet smaller in both pairs we ran. The entropy term lowers the size by a further 7% without the ridge, but at this budget we cannot separate that from a move along the normalized-L1 frontier (69.57% at 5.43%, within the spread of identical runs). Our reading, not a tested mechanism: growing steps widen the zero interval and lower the entropy of the surviving symbols (1.85 bits, against 2.9 to 3.1 for plain L1 and pruning), so the concentration the entropy term aims at is largely already there. Single runs on one network at 20 epochs: we make no claim at the budgets of Table 1 or on the other networks (Section 4.3, Appendices B.4 and C.10).

Reviewer_JsCXOpenReview ↗

Response to Rebuttal

I would like to thank the authors for the substantial effort made during the rebutal to adress my concerns, particularly for providing the requested L1 and magnitude pruning baselines, running the controlled layer-wise allocation experiment on AlexNet, and benchmarking end-to-end decoding against NNCodec. I believe these additions significantly strengthen the paper, and I have raised my score accordingly.

My remaining reservation is that the gain over magnitude pruning (0.2 to 0.4 points) rests on single runs on ResNet-18 at 20 epochs, which is close to the 0.2-point run-to-run variation the authors mention, and the gain over plain L1 is also from single runs. For the camera-ready, I would ask the authors to report mean ± std over multiple seeds for these comparisons (L1, pruning, PRESTO), along with the remaining ± entries of Table 1 as stated in the general response.

AuthorsOpenReview ↗

Thanks and camera-ready commitment

Thank you very much for the update. We will report mean ± std over several seeds for these comparisons (L1, pruning, PRESTO), together with the remaining ± entries of Table 1, in the camera-ready version.

Reviewer_u56tOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a quantization-aware training method that adds to the loss a proxy for the losslessly coded size of the quantized weights, much in spirit of the literature on entropy constrained quantization (mostly ignored).

The proxy is the value of a convex inner problem that assigns weights to codebook symbols. It combines a continuous relaxation of whether each weight is set to zero, a sparsity term and an entropy-like term on relaxed symbol counts. A Lagrangian relaxation of the count constraints makes the problem separable. The authors report better accuracy-size trade-offs than their LSQ baseline and than most published methods on several ImageNet networks to which they compare, with prune-first methods remaining ahead on AlexNet, at a substantially higher training cost.

Critical strengths

A strength is that the paper optimizes the right objective if one allows for entropy coding: The paper trains directly for the entropy-coded size of the model, which is what matters for storage. Entropy-constrained quantization of network weights has been explored before, e.g. Wiedemann et al., 2019. This paper integrates it end to end into QAT, together with sparsity and learned step sizes. It also shows that standard QAT tends to make models less compressible as training goes on.

Principled joint formulation. Sparsity and the symbol distribution are traded off within a single problem. The decomposition gives exact per-weight solutions and gradients for both the weights and the step sizes, and the authors show that optimizing the terms jointly beats running them sequentially.

The sizes are measured on the full serialized model, metadata included, which makes the comparisons more trustworthy than is common in this field.

Transparency. The paper is candid about where theory and implementation diverge, reports negative results and costs, and includes detailed reproducibility information.

Critical weaknesses

Positioning against prior work. The paper does not place itself relative to the classical literature on entropy-constrained scalar and vector quantization, in particular the seminal paper by Chou et al. 89 "Entropy-Constrained Vector Quantization" or prior work by Farvardin, or to its applications to network weights. Therefore the claim that jointly controlling sparsity and the symbol distribution is an open question is overstated. Several of the paper's empirical findings are also known properties of entropy-constrained quantizers: zeros coming from a widened dead zone, and a uniform grid followed by entropy coding working well. Vector and product quantization methods, which reach similar sizes at fixed rate, are not discussed or compared either.

The surrogate is not the code length. The Σ c log c term was chosen for convexity. At fixed non-zero mass, though, it favors a uniform symbol distribution, so it acts mainly as a second sparsity pressure, as Appendix A.6 acknowledges. The paper should justify this choice over the true code length (-log p), which classical methods handle by alternating minimization.

Limited baselines. The main trained baseline is a LSQ variant. Simple alternatives trained with the same recipe are missing, for instance LSQ plus magnitude pruning plus entropy coding, a differentiable entropy penalty, or stronger QAT methods for extreme compression. Comparisons with published results mix different checkpoints, nominal and measured sizes, and different artifacts. It also omits established methods for extreme compression, such as product quantization of weights and variants of QAT such as training with quantization noise (e.g. Fan et al., 2021). Both report comparable compression rates on ImageNet. Comparisons with published results also mix different checkpoints, nominal and measured sizes, and different artifact types.

Clarity. The paper is very dense and heavily hedged, and the main text leans on long appendices, which makes the core contribution hard to extract.

Questions for authors

please address the pointed weaknesses above

3 comments
AuthorsOpenReview ↗

Response to u56t

Dear Reviewer, thank you for the detailed review. The revised PDF (changes in blue) and the updated supplementary material are uploaded.

W1 (positioning). Agreed. Entropy-constrained scalar and vector quantization (Farvardin and Modestino, 1984; Chou et al., 1989) is the classical framework, and zeros from a widened dead zone and a uniform grid followed by entropy coding are known properties of such quantizers; we do not present them as findings, and the revised Introduction cites these works. Our claim is narrower: sparsity and the symbol distribution priced inside one convex inner problem, trained with the task loss, with learned step sizes and exactly solved per-weight subproblems, so the network receives one subgradient instead of separate penalties (HEMP+LOBSTER, LilNetX) or stages (Deep Compression). We did not run vector or product quantization or quantization-noise training under our recipe: they store codebooks and indices, a different artifact, and the revised Introduction names them as not compared. On published numbers, Stock et al. (their Table 1; nominal sizes, index bits plus float16 centroids, no entropy coding) report for the standard torchvision ResNets 73.79% at 19x on ResNet-50 and 65.81% at 29x on ResNet-18, where our measured packets give 76.32% at 19.5x and 68.33% at 33x (Tables 1 and 6). Their 76.1% at 5 MB (20x), the point the Introduction cites, starts from a semi-supervised ResNet-50 of 79.3%. Recipes and budgets differ, so we do not read this as a ranking.

W2 (the surrogate is not the code length). Right, and Appendix A.6 says so. The term acts mainly through the non-zero mass, driving mass into the zero symbol, while at fixed mass it acts against code length; the mass channel is five to six times larger, so it is mainly a second sparsity pressure, and the entropy reduction we observe is a consequence of the sparsity it creates. A new measurement agrees: on ResNet-18, adding the term to a sparsity-only run lowers the packet from 5.86% to 5.43% (top-1 69.80% to 69.67%), raises zeros from 48.0% to 52.4% and the grid level from 1.32x to 1.40x LSQ's. We use it because it is convex: the true value-stream length is concave in the counts, which would break the Lagrangian decomposition and strong duality, and alternating minimization with -log p held fixed between updates, the classical way around that, comes without that guarantee; we have not tested it in QAT. No reported size is the proxy: all are measured zstd packets.

W3 (limited baselines). Under our recipe on ResNet-18 (20 epochs; sizes as percentages of the checkpoint): LSQ plus global magnitude pruning plus zstd reaches 5.85% at 69.58%, against PRESTO's 5.83% at 69.67 to 69.91%, and 7.52% at 69.72%, against PRESTO's 7.47% at 70.10%: below at both sizes, by about 0.2 and 0.4 points of top-1, which with single pruning runs and a spread of 0.2 between identical commands we do not call a separation (without the ridge, PRESTO reaches 5.43% at 69.67%, 7% smaller than pruning at a comparable top-1; one run); PRESTO's sparsity term alone (T1 = T3 = 0, not an outside baseline) gives 5.86% at 69.80% and a plain L1 5.64% at 68.88%; NNCodec's lossless coder on the LSQ weights gives 9.22% of the quantized tensors against 9.33% with zstd, so a better coder does not close the gap; the entropy term alone gives 6.09% at 69.78%. We did not implement a soft-histogram entropy penalty (EPR-DFT is a published point in Table 1), nor product quantization or quantization-noise training. We agree that the published comparisons of Table 1 are not under equal conditions (marked *, ‡, §); the like-for-like evidence is the rows we trained ourselves (Section 4.3, Appendix C.10).

W4 (clarity). Agreed. This revision does not restructure the paper; it changes what the main text claims: the Introduction now states a narrower question and what we do not compare, and the Abstract, Section 4.4 and the Conclusions no longer attribute the AlexNet deficit to allocation. To make it easier to read within the page limit, we also shortened the captions of the tables and figures to a one-line description and moved their definitions and keys into the text next to them. The new material is in four appendix sections (C.10 to C.13) and short additions to the main text, all in blue.

I am partly satisfied by the answers provided in the rebuttal.

I still think we are still missing baselines starting from the same checkpoint, to remove confounding factor in the comparison. This would be great to add some to help the paper potential impact.

so I maintain my weak accept rating.

Thank you for your reply. We understand the remaining reservation. Long queue times on our cluster limited how many additional runs we could complete during the discussion, and several more are still waiting in the queue. Thank you again for the careful review that improved the paper, and we appreciate your patience.

Reviewer_1TPfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduced another quantization-aware training method by introducing a new regulariziation factor on top of optimizing for task loss. It combined the perspective ridge, sparsity and entropy penalties.

  • it is built on the LSQ step optimization direction, and it encourages the model to get better on sparsity and quantization symbols distribution
  • it used the Lagrangian relaxation to decompose the inner problem to tracable subproblems and an evelope argument to derive weights and step size.

The work is highly connected with some existing works, like perspective-based sparse reguarlization like Cacciola, and entropy-aware model expression like HEMP+LOBSTER, LilNetX. It's most unique contribution comes from the introduction of the specific coupled formulation that is being formed as an single convex inner problem and also it's corresponding solver.

From the experiment result, we can see:

  • the model is able to achieve significant model/storage size reduction comparing with baseline LSQ, with almost 40% to 60% more saving on model size across different model architectures, while having top1 accuracy decreased small, usually between 0.2% to 0.5%.
  • the ablation study showed the best result mostly achieved by the combination of all three factors in the regularizer, with T0 itself doesn't seem to provide any direct gains and t3 providing good additional value but being the most costly factor with 3x to more than 10x compute needs. Alexnet is a special case here, with t1 + t2 able to achieve 96% of the reduction.
  • the comparison with other public works, showed that the proposed method is generally promising - improved on HEMP+LOBSTER reported publically for ResNet-18, almost matching EPR-DFT, and slightly worse compression size than LilNetX etc. It also noted that there are also differences on scope of quantization, storage representation that may cause noice in comparison.
Critical strengths
  • A well-developed optimization framework and solver for quantization-aware training: It cleanly formed a framework that combined the learning on sparsity and quantization symbol distribution. It used the Lagrangian relaxation to decompose the inner problem to tracable subproblems and an evelope argument to derive weights and step size. This provide a concrete technical contribution and showing optimal results comparing with baseline methods and some more recent public works.

  • Extensive experiments and information analysis: the study covered multiple model architectures, with separate ablation studies and compared with multiple recent public work, was able to both clearly stated the improvement on most of the settings, but also reveal and discuss the cost increase of T3 and out-of-ditribution behaviors like AlexNet. For AlexNet behavior, it examined the concentration of parameters and form hypothesis to explain the observed performance gap. This transparency and analysis helped us to have a full picture of the methodology.

Critical weaknesses
  • Limited compute-cost vs improvement comparison with public works and not enough analysis on the cost and benefit: I appreciate the author transparently reported the significantly cost increase when introduced the t3 in the reguarlization term - on ResNet18 it introduced 29.8% more storage saving but training cost 4x, and only 6.3% more storage saving but training cost 6.7x for AlexNet. I believe it will be valuebale to compare the cost with the public works like HEMP+LOBSTER, LilNetX. and SPR etc, especially seeing the absolute storage of the final optimization is not significant. In practice, where almost all entities are cost-sensitive, the increased compute need is going to face a bigger challenge, especially with the rise of LLM and the base compute being very high already.

  • Need additional experiments to justify the value of t1: From the result comparing with LSQ, T1 itself doesn't add any benefit in compression size, while t1 + t2 and t1 + t3 all see significant improvement. I believe a separate ablation with t2+ t3 and some explanation of the result can help us understand the value t1 better. I'd also suggest adding Cacciola's SPR work for comparison too given its close methodology comparing with PRESTO.

Questions for authors
  • How does PRESTO compare with existing public works on the cost front? We should do more compute-cost analysis to understand the value of T3.

  • Is t1 helpful? what result we will be getting with t2 + t3 only?

  • How does the model performs compare with Cacciola's SPR work?

3 comments
AuthorsOpenReview ↗

Response to 1TPf

Dear Reviewer, thank you for your review. The revised PDF (changes in blue) and the updated supplementary material are uploaded.

Q1 (cost of T3 against public works). The cost sits in the dual solver of the entropy term; T1 and T2 add no time (108 s per epoch on ResNet-18, as LSQ). It is not intrinsic: calling the solver every 16 steps instead of 4, with the coefficients multiplied by 4, takes 159 s per epoch (1.5x LSQ) instead of 412 s (3.8x) with the same result, 69.89% at 5.83% (ResNet-18, one run; not verified on other networks). On the literature: SPR's regularizer adds 1.3% to the epoch time on ImageNet ResNet-18 (150 plus 50 epochs, one TITAN V); LOBSTER uses 95 epochs from pretrained networks and LilNetX 35 from scratch; for EPR and HEMP we found no ImageNet epochs or times. None reports total GPU-hours, so we claim no ranking; for reference, our headline ResNet-18 run trains for 60 epochs from the public checkpoint in about 7.4 hours on 16 A100 (about 118 GPU-hours, evaluation excluded), against 1.8 hours (29) for its LSQ baseline (Appendix B.5). What T3 buys: against T1+T2 it buys 22% of size (7.47 to 5.83%), but that arm carries the ridge; without it, T2 alone reaches 5.86% at LSQ cost and T3 lowers it by about 7% (5.43% at 69.67%, against 5.86% at 69.80%; one run). On AlexNet T1+T2 already recover 97.5% of the reduction and we would not use T3 there.

Q2 (is T1 helpful?). Not at its default dose: removing it makes the packets smaller (joint 5.83 to 5.43% at the same top-1; T2 alone 7.47 to 5.86%), which leaves two coefficients to tune. The ridge caps the growth of the quantization step sizes at a grid extreme of √(T2/T1) = 0.1, and growing steps are how the sparsity term creates zeros: the four largest tensors (71% of the weights) go from 0.044 to 0.060 in the LSQ baseline to 0.07 to 0.08 with the ridge and to 0.10 to 0.14 without it (the median over all tensors is 0.98x LSQ's with the ridge and 1.32x without). Single runs at 20 epochs on ResNet-18; the T1 = 0 joint run also uses the adjacent-facet approximation of Appendix A.3.1 (Section 4.3, Appendices B.4 and C.10).

Q3 (SPR). SPR prunes whole filters, has no quantization or entropy coding and reports pruned parameters, not bytes, so we did not put it in the tables. On the axis SPR reports, its ImageNet ResNet-18 point with 63.4% of the parameters pruned reaches 66.20% top-1, while our 100-epoch run has 69.7% of zero weights at 69.86%. Neither number ranks the methods: SPR removes whole filters, which costs more accuracy than removing single weights but speeds up inference, and its output can itself be quantized and coded. A like-for-like comparison needs SPR under our recipe, which we cannot do now; we make no claim of superiority, and per epoch SPR's regularizer is far cheaper than our entropy term (Q1).

I would like to thanks the authors to address my main concerns and providing additional experiments. It is encouraging to see that reducing the frequency of dual-solver updates can substantially lower the training overhead while maintaining comparable performance. Although this has only been demonstrated on ResNet-18, it provides a promising direction for improving the method’s practicality. I would welcome results on additional architectures in the camera-ready version.

For the ablation removing t1, I think it helps to clarify its role and shows that the default perspective ridge is not necessary to achieve strong compression in the tested setting. I appreciate the transparent reporting of this finding and encourage the authors to reflect it clearly in the paper’s contribution claims.

The additional repeated runs and end-to-end decoding checks also strengthen the empirical evidence. Based on these responses and updates, I have raised my score.

Thank you for reading our responses and for raising your score. Your review pushed us to test the method on aspects we had also been concerned about, especially the cost of the dual solver, and the paper is better for it. We have noted your suggestions. Thank you very much again.

Reviewer_SnziOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a quantization-aware training method built on LSQ. It adds a loss term that estimates post-compression model size. On six ImageNet networks, PRESTO roughly halves compressed size versus its LSQ baseline on ResNets with similar accuracy.

Critical strengths
  1. The proposed method is well motivated and the paper is open to its limitations
  2. The experiments are well designed and controlled, and the settings are clearly presented.
Critical weaknesses
  1. The joint formulation looks costly for the gain. Joint training beats a two-stage pipeline, but the gain over simpler options is modest.
  2. Comparisons with sota baselines are not under equal conditions, and most results are single runs.
Questions for authors
  1. How would PRESTO compare with NNCodec after the same amount of fine-tuning?
  2. Can you report repeated runs on at least one or two more networks?
1 comment
AuthorsOpenReview ↗

Response to Snzi

Dear Reviewer, thank you for your review. The revised PDF (changes in blue) and the updated supplementary material are uploaded.

Q1 (NNCodec after the same fine-tuning). We coded the integer weights of the LSQ baseline (ResNet-18, same checkpoint, recipe and 20 epochs as PRESTO) with NNCodec's lossless coder: 9.22% at 70.09% top-1, against 5.74% at 69.91% for PRESTO with plain zstd, both as percentages of the quantized tensors, i.e. 38% smaller. NNCodec's coder would shrink PRESTO's own symbols by a further 4.1% (2.9% on ViT-B/16), so the gap comes from training for size, not from the coder. On ViT-B/16, where Table 1 compares with NNCodec, we have no LSQ checkpoint to code, but the LSQ packet is 10.48% of the checkpoint against PRESTO's 4.42%: NNCodec's coder would have to shrink it by more than half, where it shrinks the LSQ symbols of ResNet-18 by 1.2%. Limit: we did not run NNCodec's lossy mode with fine-tuning (Appendix C.10).

Q2 (repeats). Three runs of DeiT-Small (79.398 ± 0.028% at 9.083 ± 0.034% of the checkpoint) and EfficientNet-B0 (76.806 ± 0.049% at 10.092 ± 0.049%), two of ViT-B/16 (80.339 ± 0.047% at 4.455 ± 0.056%): standard deviations are at most 0.06 points of size and 0.03 to 0.05 of top-1, against size differences of 2 to 6 points with the LSQ baseline. The repeats were run with a different upper clamp on the counts of the dual solver than the first run of each row (the tight bound instead of the network size, Appendix A.2.2), so the spread includes that difference. Table 1 now has the rows of three repeats for DeiT-Small and EfficientNet-B0 (Appendix C.12). ResNet-50, a third ViT-B/16 run and the LSQ baselines remain single runs for now; we will complete the ± entries of Table 1 in the camera-ready version.

W1 (cost of the joint formulation against simpler options). Against LSQ the gain is large and survives the repeats. The cost is not intrinsic: calling the dual solver every 16 steps instead of 4 costs 1.5x LSQ instead of 3.8x with the same result on ResNet-18 (one run; not verified on other networks). Against simpler baselines the gain is modest at 20 epochs on ResNet-18: PRESTO's own sparsity term (T1 = T3 = 0, an L1 normalized by the quantization grid, so not an outside baseline) matches the full objective at LSQ cost (69.80% at 5.86% of the checkpoint against 69.67 to 69.91% at 5.83%); a plain L1 is 0.8 points worse, and magnitude pruning plus the same packet is below at both sizes we ran, by 0.2 points at 5.85% and by 0.4 at 7.5% (single runs; identical commands differ by 0.2). The entropy term lowers the size by about 7% on top (5.43% at 69.67%, against 5.86% at 69.80%; one run). Which term matters depends on the network and on the dose of the ridge, and we tested neither other networks nor longer budgets (Section 4.3, Appendix C.10).

W2 (equal conditions; single runs). The published points of Table 1 are not under equal conditions (nominal sizes and other checkpoints are marked). The like-for-like evidence is what we train ourselves with one recipe and one packet (above), and the repeats address the single runs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: what changed

We thank all reviewers. We ran the experiments they asked for and report the results, including those that do not favor the method. The revised PDF (changes in blue) and updated supplementary material (new code, commands and logs of the new runs) are uploaded; each reviewer's points are answered under their review. In short:

  • Deployment. The packet is written to a file and decoded back: same top-1 and identical logits on the first batch, on five checkpoints of four architectures, decoding in 0.12 to 1.14 s on a CPU. Counting the whole file (batch-norm statistics, header) adds 0.01 to 0.10 points of size, and 0.91 on EfficientNet-B0, whose factor becomes 9.1x instead of 9.9x.
  • Repeats. Three runs of DeiT-Small and EfficientNet-B0, two of ViT-B/16: standard deviations of at most 0.06 points of size and 0.05 of top-1 (Table 1 has the new rows).
  • Cost. Calling the dual solver every 16 steps instead of 4 costs 1.5x the LSQ baseline instead of 3.8x, with the same result on ResNet-18 (one run; not verified on other networks).
  • Baselines. On ResNet-18 at 20 epochs (one network, one budget, single runs) we compared PRESTO with a plain L1 on the latent weights, with global magnitude pruning, and with its own sparsity term alone (T1 = T3 = 0), which is an L1 on the weights in units of the learned grid. The plain L1 to which the objective reduces where its interior minimizer is feasible (λ = 2√(T1·T2) = 2e-6 at the dose of the paper, as Reviewer JsCX derived) leaves the packet at the LSQ size (9.37% against 9.42%), while the same coefficients inside PRESTO give 7.47%: the compression comes from tying the sparsity price to the learned grid. That sparsity term is PRESTO, not an outside baseline: it matches the default objective at the cost of the LSQ baseline, while a plain L1 is about 0.8 points of top-1 below it at equal size, and magnitude pruning is below PRESTO at both sizes we ran, by 0.2 points at 5.85% and by 0.4 at 7.5% (single runs; identical commands differ by 0.2). Without the ridge, adding the entropy term lowers the size by a further 7% (5.43% against 5.86%), which at this budget we cannot separate from a move along the same frontier. In the term ablations and the sequential pipelines of Section 4.3, all with the ridge, the objective with both terms also gave the smallest packet, by 3% to 6% over the better alternative. The comparison with the LSQ baseline is unchanged: the packet is still roughly halved on both ResNets and reduced by 58% on ViT-B/16 (Table 1). We make no claim for other networks or for the budgets of Table 1.
  • AlexNet. A controlled test does not support the allocation hypothesis; we now say we cannot attribute the deficit.

Still open: the value of the entropy term at longer budgets, comparisons with the lossy mode of NNCodec, product quantization and SPR under our recipe, and the remaining ± entries of Table 1, which we would complete in the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.