- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces an approach for mixed-precision quantization. It formulates the problem as a quadratic unconstrained binary optimization problem, the solution of which is used as initialization for a PROTES refinement.
Critical strengths
The results show some improvements over the home-brewed baseline.
The authors provided several ablation studies for their method.
Critical weaknesses
- Clarity
1.1. The paper is in general hard to follow, using terms that have not been clearly defined (e.g., the notion of recovery, convolution route, menu). Importantly, the main motivation and contribution(s) are not clearly stated. I strongly suggest the authors to explain what are the limitations of the existing solutions and what they propose to address these limitations.
1.2. Some parts of the paper are explained in vague terms. For example, the different terms in Eqs. 1 and 2 are only defined in words. Some of them are more precisely defined in the appendix, but not all. As such, the paper would not be reproducible. I understand that reproducibility may be palliated by making the code public, but this would still leave the reader with a feeling of vague understanding of the work.
- Scope
2.1. Based on the beginning of Section 3 and on the experiments, the work seems to focus on convolutional neural networks. It is unclear to me whether this is a true restriction or if the framework would in general apply to other models, e.g., transformers. If it would, demonstrating this empirically would be needed.
- Experiments
3.1. The main baseline seems outdated and is a home-brewed version of an actual method HAWQ-V3, which dates back from 2021. It is unclear to me why not compare to the actual HAWQ-V3 method, and, more importantly, why not consider more recent baselines. For example, the second part of the related work section mentions more recent work, but it is unclear why they would not be applicable here.
3.2. The results focus on one model, thus casting doubts on the generalization of the approach.
Questions for authors
See Weaknesses above.
2 comments
Revised manuscript: clarity, baselines, and scope
1.1–1.2 Presentation and equations. Sections 1–3 now state the problem and contributions and define “bit-width assignment,” “candidate bit-width set” and “LSQ+ fine-tuning,” replacing “route,” “menu” and “recovery,” respectively. Eqs. 1–9 give formulas for the allocation terms in the original Eqs. 1–2, and Appendices A–C give notation and protocols. The general comment explains the estimator correction and code release.
3.1 Baselines. Table 2 reports similar means after LSQ+ for Hessian-trace (tuned), FIT-based (tuned) and EdMIPS (ported) and lists Hessian-trace knapsack (as submitted) separately (Section 5.1). The EdMIPS port reached its search cap without meeting our assignment stability criterion. Section 2 discusses LIMPQ (2022), the closest related integer-programming allocator, which requires quantization-aware training and was outside our experimental comparison. OMPQ, EMQ and CLADO allocate only weight widths, under model-size limits.
In our ablations, fixing activations at 8 bits is infeasible below 3.125% within the tested weight range. Relative to joint allocation by Ours (exact), it scores 0.57 dB lower before fine-tuning at 3.42% but 0.01–0.02 dB higher after fine-tuning at 3.42–4.10% (Table 21). Tested with our quantizers, HAWQ-V3’s tied W = A ∈ {4,8} space yields static PSNR 4.9–6.3 dB below same-budget uniform with Hessian-trace sensitivity and 2.9–4.3 dB below with our objective (Table 20). The test covers the tied search space alone.
2.1 Transformers. In the static comparison in Section 5.2 (Table 3), Restormer reaches 39.39 dB with Ours (annealed), 38.66 dB with uniform W6A7 and 39.17 dB with Hessian-trace (tuned). FIT-based (tuned) and Ours (exact) were not run. We allocate only convolutions and keep attention products, normalization and softmax FP32, so the 4.10% allocated-layer budget corresponds to 12.86% of FP32 BOPs for counted convolutions and attention products (Appendix C.1).
3.2 More models. Section 5.2 and Table 3 compare five networks before fine-tuning. On the source-disjoint split, which separates source images but can share scenes, Ours (exact) gains 0.57 dB over the best fitting uniform at 4.10% in all three seeds, while our variants change rank across fine-tuning seeds at 2.93% (Section 5.3, Figure 3). ResNet-20 shows no consistent fine-tuned gain over uniform (Section 5.5, Table 14).
Correction to 3.1. Our first reply’s “+0.13 to +0.43 dB” compared Ours (annealed) with the uniform configuration at each budget. Against the best fitting uniform, Ours (exact) gains 0.17 and 0.13 dB at 2.93 and 3.42% in all three seeds, with mean gains below 0.01 dB at 4.10 and 4.69% (Table 2 and Appendix D, Table 8).
Thank you for the careful reading. We answer each point below.
1.1 Terms, motivation and contribution. The revision will define every term at first use: "route" becomes bit-width assignment, "menu" becomes candidate bit-width set, and recovery is LSQ+ fine-tuning. Its introduction will state the gap directly. The methods discussed here make different trade-offs: HAWQ-V3 ties weight and activation precision, OMPQ, EMQ and CLADO fix activation precision, and EdMIPS updates network weights during its search. We allocate weight and activation precision per layer, jointly, under a hard BOP budget, without weight updates.
1.2 Equations and reproducibility. Every term of Eqs. 1–2 will have a formula in the main text and an entry in a notation table, and we will release the code with the final version.
2.1 Scope. It is not a restriction in principle, because the formulation extends to other operators given their damage and cost profiles. However, our pipeline currently quantizes convolutional layers only, not fully connected layers, and does not model the attention products between two activations, which have no learned weight operand and would need their own damage profiles and cost terms. We have not tested a transformer, so the revision will scope its claims to convolutional networks. To test transfer beyond denoising we added ResNet-20 on CIFAR-10. The candidate set transfers, but the weight/activation balance does not: with the denoiser's balance our allocation is below uniform, and with a balance selected on validation it is within 0.2 points of uniform, not better. The revision will report this as a limitation.
3.1 Baselines.
HAWQ-V3 as published. It gives each layer one bit-width shared by weights and activations (INT4 or INT8), so on our problem any budget below W8A8 (6.25 % BOPs) forces 4-bit activations. We tested this on HalfUNet with a HAWQ-style sensitivity, an exact solver, our quantizers and the skip constraint: before fine-tuning, test PSNR is 4.9 to 6.3 dB below uniform at our four budgets (2.9 to 4.3 dB with our own objective). This describes our static stack, not the published system, because we use dynamic activation quantization and did not fine-tune these rows, whereas HAWQ-V3 uses quantization-aware training. We therefore kept HAWQ-V3's Hessian-trace sensitivity and budgeted allocation but applied them to our joint W/A candidate set. The submitted row will be called "Hessian-trace knapsack", and a new row adds exact hard-budget allocation and a validation-tuned W/A balance.
More recent work. The QUBO papers in our related-work section address other compression problems (rounding at a fixed precision, joint pruning and quantization, channel pruning), so we cite them as related formulations and did not adapt them into bit-width allocators. The recent mixed-precision allocators OMPQ, EMQ and CLADO allocate weight bits only, under a model-size budget, so comparing them in our joint W/A space would require adding per-layer activation decisions and replacing the model-size constraint with a BOP budget.
What we added instead. Three baselines in our joint W/A space, each with three LSQ+ seeds: the tuned Hessian-trace allocator, a FIT-based allocator (FIT's metric, minimized with our exact solver) and EdMIPS's differentiable joint W/A search, ported to our candidate set and quantizers. After fine-tuning, their means and ours (annealed, corrected estimator) are within about 0.1 dB at every budget, while ours beats uniform and the submitted baseline in every seed at the three tight budgets (+0.13 to +0.43 dB over uniform). We do not claim better sensitivity terms: before fine-tuning the FIT-based allocator leads ours by +0.02 to +0.31 dB with the same tuning, and after fine-tuning EdMIPS is +0.02 to +0.08 dB ahead of ours at three of four budgets. EdMIPS, however, updates network weights and needed ten GPU searches (30 epochs each, our cap; none met our convergence criterion). Our contribution is therefore the joint, BOP-constrained formulation, which an exact MILP solves in under 2 s.
3.2 "Results focus on one model". The paper evaluates four denoisers: HalfUNet and three public checkpoints (NAFNet-32, Baseline-32, HINet-0.5×). On the public checkpoints, calibration and evaluation use disjoint source images, and with the corrected weight-error estimator (see our reply to Reviewer 2nLz) our allocation leads the tuned Hessian-trace allocator on all three before fine-tuning (+0.05 to +0.21 dB). The primary HalfUNet split does share source images between its partitions. The revision will state this, and a rerun on a source-image-disjoint split is scheduled, as is ResNet-20 fine-tuning.