- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This article proposes IDScore, a task conditional likelihood correction method for low bit quantization perception training. This method utilizes task gradients to select adjacent quantization targets, and combines latent parameters with quantization thresholds to construct ordinal likelihood scores, which are added as bounded residuals to existing backpropagation rules. In theory, this article analyzes the consistency of target direction, residual allocation under limited budget, and local threshold approximation properties. The experiment covered visual and language tasks, multiple bit widths, and up to 430 million parameter scales on LSQ and QuEST, and further evaluated the combination effect with FOGZO and CAGE, optimizer adaptability, computational cost, and ablation results.
Critical strengths
-
It can improve the tested QAT baseline in both visual and linguistic scenarios, as well as in various quantization accuracies. In some high demand low bit settings, the improvement is particularly significant. At the same time, the measurement cost introduced by this method in the QuEST benchmark test report is relatively small, which makes it practically applicable.
-
This article not only reports the performance of the final task, but also investigates different amplitude allocation, residual support, optimizer paths, and combinations with FOGZO and CAGE. These experiments help to separate the several components of the proposed correction method and demonstrate that IDScore can serve as a supplement to existing gradient estimation methods without completely replacing the underlying QAT method.
Critical weaknesses
-
The biggest problem is that the current evidence is not sufficient to clearly prove that a complete ordinal likelihood construction is necessary for the reported performance improvement. The core motivation of IDScore is to utilize ordered threshold structures through target conditional ordinal likelihood. However, Boundary-NLL control has improved the performance of QuEST from 37.262 PPL to 36.611 PPL, while the full IDScore has reached 36.502. Therefore, in this case, a simpler single bound likelihood seems to be able to restore most of the performance improvement. This makes it difficult for me to determine whether aggregating all quantization threshold information is crucial for this method, especially when ordinal scores already exist.
-
This article proposes IDScore to improve proxy gradient estimation, but the main evaluation focuses on downstream accuracy or perplexity. Is there direct evidence to suggest that IDScore can generate more reliable gradient estimates, such as higher consistency with finite difference or zero order reference values in easily manageable scenarios?
-
The PPL of Boundary-NLL is 36.611, while the complete IDScore is 36.502 and QuEST is 37.262. Can the author provide a comparison of the matching between Boundary-NLL and the complete ordinal IDScore in stages W1A1, W2A2, W3A3, and W4A4?
Questions for authors
Please see the weaknesses.
2 comments
Response to Weakness 2: Direct comparison with a zeroth-order reference.
Thank you for your insightful comment. We added a controlled local diagnostic comparing LSQ-STE and LSQ+IDScore at the same frozen ResNet-18/ImageNet-100 checkpoints while retaining the same forward quantizer. For each sampled weight pair, we evaluate the task loss at all 16 joint quantized states and construct a local Gaussian-smoothed loss from these evaluations.
Let be the normalized coordinates of a sampled weight pair. Let be the task loss when the pair is set to quantization levels indexed by , with all other weights, quantization scales, and validation inputs held fixed. Define , where is the quantization cell for level and are independent standard Gaussian variables. The smoothed loss and its reference gradient are
= \sum_{k=0}^{3}\sum_{\ell=0}^{3} p_{ik}(u_i;\sigma)\\,p_{j\ell}(u_j;\sigma)\\, \mathcal{L}_{k\ell},$$ $$\mathbf{h}_{\sigma} = \nabla_{(u_i,u_j)} \widetilde{\mathcal{L}}_{\sigma}(u_i,u_j).$$ We compute cosine similarity by pooling the sampled coordinates within each seed, with both backward signals expressed in the same normalized coordinates. The reference is the gradient of this local smoothed loss, rather than of the hard-quantized objective. The paired cosine gain is positive for each of the three seeds at both smoothing scales. The mean gain is $9.33\times10^{-5}$ at $\sigma=1$ and $7.27\times10^{-5}$ at $\sigma=0.5$. These results indicate a small but consistent improvement in local directional alignment across the tested seeds. | Smoothing scale | Seed | LSQ-STE | LSQ+IDScore | Paired gain ($10^{-5}$) | |---|---|---:|---:|---:| | $\sigma=1$ (primary) | 0 | 0.92027323 | 0.92033914 | +6.591 | | | 3 | 0.93780127 | 0.93789688 | +9.561 | | | 4 | 0.90085123 | 0.90096947 | +11.824 | | | Mean | 0.91964191 | 0.91973516 | +9.33 | | $\sigma=0.5$ | 0 | 0.95049302 | 0.95053788 | +4.486 | | | 3 | 0.95666936 | 0.95675166 | +8.229 | | | 4 | 0.93895217 | 0.93904298 | +9.081 | | | Mean | 0.94870485 | 0.94877751 | +7.27 | We use LSQ for this comparison because IDScore preserves the support and signs of the weight gradient in the tested setting, allowing us to examine the effect of reweighting relative gradient magnitudes. In QuEST, the correction can additionally supply nonzero signals outside the trust mask, so alignment changes would reflect both support changes and magnitude reweighting. We have added this comparison to Section 4.4, "Alignment with a zeroth-order reference." This diagnostic complements the downstream performance results by directly examining alignment with a reference constructed from task-loss evaluations.
Response to Weaknesses 1 and 3: Added value of all-threshold aggregation
Thank you for pointing out that Boundary-NLL already captures most of the improvement. We agree with this observation: in the original W2A2 experiment, Boundary-NLL reduces PPL from 37.262 to 36.611, while ordinal IDScore further reduces it to 36.502. Our main contribution is a task-conditioned likelihood correction for QAT, with the all-threshold ordinal likelihood as our default construction. The gains from Boundary-NLL support the usefulness of this shared likelihood-based correction principle. Our revised comparison therefore examines the additional value of all-threshold aggregation.
For a comparison with matched residual support, we added Boundary-NLL-matched, which uses the same residual support and post-processing rules as Ordinal. The original Boundary-NLL scores only coordinates with feasible neighboring transitions. The matched variant additionally handles coordinates whose clipped target remains at the current level, using the nearest finite boundary of the retained cell. Its W2A2 PPL is 36.673. We retain the original 36.611 result in Table 14 and distinguish these two implementations in Appendix D.5.
The W2A2–W4A4 comparisons use the same code snapshot and matched training configurations within each bit width. All runs use QuEST-30M on C4 with a sequence length of 512 and AdamW with a cosine learning-rate schedule. W1A1 uses 11,444 updates with a global batch of 256 sequences (about 1.5B tokens), W2A2 uses 91,552 updates with a global batch of 64 (about 3B tokens), and W3A3 and W4A4 use 91,552 updates with a global batch of 32 (about 1.5B tokens each). For the W2A2–W4A4 reruns, Boundary-NLL-matched and Ordinal use the same training configuration within each bit width.
Across three seeds at each of W2A2, W3A3, and W4A4, Ordinal achieves lower PPL in all nine paired comparisons, with mean paired reductions of 0.172, 0.197, and 0.141, respectively (as shown in the table below). At W1A1, there is only one threshold, so the two score constructions coincide under the same adapter and post-processing rules. We therefore did not separately train this identical control. The reported W1A1 setting uses 11,444 updates with global batch size 256, corresponding to approximately 1.5B input tokens.
Table 1. Final validation PPL on QuEST-30M (lower is better).
*At W1A1, there is only one threshold, so the two score constructions coincide under the same adapter and post-processing rules. We therefore did not separately train this identical control. Positive paired gains indicate lower PPL for Ordinal.
The distinction between the constructions is that a single-boundary score encodes the selected threshold event, whereas the ordinal score encodes the target level's relationships to all thresholds. These constructions produce different coordinate-wise correction magnitudes. The matched comparisons support the additional empirical value of its ordinal construction in the tested multi-bit settings.
This additional benefit comes with 6.34% and 2.03% more step time at W3A3 and W4A4, respectively, relative to Boundary-NLL-matched, with nearly unchanged peak allocated memory. W2A2 timings vary more across trials, so we do not interpret the negative mean time difference as a speedup. The complete timing measurements and protocol are reported in Appendix D.5, Table 17.
Together, these results support our choice of the ordinal construction as the default implementation. The revised text following Eq. (5) clarifies the role of all-threshold aggregation, Section 4.4 discusses its added empirical value, and Appendix D.5 reports the cross-bit-width comparisons and efficiency measurements. All corresponding revisions are highlighted in blue.