- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how well the effects of individual quantization changes predict their combined impact on low-bit LLMs. The authors distinguish errors in estimating individual changes from errors caused by interactions between changes, and study these effects across several quantized models.
The experiments show that accurate estimates of individual changes can lead to large errors when multiple changes are applied together. The authors further analyze these interactions using discrete state lattices and show that pairwise information can improve predictions of unseen combinations. However, the improvements in prediction do not always translate into better model selection, and the benefits of interaction-aware methods vary across settings.
Critical strengths
-
Clear motivation and well-designed experiments. By separating individual estimation errors from interaction effects, the experiments provide convincing evidence that accurate predictions are not sufficient for reliable combination selection.
-
Sound technical analysis with meaningful empirical findings. In particular, the paper demonstrates contextual sign reversals and shows that incorporating pairwise interactions improves predictions of unseen combinations.
Critical weaknesses
-
Limited generalizability of the findings. The main interaction experiments focus on small 8-move lattices, with five overlapping moves between the two Qwen settings. While the results demonstrate clear contextual interactions, no statistically significant sign reversals are confirmed on Llama-3.2-3B.
-
Incomplete comparison with nonlinear additive baselines. The paper shows that contextual reversals cannot be exactly captured by monotone additive models, but such models may still provide useful approximations in practice. Although Table 12 includes a monotone single-index baseline, it does not compare its loss-space coverage or selection regret with those of the quadratic model.
-
Limited evidence of benefits from additional combination queries. The endpoint beam does not consistently outperform simpler support-limited additive selection. For example, results indicate that cap-four addition achieves better ARC-KL than the beam, while the beam's NLL improvement over cap-two addition is not statistically resolved. These results leave the additional computational cost of combination queries difficult to justify across settings.
Questions for authors
-
Could the authors test more quantization combinations from the existing data and report how often significant contextual reversals lead to better loss coverage?
-
Could the authors compare the monotone-index model in Table 12 with the quadratic model using the same quality coverage and selection error metrics?
2 comments
Response to the rebuttal
Thank you for the detailed rebuttal and additional experiments. The new results help address my concerns regarding contextual reversals. However, the generalizability of these findings of the proposed approach remain somewhat limited, particularly given the inconsistent downstream improvements. I appreciate the authors' efforts but will maintain my original rating.
Response to Reviewer NU34
Thank you for pointing directly to the two missing quantitative links: tying reversals to downstream decision coverage, and evaluating the monotone single-index baseline with the same decision metrics as the quadratic model.
Q1. Across more existing combinations, how often do resolved reversals translate into better coverage? We enumerated all 3,394 four- to eight-dimensional sub-lattices (faces) of both 256-state Qwen grids (1,697 faces each; Appendix D.1, Table 16), evaluating local base/singleton/pair anchors against unseen higher-support combinations while retaining the parent simultaneous max-t reversal family. At dimensions 4 and 5, quadratic selection improves coverage over exact addition roughly 3x more often when an ARC reversal is present on Stratified—whereas on Witness (which lacks strong helpful reversals), gains are rare either way:
Note that overlapping faces form a descriptive census (significance applies to the parent reversal tests, not per-face counts, and Appendix D.1 includes negative/zero differences plus 1,154 support-split sensitivity checks). Overall, context reversals identify regions where pair modeling helps, rather than guaranteeing pair gains everywhere.
Q2. Monotone single-index vs. quadratic on quality coverage and selection regret. Table 4 (§4) now puts all five models on identical observations across 2,000 paired bootstrap refits (small-support splits: ≤3 on Qwen, ≤2 on Llama):
Two boundaries stay explicit in Appendix D: regret CIs are pointwise, and under a strict equal-K returned-set control, no pairwise vs. additive coverage difference survives Bonferroni correction across the five tested K sizes.
Generalizability & value of combination queries. On standard INT3 Llama-3.2-3B, the 8-move replacement lattice still resolves no sign reversals; we separate that from a fixed-edit scale control (applying the identical 4-projection scale update across two standard INT3 contexts), which resolves opposite NLL signs on WikiText (+0.0013 vs. −0.0008) and C4 (+0.0011 vs. −0.0011; Appendix L.2). Practically, rather than spending extra queries to beat cap-2 on an expensive beam, PTV redirects complete combination queries toward fast acceptance on cheap proposals—achieving +3.55 [1.68, 5.48] pp normalized ARC-Challenge accuracy over full-price cap-two at 12% of total forward work on 3B (Table 5), while Figure 4 transparently shows the settings (new panels, act-order 1B, Qwen3-1.7B) where task gains vanish.