- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BlockFAMQ, a mixed-precision allocation method for Mamba quantization. It scores frozen quantization candidates with token-level gradient-residual products, preserves within-block cross-coordinate interactions, and selects precisions under an exact serialized-byte budget. Experiments on Mamba-1 models from 130M to 2.8B show lower NLL and higher downstream accuracy than diagonal and activation-weighted allocation, especially at 2.5-3.5 bits. Direct candidate KL is stronger at 3.0 bits, while a restricted block-to-KL hybrid recovers part of that gain.
Critical strengths
The common candidate bank, partition, serializer, and byte budget make the main allocator comparison relatively clean. The budget sweep shows consistent gains at aggressive bit rates and includes several useful sensitivity studies.
Critical weaknesses
- Direct candidate KL outperforms BlockFAMQ at 3.0 bits, but is evaluated only on the two largest Mamba-1 models. The paper therefore does not establish when the proposed approximation is preferable in its main target regime, weakening the case for the method.
- Generality is not sufficiently established: most evidence is on Mamba-1, while cross-family transfer is limited to isolated operating points and changes both architecture and packing. It is therefore unclear whether the observed gains persist across model families and budgets.
Questions for authors
- Can the authors provide direct-KL versus BlockFAMQ results for the 130M and 370M models and across the main budgets, together with the corresponding scoring cost?
- Can the authors add or clarify matched experiments across additional model families, architectures, and budgets, keeping packing conditions comparable so the source of the reported gains is identifiable?
2 comments
Response to Reviewer roAG (2/2): Cross-family comparisons and NLL scope
Cross-family comparisons
Table 28 already includes Mamba-2 2.7B, Falcon-Mamba 7B, Pythia-1.4B, and Llama-3.2-3B, but those operating points have model-specific packing and effective-bit targets. They establish transfer within the reported configurations, not a controlled architecture comparison.
In Table 28, block allocation lowers NLL relative to diagonal allocation on all four families: by 0.035 nats on Mamba-2 2.7B at 4.05 bits, 0.027 on Falcon-Mamba 7B at 4.02 bits, 0.009 on Pythia-1.4B at 4.04 bits, and 0.007 on Llama-3.2-3B at 4.03 bits. The two recurrent models show the larger gaps, with intervals of [−0.047, −0.023] and [−0.037, −0.017] (Section 6). Rows share a partition and candidate bank within each model, but the panel does not match architecture, pretraining data, or model size across rows.
The matched-scale comparison is Panel B of Table 3. Mamba-1 1.4B and Pythia-1.4B share target bits, candidate precision grid, score-token counts, and bootstrap construction. The block–diagonal gap is −0.133 versus −0.028 nats at 3.5 bits and −0.023 versus −0.009 at 4.0 bits. The paired difference of these gaps is −0.105 [−0.130, −0.080] at 3.5 bits and −0.014 [−0.026, −0.003] at 4.0 bits. This holds model scale and pretraining family approximately fixed, but it does not isolate recurrence from parameter structure or residual geometry.
Appendix L of the revision adds the attention–recurrent Falcon-H1-7B at 2.5 and 3.0 bits, described in our reply to Reviewer jXg8.
NLL scope of the block score
Theorems 5–6 remain valid conditional results, and the revision makes the scope of the NLL claim explicit. Theorem 5 expands predictive KL, whose value and gradient vanish at zero displacement. Evaluation NLL need not have zero value or gradient at zero displacement, so the revision gives it a separate Taylor bound.
Let be the complete serialized displacement, where is the residual of the fixed tensors and is the residual of unit . Let be twice continuously differentiable with -Lipschitz Hessian along every feasible displacement segment, and let . Let be the within-unit empirical Fisher on all coordinates, with unit 0 holding the fixed tensors as in Appendix B.4. Then
-\tfrac12e^\top\widehat F_Be\right| \le \tfrac12\|H-\widehat F_B\|_{\mathrm{op}}\|e\|^2 +\tfrac{\rho_H}{6}\|e\|^3.$$ The integral Taylor remainder is bounded by $\rho_H\|e\|^3\int_0^1t(1-t)\,dt=\rho_H\|e\|^3/6$. Adding and subtracting the empirical-Fisher quadratic gives the inequality. Here $\tfrac12e^\top\widehat F_Be$ is the raw block score of Equation 5 summed over units, plus a fixed-unit term that is the same for every assignment. The bound identifies three possible error sources: the first-order NLL term, the curvature mismatch, and the finite-step remainder. The mismatch also contains the difference between calibration and evaluation data, because $\widehat F_B$ uses calibration positions and $H$ uses the evaluation set. The bound is not a numerical NLL guarantee, because $\rho_H$ and $\|H-\widehat F_B\|_{\mathrm{op}}$ are not estimated. Unlike predictive KL, evaluation NLL generally has a nonzero first-order term; using the calibration empirical Fisher introduces curvature mismatch. Both expansions retain a finite-step remainder. The raw score is the primary objective, so the bound applies to it directly. For the optional clipped score of Equation 7, let $S_{\mathrm{clip}}(e)$ be the clipped score summed over units, plus the same fixed-unit term. When $S_{\mathrm{clip}}(e)$ replaces $\tfrac12e^\top\widehat F_Be$ on the left-hand side, the triangle inequality adds $|S_{\mathrm{clip}}(e)-\tfrac12e^\top\widehat F_Be|$ to the right-hand side. The extra term appears because a clipped score is generally not a candidate-independent quadratic. Range clipping inside the quantizer is a different operation. It only changes which residual a candidate has, so the bound still applies to that residual. In both cases, $e$ must be the actual serialized residual. For fixed curvature-mismatch and Hessian-Lipschitz constants, the corresponding upper-bound terms scale quadratically and cubically with $\|e\|$, respectively. This is consistent with the larger direct-KL gains at 3.0 bits in Table 4, but the bound does not show that these terms cause them. The block score keeps the narrower conditional allocation claim of Theorem 6. The revision states this bound as Proposition 10 in the new Appendix K. Please let us know whether these results answer your questions. We will reply to any further point during the discussion period.
Response to Reviewer roAG (1/2): Direct-KL results across model sizes and budgets
Thank you for the review. The table summarizes our answers. Part 1 gives the direct-KL grid and timing details; Part 2 covers the cross-family comparisons and NLL bound.
Direct KL across model sizes and byte budgets
Direct KL beats BlockFAMQ at aggressive budgets on the larger Mamba-1 models, and the difference shrinks to 0.003 nats or less at 4.0 and 4.5 bits. Table 4 reports 1.4B and 2.8B at 3.0, 3.5, and 4.0 bits. Direct KL improves NLL over BlockFAMQ by 0.041 and 0.035 nats at 3.0 bits, respectively. This identifies a genuine limitation of the local approximation. Its advantage decreases at 3.5 and 4.0 bits, where the reported paired intervals include zero. For this response, the comparison was extended to all four Mamba-1 sizes at five effective-bit targets, adding 14 cells to the six in Table 4.
The extension retains the frozen candidate bank, partition, complete-checkpoint serializer, calibration draws, and integer-byte optimizer. Each entry gives direct-KL-minus-BlockFAMQ NLL / direct-KL-to-block scoring-time ratio. Negative NLL differences favor direct KL. Comparisons use the same serialized-byte ceiling within each model and budget. The 1.4B and 2.8B cells at 3.0, 3.5, and 4.0 bits repeat Table 4.
Direct KL helps most at 2.5 bits, where its advantage grows with model size: 0.020, 0.045, 0.090, and 0.105 nats from 130M to 2.8B. At 4.0 and 4.5 bits, every difference is 0.003 nats or smaller, in either direction. The Table 4 intervals at 4.0 bits include zero, so we do not read these small differences as wins for either method.
Scoring time includes all candidates considered by the allocator, including shortlist construction, on common hardware and precision. Candidate-bank construction is separate from scoring and included in end-to-end calibration cost. Comparisons are paired within independent calibration draws. Individual tokens are not independent replications.
Every grid cell averages the same five calibration draws as the main study. The six cells that repeat Table 4 have paired 95% intervals there. Table 5 separately reports serialized-byte audits for BlockFAMQ and the block-to-KL hybrid at those model sizes and budgets; it does not report direct-KL checkpoint sizes.
BlockFAMQ scores the frozen candidate bank once and reuses its scores across budgets. The reported direct-KL-to-BlockFAMQ scoring-time ratios vary across budgets; for example, they are 2.51×, 2.44×, and 2.41× for 1.4B at 3.0, 3.5, and 4.0 bits. These are measured runtime ratios, not candidate-count ratios. The block-to-KL shortlist is rebuilt for each budget because its anchor is the budget-specific BlockFAMQ assignment.