acceptodds
Under review as a conference paper at ICLR 2027

BlockFAMQ: Realized-Residual Curvature for Complete-Checkpoint Mamba Quantization

14 views

Abstract

Mixed-precision quantization must decide which realized rounding residuals receive scarce high-precision bytes, but diagonal sensitivity scores discard the signed coordinate interactions that distinguish deterministic candidates. We introduce BlockFAMQ, a matrix-free allocator that scores squared per-token gradient–residual products and solves the resulting integer-byte multiple-choice problem under a complete-checkpoint budget. Across Mamba-1 130M, 370M, 1.4B, and 2.8B, the mean BlockFAMQ-minus-diagonal WikiText-2 NLL gap contracts from −0.218 at 2.5 effective bits to −0.026 at 4.0 bits and −0.009 at 4.5 bits. At 3.5 bits, BlockFAMQ improves six-task accuracy by 2.08 points and recovers 65.7–80.1% of the diagonal quantization gap; an activation-weighted reconstruction baseline narrows but does not remove the advantage. At 3.0 bits, direct candidate KL improves on BlockFAMQ by 0.038 nats across 1.4B and 2.8B, while a feasibility-anchored block-to-KL shortlist recovers 0.033 nats at 1.62× the block-scoring cost on average. Precision shifts toward x_proj and in_proj, whose realized residuals have the largest cross-term tails, and the block effect is larger on Mamba-1 1.4B than on matched-scale Pythia-1.4B. The direction persists on Mamba-2 2.7B and Falcon-Mamba 7B; packed W4A8 composition retains 96.8–97.7% of Quamba2 decode throughput while lowering NLL. These results support realized block curvature as a budget-dependent allocation correction, with direct KL becoming more useful when aggressive quantization leaves the local quadratic regime.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, mixed-precision quantization, state-space models, Mamba, empirical Fisher information, model compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_roAGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BlockFAMQ, a mixed-precision allocation method for Mamba quantization. It scores frozen quantization candidates with token-level gradient-residual products, preserves within-block cross-coordinate interactions, and selects precisions under an exact serialized-byte budget. Experiments on Mamba-1 models from 130M to 2.8B show lower NLL and higher downstream accuracy than diagonal and activation-weighted allocation, especially at 2.5-3.5 bits. Direct candidate KL is stronger at 3.0 bits, while a restricted block-to-KL hybrid recovers part of that gain.

Critical strengths

The common candidate bank, partition, serializer, and byte budget make the main allocator comparison relatively clean. The budget sweep shows consistent gains at aggressive bit rates and includes several useful sensitivity studies.

Critical weaknesses
  1. Direct candidate KL outperforms BlockFAMQ at 3.0 bits, but is evaluated only on the two largest Mamba-1 models. The paper therefore does not establish when the proposed approximation is preferable in its main target regime, weakening the case for the method.
  2. Generality is not sufficiently established: most evidence is on Mamba-1, while cross-family transfer is limited to isolated operating points and changes both architecture and packing. It is therefore unclear whether the observed gains persist across model families and budgets.
Questions for authors
  1. Can the authors provide direct-KL versus BlockFAMQ results for the 130M and 370M models and across the main budgets, together with the corresponding scoring cost?
  2. Can the authors add or clarify matched experiments across additional model families, architectures, and budgets, keeping packing conditions comparable so the source of the reported gains is identifiable?
2 comments
AuthorsOpenReview ↗

Response to Reviewer roAG (1/2): Direct-KL results across model sizes and budgets

Thank you for the review. The table summarizes our answers. Part 1 gives the direct-KL grid and timing details; Part 2 covers the cross-family comparisons and NLL bound.

Your pointOur answerWhere in the revision
Direct KL across model sizes and budgetsDirect KL beats BlockFAMQ by 0.041 and 0.035 nats at 3.0 bits on 1.4B and 2.8B, and the paired intervals include zero at 3.5 and 4.0 bits. Across four sizes and five budgets, its advantage is largest at 2.5 bits, and every difference at 4.0 and 4.5 bits is 0.003 nats or smaller.Table 4; grid below
Cost of direct KLDirect KL takes 1.75× to 3.20× the BlockFAMQ scoring time, more on larger models. These are measured scoring-time ratios for the evaluated model–budget settings.Table 4; Appendix C.5
Cross-family comparisonsBlock allocation lowers NLL relative to diagonal allocation on all four families in Table 28, most on Mamba-2 2.7B and Falcon-Mamba 7B (0.035 and 0.027 nats). At matched scale and 3.5 bits, the gap is −0.133 nats on Mamba-1 1.4B against −0.028 on Pythia-1.4B.Table 28; Table 3, Panel B
NLL scope of the block scoreProposition 10 splits the gap between the NLL change and the block score into a first-order term, a curvature mismatch, and a finite-step remainder. It identifies these error sources and is not a numerical NLL guarantee.Appendix K, Proposition 10
A larger hybridFalcon-H1-7B-Base at 2.5 and 3.0 bits, described in our reply to Reviewer jXg8.Appendix L

Direct KL across model sizes and byte budgets

Direct KL beats BlockFAMQ at aggressive budgets on the larger Mamba-1 models, and the difference shrinks to 0.003 nats or less at 4.0 and 4.5 bits. Table 4 reports 1.4B and 2.8B at 3.0, 3.5, and 4.0 bits. Direct KL improves NLL over BlockFAMQ by 0.041 and 0.035 nats at 3.0 bits, respectively. This identifies a genuine limitation of the local approximation. Its advantage decreases at 3.5 and 4.0 bits, where the reported paired intervals include zero. For this response, the comparison was extended to all four Mamba-1 sizes at five effective-bit targets, adding 14 cells to the six in Table 4.

The extension retains the frozen candidate bank, partition, complete-checkpoint serializer, calibration draws, and integer-byte optimizer. Each entry gives direct-KL-minus-BlockFAMQ NLL / direct-KL-to-block scoring-time ratio. Negative NLL differences favor direct KL. Comparisons use the same serialized-byte ceiling within each model and budget. The 1.4B and 2.8B cells at 3.0, 3.5, and 4.0 bits repeat Table 4.

Mamba-1 model2.5 bits3.0 bits3.5 bits4.0 bits4.5 bits
130M−0.020 / 1.80×+0.006 / 1.75×−0.002 / 1.82×+0.003 / 1.78×−0.001 / 1.85×
370M−0.045 / 2.05×−0.012 / 2.00×+0.002 / 2.08×−0.001 / 2.03×+0.001 / 2.10×
1.4B−0.090 / 2.60×−0.041 / 2.51×−0.008 / 2.44×−0.003 / 2.41×+0.001 / 2.40×
2.8B−0.105 / 3.20×−0.035 / 3.05×−0.006 / 2.90×−0.002 / 2.84×−0.001 / 2.85×

Direct KL helps most at 2.5 bits, where its advantage grows with model size: 0.020, 0.045, 0.090, and 0.105 nats from 130M to 2.8B. At 4.0 and 4.5 bits, every difference is 0.003 nats or smaller, in either direction. The Table 4 intervals at 4.0 bits include zero, so we do not read these small differences as wins for either method.

Scoring time includes all candidates considered by the allocator, including shortlist construction, on common hardware and precision. Candidate-bank construction is separate from scoring and included in end-to-end calibration cost. Comparisons are paired within independent calibration draws. Individual tokens are not independent replications.

Every grid cell averages the same five calibration draws as the main study. The six cells that repeat Table 4 have paired 95% intervals there. Table 5 separately reports serialized-byte audits for BlockFAMQ and the block-to-KL hybrid at those model sizes and budgets; it does not report direct-KL checkpoint sizes.

BlockFAMQ scores the frozen candidate bank once and reuses its scores across budgets. The reported direct-KL-to-BlockFAMQ scoring-time ratios vary across budgets; for example, they are 2.51×, 2.44×, and 2.41× for 1.4B at 3.0, 3.5, and 4.0 bits. These are measured runtime ratios, not candidate-count ratios. The block-to-KL shortlist is rebuilt for each budget because its anchor is the budget-specific BlockFAMQ assignment.

AuthorsOpenReview ↗

Response to Reviewer roAG (2/2): Cross-family comparisons and NLL scope

Cross-family comparisons

Table 28 already includes Mamba-2 2.7B, Falcon-Mamba 7B, Pythia-1.4B, and Llama-3.2-3B, but those operating points have model-specific packing and effective-bit targets. They establish transfer within the reported configurations, not a controlled architecture comparison.

In Table 28, block allocation lowers NLL relative to diagonal allocation on all four families: by 0.035 nats on Mamba-2 2.7B at 4.05 bits, 0.027 on Falcon-Mamba 7B at 4.02 bits, 0.009 on Pythia-1.4B at 4.04 bits, and 0.007 on Llama-3.2-3B at 4.03 bits. The two recurrent models show the larger gaps, with intervals of [−0.047, −0.023] and [−0.037, −0.017] (Section 6). Rows share a partition and candidate bank within each model, but the panel does not match architecture, pretraining data, or model size across rows.

The matched-scale comparison is Panel B of Table 3. Mamba-1 1.4B and Pythia-1.4B share target bits, candidate precision grid, score-token counts, and bootstrap construction. The block–diagonal gap is −0.133 versus −0.028 nats at 3.5 bits and −0.023 versus −0.009 at 4.0 bits. The paired difference of these gaps is −0.105 [−0.130, −0.080] at 3.5 bits and −0.014 [−0.026, −0.003] at 4.0 bits. This holds model scale and pretraining family approximately fixed, but it does not isolate recurrence from parameter structure or residual geometry.

Appendix L of the revision adds the attention–recurrent Falcon-H1-7B at 2.5 and 3.0 bits, described in our reply to Reviewer jXg8.

NLL scope of the block score

Theorems 5–6 remain valid conditional results, and the revision makes the scope of the NLL claim explicit. Theorem 5 expands predictive KL, whose value and gradient vanish at zero displacement. Evaluation NLL need not have zero value or gradient at zero displacement, so the revision gives it a separate Taylor bound.

Let e=(e0,e1,…,eU)e=(e_0,e_1,\dots,e_U) be the complete serialized displacement, where e0e_0 is the residual of the fixed tensors and eue_u is the residual of unit uu. Let LL be twice continuously differentiable with ρH\rho_H-Lipschitz Hessian along every feasible displacement segment, and let H=∇2L(w)H=\nabla^2L(w). Let F^B=blockdiag(F^00,F^11,…,F^UU)\widehat F_B=\mathrm{blockdiag}(\widehat F_{00},\widehat F_{11},\dots,\widehat F_{UU}) be the within-unit empirical Fisher on all coordinates, with unit 0 holding the fixed tensors as in Appendix B.4. Then

-\tfrac12e^\top\widehat F_Be\right| \le \tfrac12\|H-\widehat F_B\|_{\mathrm{op}}\|e\|^2 +\tfrac{\rho_H}{6}\|e\|^3.$$ The integral Taylor remainder is bounded by $\rho_H\|e\|^3\int_0^1t(1-t)\,dt=\rho_H\|e\|^3/6$. Adding and subtracting the empirical-Fisher quadratic gives the inequality. Here $\tfrac12e^\top\widehat F_Be$ is the raw block score of Equation 5 summed over units, plus a fixed-unit term that is the same for every assignment. The bound identifies three possible error sources: the first-order NLL term, the curvature mismatch, and the finite-step remainder. The mismatch also contains the difference between calibration and evaluation data, because $\widehat F_B$ uses calibration positions and $H$ uses the evaluation set. The bound is not a numerical NLL guarantee, because $\rho_H$ and $\|H-\widehat F_B\|_{\mathrm{op}}$ are not estimated. Unlike predictive KL, evaluation NLL generally has a nonzero first-order term; using the calibration empirical Fisher introduces curvature mismatch. Both expansions retain a finite-step remainder. The raw score is the primary objective, so the bound applies to it directly. For the optional clipped score of Equation 7, let $S_{\mathrm{clip}}(e)$ be the clipped score summed over units, plus the same fixed-unit term. When $S_{\mathrm{clip}}(e)$ replaces $\tfrac12e^\top\widehat F_Be$ on the left-hand side, the triangle inequality adds $|S_{\mathrm{clip}}(e)-\tfrac12e^\top\widehat F_Be|$ to the right-hand side. The extra term appears because a clipped score is generally not a candidate-independent quadratic. Range clipping inside the quantizer is a different operation. It only changes which residual a candidate has, so the bound still applies to that residual. In both cases, $e$ must be the actual serialized residual. For fixed curvature-mismatch and Hessian-Lipschitz constants, the corresponding upper-bound terms scale quadratically and cubically with $\|e\|$, respectively. This is consistent with the larger direct-KL gains at 3.0 bits in Table 4, but the bound does not show that these terms cause them. The block score keeps the narrower conditional allocation claim of Theorem 6. The revision states this bound as Proposition 10 in the new Appendix K. Please let us know whether these results answer your questions. We will reply to any further point during the discussion period.
Reviewer_4sGxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BlockFAMQ addresses the allocation step of mixed-precision PTQ for Mamba/SSMs: given a frozen bank of per-bit candidates, which blocks get scarce high bitwidth. It argues diagonal scores drop cross-coordinate residual interactions that don't cancel under deterministic rounding, and instead scores candidates by squared per-token gradient–residual inner products (matrix-free), solving a multiple-choice knapsack under a complete-checkpoint budget. On Mamba-1 130M–2.8B, the block-minus-diagonal WT2 NLL gain shrinks from −0.218 (2.5 bits) to −0.009 (4.5 bits); at 3.0 bits direct KL does better. Effect is larger than matched-scale Pythia and transfers to Mamba-2/Falcon-Mamba.

Critical strengths
  1. A clear, testable claim. The paper says residuals have direction, and diagonal scores miss cross terms that don't cancel under rounding. This is backed by a proof (Prop. 2) and explicit constructions in App. B, not left as intuition.

  2. Gain isolated to pricing. All allocators share the same frozen candidate bank, partition, and byte solver, so any advantage comes from the scoring step alone.

  3. Honest protocol. Results are given as a budget-response curve (5 budgets × 4 sizes × 5 draws) with paired bootstrap CIs, not a single cherry-picked point.

Critical weaknesses
  1. No quantitative comparison to real competitors; closest prior art (FGMP, FAMPWQ) distinguished only in prose. All baselines are author-built reimplementations.
  2. No comparative cost experiment: scoring cost given only as an absolute (115 GPU-hours), never head-to-head against the cheaper diagonal/AWRE.
  3. Runtime overhead is only a throughput-retention ratio vs Quamba2 — no absolute latency, tokens/s, or memory, and no speedup demonstrated.
  4. Gains concentrate at 2.5–3.5 bits and nearly vanish at the common 4.0–4.5 bits.
  5. Beaten by direct KL at 3.0 bits, squeezing its useful window from both ends.
  6. The core contribution feels incremental:The key change is essentially replacing the diagonal score Σgᵢ²eᵢ² with the block form (Σgᵢeᵢ)². Exploiting second-order cross terms is a well-established idea in curvature-aware quantization, so this reads as an incremental refinement rather than a new principle.
Questions for authors
  1. Head-to-head against FGMP/FAMPWQ at matched budgets?
  2. Scoring cost relative to diagonal/AWRE, not just absolute?
  3. Absolute latency/tokens-per-second/memory for mixed- vs uniform-precision?
  4. How much of the diagonal gap is specific to your diagonal implementation?

Thank you for your response. However, given the limited level of innovation and the fact that some of my concerns and questions remain unaddressed, I have decided to maintain my score.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4sGx (1/2): Prior methods and scoring cost

Thank you for the review. The table summarizes our answers, and the sections after it give the details.

Your pointOur answerWhere in the revision
Comparison with FGMP and FAMPWQA FAMPWQ-style allocation adaptation reaches 50.83 six-task accuracy against 51.15 for BlockFAMQ at four bits, with matched candidates and budgets. A native FGMP/FAMPWQ reproduction on the aggressive-bit grid is still missing.Table 21
Scoring costOn Mamba-1 1.4B at 3.5 bits, BlockFAMQ lowers NLL by 0.133 nats relative to diagonal Fisher at about six times the diagonal scoring time. Direct KL and the shortlist take 2.44× and 1.51× the BlockFAMQ scoring time, and the shortlist time includes the full BlockFAMQ pass.Appendix J, Table 29; Table 4
Inference costPacked W4A8 on 1.4B keeps 97.7% of the uniform decode throughput and lowers NLL from 2.503 to 2.468. This is a quality gain at a small throughput cost, not a speedup.Table 12
Contribution beyond "off-diagonal curvature matters"Appendix B shows that diagonal information can fail to rank deterministic residuals. The experiments change only the allocation objective under exact byte budgets, and the advantage over diagonal scoring holds from 512 to 4,096 scored positions.Appendix B; Table 10
Consistency of Table 3Each tensor-class mean was printed 0.07 bits too high and is corrected. The shifts between allocators and the 3.05-bit payload mean are unchanged.Table 3

Prior methods

The submission contains quantitative competitor and absolute-runtime evidence. Table 21 contains a FAMPWQ-style allocation adaptation at four bits. Six-task accuracy, averaged equally across the four Mamba-1 sizes, is 50.83 for that adaptation and 51.15 for BlockFAMQ under the matched candidate and budget protocol. This is a quantitative allocation comparison, but not a reproduction of the full native FAMPWQ pipeline. The native-system comparisons with Quamba2 and SHARP-Q address a different question.

We distinguish two protocols for FGMP/FAMPWQ. An allocation-only comparison holds candidates and packing fixed and labels the baseline as a score adaptation. A native end-to-end comparison retains the competing method's reconstruction and search procedure while matching the model, calibration-data access, evaluated task, and complete-checkpoint ceiling, and charging method-specific work. Activation precision is matched where supported. Native comparisons permit different reconstructions, so their results cannot be attributed solely to the allocator.

Table 21 follows the allocation-only protocol. The adaptation is not a native FGMP/FAMPWQ reproduction across the aggressive-bit grid; that broader head-to-head comparison remains missing.

Scoring cost

The scoring-cost comparison uses the same frozen bank and calibration sample, charging gradient and activation computation to each method. The operating point is Mamba-1 1.4B at 3.5 bits, and the quality metric is absolute WT2 NLL.

Candidate construction costs 272 GPU-minutes per draw. For diagonal Fisher, AWRE, and BlockFAMQ, construction and scoring are incurred once per calibration draw and reused across the five budget targets. Only allocation and serialization repeat for these objectives. The block-to-KL shortlist is rebuilt for each budget, because its anchor is that budget's BlockFAMQ assignment (Section 4.6). At this target, direct-KL and shortlist scoring take 2.44× and 1.51× the BlockFAMQ scoring time, the ratios reported in Table 4.

The shortlist time includes the full BlockFAMQ scoring pass. Of its 1,670 minutes, 1,108 are that pass and the rest is direct-KL rescoring of the shortlist, so its 1.51× is 1.00× for the BlockFAMQ pass plus 0.51× for rescoring. Each budget's shortlist time is therefore a standalone total for that budget. Bank construction is excluded from both ratios and charged once, in the last column below.

Allocation objectiveAbsolute WT2 NLLScoring GPU-minutes per drawConstruction + scoring GPU-minutes per draw
Diagonal Fisher2.535180452
Activation-weighted reconstruction2.46255327
BlockFAMQ2.4021,1081,380
Direct candidate KL≈2.3942,7002,972
Block-to-KL shortlist≈2.3951,6701,942

The two ≈ values are the BlockFAMQ NLL plus the paired differences in Table 4 (−0.008 and −0.007 nats), so each carries a rounding error of up to 0.001. The revision adds this table as Table 29 in the new Appendix J.

The relevant outcome is a cost–quality tradeoff, not dominance on both axes. The paired intervals in Tables 2 and 4 resample calibration draws and source-document clusters, so tokens within one draw are not treated as independent replications (Section 6 and Appendix C.4).

AuthorsOpenReview ↗

Response to Reviewer 4sGx (2/2): Inference measurements, contribution, and byte accounting

Absolute inference measurements are already in Table 12, at a separate packed W4A8 operating point. On Mamba-1 1.4B W4A8, uniform versus block allocation gives NLL 2.503 versus 2.468, peak memory 2.31 versus 2.27 GiB, prefill latency 27.4 versus 27.9 ms, and decode throughput 162.0 versus 158.2 tokens/s. This is a quality improvement with a small throughput reduction, not an inference speedup.

Contribution and byte accounting

The contribution is not the general observation that off-diagonal curvature can matter. For deterministic candidate residuals, Appendix B supplies allocation-reversal and diagonal-indistinguishability constructions; the empirical study then changes only the allocation objective under exact serialized budgets. Together these identify what diagonal information omits and quantify when retaining it changes quality. They do not imply that block curvature is uniformly preferable to direct candidate KL, or that a short scoring formula must have a large effect at every bit rate.

Appendix B's reversal and indistinguishability examples establish that diagonal information can fail to rank deterministic residuals. They do not establish the prevalence of that failure in every trained model. In the implementation, the diagonal score of Equation 6 uses the same residual and gradients as the block score of Equation 5. It drops only the within-unit cross products (Section 4.1). On trained Mamba-1 1.4B at 3.5 bits, Table 10 shows that the advantage over this diagonal score holds as the scoring sample shrinks. Reducing the BlockFAMQ sample from 4,096 to 512 positions changes the block-minus-diagonal NLL difference from −0.133 to −0.127 nats and cuts scoring time from 1,108 to 149 GPU-minutes. The paired interval at 4,096 positions is [−0.155, −0.111].

Table 3's class rows were recomputed from the saved per-model precision assignments. Each of the 14 tensor-class means in Panel A was printed 0.07 bits too high. The revision prints the corrected values:

Tensor classDiagonal FisherBlockFAMQShift
in_proj.weight2.993.04+0.05
out_proj.weight3.143.01−0.13
x_proj.weight3.283.63+0.35
dt_proj.weight3.213.29+0.08
conv1d.weight3.173.08−0.09
A_log3.583.49−0.09
D3.353.25−0.10
Allocatable payload mean3.053.050.00

The correction changes only the displayed class means. Both allocators receive the same 0.07-bit correction in every class, so the shifts between allocators and their interpretation are unchanged. The payload mean stays 3.05 bits for both allocators. Within each model, it weights every allocatable tensor by its parameter count, and the four models are then averaged equally. It is therefore not the plain mean of the seven class rows.

Please let us know whether these changes address your concerns. We will reply to any further point during the discussion period.

Reviewer_jXg8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BlockFAMQ, a mixed-precision allocation method for quantizing Mamba models under a complete-checkpoint byte budget. The central observation is that conventional diagonal Fisher scores discard signed cross-coordinate interactions in deterministic quantization residuals. BlockFAMQ instead scores each candidate using squared token-level gradient–residual inner products, retaining within-block empirical-Fisher interactions without explicitly constructing the curvature matrix. It then selects one precision candidate per block through an exact multiple-choice knapsack solver using serialized byte costs.

Experiments on Mamba-1 models from 130M to 2.8B show that BlockFAMQ consistently outperforms diagonal and activation-weighted allocation, particularly under aggressive 2.5–3.5-bit budgets. The paper also includes transfer experiments on Mamba-2, Falcon-Mamba, Pythia, and Llama, as well as integration with Quamba2.

Critical strengths
  1. The paper identifies a clear limitation of diagonal sensitivity metrics: deterministic residual coordinates need not be independent or zero-mean, so their cross terms may materially change candidate rankings.
  2. The realized-residual score is conceptually well motivated and matrix-free, avoiding explicit construction of large Fisher blocks.
  3. The controlled experimental design is strong. Candidate reconstruction, partitioning, serialization, and byte budgets are held fixed when comparing allocation objectives.
Critical weaknesses
  1. The main experiments are limited to relatively old and small Mamba-1 models. Transformer evaluation is restricted to Pythia-1.4B and Llama-3.2-3B, where the gains are much smaller.
  2. The method is computationally expensive. The reported five-draw calibration for Mamba-1 1.4B requires 115 A100 GPU-hours. Its scalability to modern 7B–70B hybrid models is therefore unclear.
  3. The largest improvements occur at 2.5–3.5 bits, where direct candidate KL also becomes clearly better than BlockFAMQ. This suggests that the local quadratic approximation becomes least reliable in precisely the regime where the method is most useful.
Questions for authors
  1. Can BlockFAMQ scale to recent hybrid architectures such as Qwen3.5/Qwen3.6 with Gated DeltaNet, gated attention, and MoE layers?
  2. Given the 115 A100-GPU-hour calibration cost for a 1.4B model, what is the expected cost for 7B, 27B, or larger models? Can candidate streaming or gradient computation be substantially reduced?
  3. Can the authors better isolate whether the larger gains on Mamba arise from recurrence itself or from differences in block partitioning and parameter structure?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jXg8 (1/3): Overview, calibration cost, and model scale

Thank you for the review. The table summarizes our answers, and the sections after it give the protocol and every number.

Your pointOur answerWhere in the revision
Calibration costOne Mamba-1 1.4B calibration draw costs 23 A100-GPU-hours, and the 115 GPU-hours cover five draws. With 512 scored positions, scoring takes 149 GPU-minutes instead of 1,108, and the block–diagonal difference is −0.127 nats instead of −0.133.Appendix C.5; Table 10
Larger models at aggressive budgetsOn the attention–recurrent Falcon-H1-7B-Base, BlockFAMQ lowers NLL relative to diagonal Fisher by 0.189 and 0.116 nats at 2.5 and 3.0 bits, from one calibration draw. Every serialized checkpoint stays under its byte cap.Section 6; Appendix L, Table 30
Calibration cost on the hybridBlockFAMQ costs 362 GPU-hours for both targets, against 119 for diagonal Fisher and 88 for AWRE. With 1,024 scored positions, its cost falls by 58.3% and NLL rises by 0.066 and 0.045 nats.Appendix L, Tables 31–32
Direct KL and the allocation claimDirect KL beats BlockFAMQ by 0.041 and 0.035 nats at 3.0 bits on 1.4B and 2.8B. A shortlist keeps 0.035 and 0.031 nats at 1.58× and 1.66× the BlockFAMQ scoring time.Table 4
Scaling to 27B/70B hybrid or MoE modelsThese results do not establish it. Part 3 lists the measurements such models would need.This response

Calibration cost

The 115 A100-GPU-hours cover five calibration draws. Appendix C gives 272 GPU-minutes of candidate construction and 1,108 minutes of scoring per draw, totaling 23 GPU-hours. This is still substantial, but it distinguishes repeated experimental calibration from the cost of one allocation. Table 10 already shows a lower-cost operating point on Mamba-1 1.4B at 3.5 bits: 512 scored positions take 149 scoring minutes and yield a block-minus-diagonal NLL difference of −0.127, compared with 1,108 minutes and −0.133 at 4,096 positions. This is a measured reduction in scoring work, not a claim that reduced sampling always preserves allocation quality.

Model scale and aggressive budgets

The submission is not limited to sub-7B models: Table 28 includes Falcon-Mamba 7B, with a block-minus-diagonal NLL difference of −0.027 at 4.02 complete-checkpoint bits. That isolated transfer point does not establish a full low-bit budget sweep or efficient calibration for modern gated-delta, attention–recurrent, or MoE hybrids.

The additional comparison evaluates allocation at 2.5- and 3.0-bit complete-checkpoint targets on a larger attention–recurrent hybrid, Falcon-H1-7B-Base, which is distinct from the pure recurrent Falcon-Mamba 7B in Table 28.

Diagonal Fisher, activation-weighted reconstruction (AWRE), and BlockFAMQ share the candidate tensors, fixed-tensor reconstruction, coordinate partition, scored positions, retained-tensor policy, integer-byte solver, and serializer within each calibration draw. The allocation objective is the method-specific change. Serialized checkpoints are reloaded before evaluation, and quality is measured using WikiText-2 NLL and the same six-task accuracy measure as the submission.

Scoring jobs run on one A100-80GB GPU with a one-context calibration microbatch held identical across objectives, and quality evaluation keeps the paper's FP32 path. Calibration uses the English C4 training split at a pinned revision with a saved document manifest: 128 documents of 2,048 tokens for candidate construction and 128 different documents for scoring, each contributing 32 predetermined positions.

AuthorsOpenReview ↗

Response to Reviewer jXg8 (2/3): Checkpoint feasibility, hybrid quality, and calibration cost

Complete-checkpoint feasibility

The storage policy determines whether an aggressive target is feasible. Falcon-H1-7B-Base has 799,014,912 parameters in its untied embedding and output-head matrices. Retaining both at eight bits while using at least two payload bits elsewhere requires at least 9,588,178,944 unique parameters to reach a 2.5-bit target, before overhead.

Holding those matrices at four bits instead lowers the necessary payload-only threshold to 3,196,059,648 parameters. This is a different retained-precision policy, and all compared allocators use it. Under either policy, complete-checkpoint accounting includes every retained tensor, scale, tag, and padding byte; the payload threshold alone is insufficient to establish serializer feasibility.

Before evaluating a target of BB complete-checkpoint bits, the serializer must establish

M_{\rm cap}(B)=\lfloor BN_{\rm unique}/8\rfloor,\qquad M_{\min}\le M_{\rm cap}(B).$$ The cap includes retained embeddings, the head, normalization, biases, recurrent parameters, scales, padding, and container bytes. For disjoint fractions $f_E$ retained at eight bits and $f_{32}$ at FP32, with two payload bits elsewhere, a necessary floor is $2+6f_E+30f_{32}+b_{\rm metadata}$. The model's [configuration](https://huggingface.co/tiiuae/Falcon-H1-7B-Base/blob/main/config.json) gives $2(130{,}048)(3{,}072)=799{,}014{,}912$ embedding and head weights. Both targets hold the input embedding and the output head at four bits for all three allocators, and biases and normalization parameters stay in FP32. The published checkpoint stores 15,171,297,472 bytes in 751 BF16 tensors, so $N_{\rm unique}=7{,}585{,}648{,}736$. This equals the count implied by the configuration's tensor shapes, the index contains no buffers or aliased tensors, and the embedding and head are untied. The byte caps $\lfloor BN_{\rm unique}/8\rfloor$ are 2,370,515,230 bytes at 2.5 bits and 2,844,618,276 bytes at 3.0 bits, and the integer-byte solver enforces $M_{\rm serialized}\le\lfloor BN_{\rm unique}/8\rfloor$ for every output checkpoint. The four-bit policy leaves about 2.32 bits per remaining parameter at the 2.5-bit target. The largest complete checkpoint across the three allocators is 2,369,691,648 bytes at 2.5 bits and 2,843,742,208 bytes at 3.0 bits, leaving 823,582 and 876,068 bytes below the caps, or 2.499131 and 2.999076 effective bits ($8M/N$). These sizes include every packed weight shard, retained tensor, scale, precision tag, padding byte, serializer header, and index entry. #### Quality at aggressive budgets The quality and calibration-cost comparisons use 4,096 scored positions for all three allocators in one calibration draw. | Target bits | Allocator | WikiText-2 NLL | Mean six-task accuracy (%) | | ---: | --- | ---: | ---: | | 2.5 | Diagonal Fisher | 2.817 | 51.6 | | 2.5 | AWRE | 2.744 | 52.3 | | 2.5 | BlockFAMQ | 2.628 | 53.4 | | 3.0 | Diagonal Fisher | 2.512 | 54.8 | | 3.0 | AWRE | 2.475 | 55.2 | | 3.0 | BlockFAMQ | 2.396 | 56.0 | BlockFAMQ reduces NLL relative to diagonal allocation by 0.189 and 0.116 nats at the two targets, with accuracy gains of 1.8 and 1.2 percentage points. Its NLL improvements over AWRE are 0.116 and 0.079 nats. These results support a quality advantage at the evaluated checkpoint and targets; they do not establish uniform superiority across hybrid or MoE architectures. #### Calibration cost on Falcon-H1-7B-Base Calibration accounting separates bank construction, objective scoring, and packing/reloading of the two budget-specific checkpoints. Bank construction and scoring are reused across budgets. Each row below charges the common bank once for a standalone allocator using both targets; the totals exclude evaluation and CPU-only solver work. | Allocator | Bank construction (GPU-h) | Objective scoring (GPU-h) | Packing/reloading both budgets (GPU-h) | Total (GPU-h) | | --- | ---: | ---: | ---: | ---: | | Diagonal Fisher | 74 | 44 | 1 | 119 | | AWRE | 74 | 13 | 1 | 88 | | BlockFAMQ | 74 | 287 | 1 | 362 | For the combined three-allocator comparison, bank construction is shared and the scoring stages are separate. The full-score matched sweep therefore costs 421 GPU-hours: $74+44+13+287+3$, excluding evaluation, CPU-only work, and the additional sensitivity settings in Part 3. The quality gains come with higher calibration cost. The lower-cost 1.4B operating point from Table 10, described above, establishes the sampling–cost tradeoff at that scale. The larger-hybrid sensitivity results appear in Part 3.
AuthorsOpenReview ↗

Response to Reviewer jXg8 (3/3): Scoring sensitivity, direct KL, and remaining scope

Scoring-size sensitivity

The sensitivity comparison uses one calibration draw and nested position subsets fixed before test evaluation, retaining the same candidate bank, partition, precision policy, byte solver, and serializer. Diagonal Fisher and AWRE remain 4,096-position references. Comparisons against them at smaller BlockFAMQ sample sizes therefore measure the quality–cost tradeoff rather than isolate the scoring objective at a matched sample size.

Standalone totals include 74 GPU-hours for bank construction and 1 GPU-hour for packing/reloading both budgets.

BlockFAMQ scored positionsScoring (GPU-h)Standalone total (GPU-h)NLL at target 2.5 bitsNLL at target 3.0 bits
512391142.7712.489
1,024761512.6942.441
2,0481482232.6482.408
4,0962873622.6282.396

Using 1,024 positions reduces accounted calibration cost by 58.3% relative to full-score BlockFAMQ, with NLL penalties of 0.066 and 0.045 at the two targets. The cheapest setting is not uniformly preferable: at 512 positions, NLL is worse than the full-sample AWRE reference by 0.027 and 0.014, and calibration cost is higher. The six-task accuracy results in Part 2 apply to the 4,096-position setting.

On the hybrid, 512 positions keep 0.046 of the 0.189-nat gain over diagonal allocation at 2.5 bits. This differs from Mamba-1 1.4B at 3.5 bits, where 512 scored tokens already give −0.127 of the −0.133 block-minus-diagonal difference (Table 10). The two studies also differ in bit target.

Standalone rows cannot be summed to obtain the physical cost of a shared-bank sensitivity sweep. Reused construction, gradients, and nested score computations are charged once. Appendix C.5's 23 A100-GPU-hours per Mamba-1 1.4B draw covers construction and scoring; its scope differs from the larger-model totals, which also include packing and reloading.

The revision adds this study as Appendix L, Tables 30–32.

Direct candidate KL and the allocation claim

The allocation benefit depends on the budget and comparator. In submitted Table 4, direct candidate KL improves on BlockFAMQ by 0.041 and 0.035 nats at 3.0 bits on Mamba-1 1.4B and 2.8B, respectively. The feasibility-anchored block-to-KL shortlist recovers 0.035 and 0.031 nats at 1.58× and 1.66× block-scoring cost, compared with 2.51× and 3.05× for exhaustive candidate KL. These results support a selective-refinement route when the local approximation is insufficient.

The direct-KL and shortlist measurements apply to those Mamba-1 models and budgets. The larger-hybrid comparisons in Parts 2 and 3 establish quality–calibration-cost tradeoffs against diagonal and AWRE allocation at the evaluated targets. They do not rank direct KL on the hybrid or establish robustness across calibration draws, inference speedup, or transfer to untested model families.

Remaining scope

Mamba-2, Falcon-Mamba, and Falcon-H1 results provide transfer evidence at the tested operating points. They do not establish cost or quality on gated-delta or MoE hybrids. Defining a residual score for those parameter blocks is straightforward, but expert coverage, candidate construction and streaming, gradient and activation cost, retained-tensor storage, and serialization require architecture-specific measurements. We do not infer their GPU-hours by linearly scaling the 1.4B or 7B results.

The Pythia comparison identifies an architectural contrast, not recurrence as a causal mechanism. A recurrence-specific test must control parameter counts per block, residual norms, candidate levels, serialization, and byte budgets while varying the recurrent mechanism. Matching block widths alone would not match residual geometry. Our supported conclusion is that the block effect is larger in the tested Mamba configuration.

The results support a budget-dependent allocation correction with a measured calibration-quality tradeoff. Table 4 and the extended direct-KL grid in our reply to Reviewer roAG show where the local approximation is useful. Neither establishes scaling to 27B/70B hybrid or MoE models.

Please let us know whether these results answer your questions. We will reply to any further point during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for questions that sharpened the paper's claim. We have uploaded a revised manuscript and posted detailed responses under each review. The revision adds measurements and a bound clarifying the scope of the method.

  • Aggressive budgets on a larger hybrid. On the attention–recurrent Falcon-H1-7B-Base, BlockFAMQ lowers WikiText-2 NLL relative to diagonal Fisher by 0.189 and 0.116 nats at 2.5 and 3.0 complete-checkpoint bits, and relative to AWRE by 0.116 and 0.079 nats. These results come from one calibration draw, and every serialized checkpoint stays under its byte cap (Appendix L).
  • Where direct KL helps. Direct candidate KL beats BlockFAMQ by 0.041 and 0.035 nats at 3.0 bits on Mamba-1 1.4B and 2.8B, and its paired intervals include zero at 3.5 and 4.0 bits. A block-to-KL shortlist keeps 0.035 and 0.031 nats of that gain at 1.58× and 1.66× the BlockFAMQ scoring time (Table 4).
  • Cost against cheaper objectives. On Mamba-1 1.4B at 3.5 bits, BlockFAMQ lowers NLL by 0.133 nats relative to diagonal Fisher and by 0.060 nats relative to AWRE, at about six times the diagonal scoring time (Appendix J, Table 29).
  • The NLL scope of the block score. Proposition 10 (Appendix K) splits the gap between the NLL change and the block score into a first-order term, a curvature mismatch, and a finite-step remainder. It identifies these error sources and is not a numerical NLL guarantee.

Revisions at a glance

ChangeRequested byWhere in the revision
Falcon-H1-7B-Base at 2.5 and 3.0 bits, with byte caps, calibration costs, and a scored-position studyjXg8, roAGSection 6; Appendix L, Tables 30–32
Scoring cost against quality for five allocation objectives4sGxAppendix J, Table 29
NLL scope of the block scoreroAGAppendix K, Proposition 10
Direct KL on four Mamba-1 sizes at five budgetsroAGReply to roAG, Part 1: full grid; manuscript Table 4: six model–budget settings
Calibration-draw count and shortlist cost scope4sGx, roAGTable 4 caption; Appendices C.5 and J
Corrected tensor-class rows of Table 34sGxTable 3

What the paper claims

BlockFAMQ isolates the allocation step: candidate reconstructions, block partition, serializer, and byte optimizer remain fixed while the scoring objective changes. Retaining signed interactions in the realized rounding residual can change which blocks receive extra precision. The contribution is this controlled, budget-dependent allocation correction, not a new general principle of second-order quantization. The diminishing gain at higher bit rates and the advantage of direct KL at aggressive budgets are part of the result.

Detailed responses appear under the reviews of jXg8 (three parts), 4sGx (two parts), and roAG (two parts).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.