acceptodds
Under review as a conference paper at ICLR 2027

ReAlignQ: Post-Compression Refinement for Joint Sparsification and Quantization of LLMs

Abstract

Quantization and sparsification compress large language models by restricting numerical precision and nonzero support, respectively. Support selected in continuous weight space may be poorly matched to the values that its retained coordinates can realize on the quantization grid. Moreover, criteria applied to individual coordinates overlook interactions among simultaneous changes in support membership and quantized values. We therefore propose ReAlignQ, a refinement framework that jointly selects support exchanges and legal discrete values for incoming weights under fixed quantization scales and a prescribed sparsity budget. For each linear weight matrix, ReAlignQ models interactions among screened candidates using an empirical gradient Gram surrogate built from gradients of individual calibration samples. It retains the resulting gradient signatures only for screened candidates, which reduces the storage needed to model their interactions. We further implement sparse and quantized operators with exact bitmap masks. These operators execute refined nonuniform masks without changing them. We also provide theoretical results that characterize support selection over legal codes, bound the empirical gradient Gram interactions omitted by diagonal screening, and establish surrogate descent while preserving sparsity for accepted exchanges. Across Llama and Qwen models and multiple sparsity levels, ReAlignQ consistently improves language-modeling quality without retraining, changing parent scales, or relaxing sparsity budgets. On Llama 3 8B with an EIWR W8A16 parent at 70% sparsity, it reduces WikiText-2 perplexity from 35.48 to 25.57 and improves mean zero-shot accuracy by 2.6 percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Joint Sparsification and Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_cn99OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ReAlignQ, a post-compression refinement method for jointly sparse and quantized LLMs. It performs support-preserving weight exchanges while selecting legal quantized codes for newly activated weights, using gradient-based screening and an empirical gradient-Gram surrogate to model candidate interactions. Experiments show improvements without changing the parent quantization scales or sparsity budgets.

Critical strengths
  • ReAlignQ addresses a practical mismatch between support selection and quantized code assignment under fixed compression constraints.
  • I appreciate the paper’s theoretical analysis and its implementation of operators tailored to the refined sparse-quantized models. Although the current evaluation does not demonstrate substantial practical acceleration, these contributions remain valuable from a research perspective.
  • Empirical results show improvements
Critical weaknesses
  • I think it's necessary to compare with dense models' performance. At 70% sparsity, the compressed models already suffer substantial performance degradation, which may limit their practical usability. I therefore encourage the authors to provide examples of actual model responses.
  • The practical value of this post-compression setting should be better motivated. Moreover, the refinement stage introduces additional computational cost, which is not sufficiently characterized in the experiments.
Questions for authors

See above

4 comments

Thank you for your positive assessment of our problem formulation, theoretical analysis, and bitmap operators. Below we address both concerns with same-evaluator Dense FP16 comparisons across 50%–70% sparsity, actual model continuations, bit-budget accuracy comparisons, and a full breakdown of refinement cost vs. matched-time fine-tuning.

W1. Comparison with the dense model across sparsity levels, and actual model responses. Under the same 6-task lm-eval 0-shot protocol, 50% W8A16 + ReAlignQ retains 93.7% of Dense FP16 accuracy (69.5% vs. 74.2%) at 5.0 linear bits/entry. At 70% sparsity (3.4 linear bits/entry; 3.27 vs. 14.96 GiB serialized footprint on Llama 3 8B in Table 5, and 66.2% vs. 74.4% Dense on Qwen3-14B), ReAlignQ lifts 0-shot closed-book TriviaQA EM on Llama 3 8B from 28.6% to 41.8% (+13.2 points) and improves two illustrative continuations below.

  1. Full spectrum vs. Dense FP16: Dense FP16 scores under the exact Table 1–2 harness:
    • Llama 3 8B: 6.32 PPL / 74.2% Avg. (PIQA 80.9, BoolQ 81.0, Hella. 79.2, ARC-e 77.7, ARC-c 53.2, Wino. 73.4).
    • Qwen3 8B: 9.71 PPL / 71.1% Avg.; Qwen3 14B: 8.64 PPL / 74.4% Avg.
Model & SettingLinear Base Bits/Entry*Parent → +ReAlignQ PPL↓Zero-Shot Avg.↑ (Parent → +ReAlignQ)
Llama 3 8B — Dense FP1616.0 bpw (14.96 GiB, Table 5)6.3274.2%
Llama 3 8B — 50% W8A16 (SparseGPT+GPTQ)5.0 bpw (0.5×8+10.5\times 8 + 1)9.94 → 9.6967.4% → 69.5% (93.7% of Dense)
Llama 3 8B — 60% W8A16 (SparseGPT+GPTQ)4.2 bpw (0.4×8+10.4\times 8 + 1)15.89 → 13.5661.8% → 63.9% (+2.1)
Llama 3 8B — 50% W4A16 (SparseGPT+GPTQ)3.0 bpw (0.5×4+10.5\times 4 + 1)12.33 → 11.6965.3% → 65.3%
Llama 3 8B — 70% W8A16 (SparseGPT+GPTQ / EIWR)3.4 bpw (3.27 GiB, Table 5)53.09→32.93 / 35.48→25.5748.2%→51.6% / 50.8%→53.4%
Qwen3 8B — Dense FP16 vs. 70% EIWR W8A1616.0 → 3.4 bpw9.71 vs. 24.81 → 20.8171.1% vs. 57.4% → 59.1%
Qwen3 14B — Dense FP16 vs. 70% EIWR W8A1616.0 → 3.4 bpw8.64 vs. 19.43 → 16.2474.4% vs. 66.0% → 66.2% (89.0% of Dense)

Note: Linear payload + bitmap bits/entry (ρb+1\rho b + 1 at density ρ\rho and bb-bit codes; 16.0 bpw for FP16), before scale/offset metadata; 14.96 GiB and 3.27 GiB are whole-model serialized footprints from Table 5.

50%–60% sparsity (and 70% on 14B, keeping 89% of Dense accuracy) serves memory-constrained deployment, while 70% on 8B stress-tests severe compression (retaining a clear gap to Dense, yet recovering up to 20.16 PPL over the parent).

  1. 0-shot closed-book QA & actual model continuations (70% sparsity): On 1,000 TriviaQA validation questions (standard 0-shot lm-eval closed-book protocol, greedy up to first newline, official alias-normalized EM), accuracy rises from 28.6% (Parent) to 41.8% (+ReAlignQ, +13.2 points), vs. 57.4% (Dense).

Greedy continuations (first ~30 words shown) on Llama 3 8B (70% EIWR W8A16) illustrate this shift:

  • Prompt 1: "A vaccine does not treat an active infection; it"
    • Dense FP16: "prevents you from getting the disease. Vaccines are made from a dead or weakened form of the virus or bacteria that causes the disease. The vaccine stimulates your"
    • 70% Parent (35.48 PPL): "only treats a disease. A vaccine does not treat an active infection; it only treats a disease. A vaccine does not treat an active infection; it only treats"
    • 70% + ReAlignQ (25.57 PPL): "only prevents future infections. The vaccine is not a cure, but a preventive measure that prepares the immune system to recognize and fight off the pathogen before"
  • Prompt 2: "Antibiotics kill bacteria by interfering with"
    • Dense FP16: "their ability to make proteins. The antibiotics that are most effective against bacteria are those that target the ribosome, the cellular machine that makes proteins. The ribosome is"
    • 70% Parent (35.48 PPL): "the host's immune system. The host's immune system is the host's immune system. The host's immune system is the host's immune system. The host's immune system is the host's immune"
    • 70% + ReAlignQ (25.57 PPL): "their growth and ability to reproduce, helping the body clear bacterial infections such as strep throat, pneumonia, and ear infections by disrupting essential cell wall or"

In these two examples, the 70% parent contradicts the premise ("only treats a disease") or repeats "the host's immune system", whereas the refined continuations follow the prompt premise ("only prevents future infections", bacterial growth/infection control).

W2. Practical value of the post-compression setting, and full characterization of refinement cost. ReAlignQ improves quality across the four tested compression pipelines without altering parent scales or adding deployment parameters. At 3.0 linear bits/entry, 50% Sparse W4A16 + ReAlignQ outperforms unpruned W3A16 by 1.73 PPL, and its one-time 3.97 GPU-hour offline solve beats matched-time fine-tuning across the tested starting states.

  1. Why post-compression refinement matters in practice:
    • Broad compatibility: ReAlignQ refines JSQ, OBR-GPTQ, EIWR, and SparseGPT+GPTQ checkpoints while keeping parent scales, sparsity budget, and prefill throughput (8,638.6 vs. 8,642.4 tok/s parent, 8,406.0 Dense, in 3.27 GiB, Table 5).
    • Unpruned quantization-only models: Via natural zeros in per-row W4A16/W3A16, base 1-in/1-out zero-swaps lower PPL (10.62 → 10.18, 15.40 → 14.12); adding ±1\pm 1 neighbor-recoding reaches 9.89 and 13.42 on standard dense kernels.
    • Matched 3.0-bpw linear bit-budget: In uniform per-row W4/W3/W2 baselines, going below 4 bpw assigns W3/W2 to every weight, whereas joint compression prunes redundant entries (1 bitmap bit/entry) so retained weights keep higher precision. On Llama 3 8B, 50% Sparse W4A16 + ReAlignQ and Quant-only W3A16 both use 3.0 linear base bits/entry (0.5×4+1=3.00.5\times 4 + 1 = 3.0 vs. 3.03.0), yet joint + ReAlignQ beats even recode-refined W3A16 by 1.73 PPL and +3.2 zero-shot points (W2A16 at 2.0 bpw collapses >100 PPL):
Method (Llama 3 8B, per-row GPTQ pipeline)Linear Base Bits/EntryParent → Refined PPL↓Zero-Shot Avg.↑
Quantization-only W4A16 (base swap: 10.18; +recode: 9.89)4.0 bpw10.62 → 9.8966.5% → 68.1%
50% Sparse W8A16 + ReAlignQ (Table 2)5.0 bpw (0.5×8+10.5\times 8 + 1)9.94 → 9.6967.4% → 69.5%
Quantization-only W3A16 (base swap: 14.12; +recode: 13.42)3.0 bpw15.40 → 13.4259.8% → 62.1%
50% Sparse W4A16 + ReAlignQ (Table 2)3.0 bpw (0.5×4+10.5\times 4 + 1)12.33 → 11.6965.3% → 65.3%
  1. Stage-wise breakdown of refinement cost & comparison with same-budget fine-tuning: On Llama 3 8B (EIWR W8A16, 70%, 256 C4 sequences), the 3.97 GPU-hour offline solver cost (peak 37,152 MiB; 1.03 GiB signatures discarded after calibration) breaks down into:

    • Pass 1 (full-model gradient statistics g,Fg, F + screening): 3.28 GPU-h (82.6%);
    • Pass 2 (retaining candidate signatures UU on ∣C∣=1.08M|\mathcal{C}|=1.08\text{M} entries): 0.58 GPU-h (14.6%);
    • In-memory coupled solver + materialization: 0.11 GPU-h (0.04h Stage 1 + 0.07h Stage 2 & materialization, 2.8%; vs. 3.36h and 30.42 PPL for exact Prop. 1 diagonal without Pass 2). Adding 5-ratio validation selection takes 0.18 GPU-h (4.15h end-to-end).

    Under the identical 256 C4 sequences, disjoint selection set, and ≤4.15 GPU-hour\le 4.15\text{ GPU-hour} end-to-end budget, ReAlignQ outperforms tuned STE/QAT, continuous FT + projection, and merged LoRA across tested starting states (with an expanded swap cap in the fixed-n0n_0 row):

Starting State / Final Nonzeros (Llama 3 8B, 70%)Tuned STE / QATContinuous FT + Proj.Merged LoRAReAlignQ (ours)
Parent (35.48), fixed n0n_0 (≤4.15h\le 4.15\text{h})31.8031.1229.5027.86 (cap128, 4.13h)*
Parent (35.48), target KK (≤4.15h\le 4.15\text{h})29.7630.0228.9425.57 (4.15h end-to-end)
Shared Stage-1 (27.46), KK (≤4.15h\le 4.15\text{h} total)26.4826.7326.1925.57 (4.15h end-to-end)
Double budget: Stage-1 + 4.15h FT (8.05h)26.3526.5126.0825.57 (at 4.15h)

Note: Default cap 8 (4.09h incl. selection) gives 31.68 at fixed n0n_0; batched GPU frontier search at cap 128 adds 0.04h (4.13h ≤4.15h\le 4.15\text{h}) and reaches 27.86. Unmerged LoRA from Stage 1 gives 25.94 PPL (+0.08 GiB extra dense weights). Five disjoint C4 seeds further confirm stability (25.64 ± 0.14 PPL).

We will incorporate these Dense comparisons, TriviaQA/continuation evaluations, footprint analyses, and fine-tuning baselines into the revision.

I really appreciate authors' effort during rebuttal. My concerns have been adequately addressed. I will update my score.

Thank you for your careful review, constructive feedback, and for raising your score. We are glad that our clarifications and additional experiments have addressed your concerns. We will incorporate these results and presentation improvements into the revised manuscript.

Reviewer_MXpwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to recover model quality after joint sparsification and quantization. The key idea is that a coordinate chosen in continuous weight space may become less useful once its value is put on a quantization grid. Also, looking at coordinates one by one can miss interactions between support changes. To solve these problems, ReAlignQ starts from a compressed parent model. It screens activation and release candidates using diagonal gradient statistics. It also keeps per-sample gradient signatures to evaluate interactions inside each linear weight matrix. After building a feasible state, the method jointly chooses an outgoing coordinate, an incoming coordinate, and a legal code for the incoming weight. It does this while keeping the support size and parent scales fixed.

Experiments across Llama and Qwen models show improvements over several parent compression methods. For example, on Llama 3 8B with an EIWR W8A16 parent at 70% sparsity, ReAlignQ reduces WikiText-2 perplexity from 35.48 to 25.57 and improves mean zero shot accuracy by 2.6 points. The paper also evaluates a sparse inference operator to examine whether the recovered quality can be preserved without increasing deployment storage.

Critical strengths
  1. The refinement problem is well motivated. The usefulness of a nonzero coordinate depends on the quantized values it can take. Therefore, by choosing support and incoming codes together, ReAlignQ improves an existing compression method instead of replacing it.

  2. The paired improvements across JSQ, OBR-GPTQ, EIWR, and SparseGPT followed by GPTQ are important. Each refined model is compared with its own compressed parent. The component ablations also support the design. In addition, the disjoint C4 holdout gives extra evidence that the method transfers beyond the refinement samples in the EIWR setting.

  3. The authors clearly separate descent of the local surrogate from improvement in the true loss. This keeps the theoretical claims properly limited. Moreover, reporting offline refinement cost together with perplexity, storage, and prefill throughput makes the practical contribution easier to judge.

Critical weaknesses
  1. My main concern is that the paper does not clearly separate feasible state construction from exchanges that keep the nonzero count fixed. At 70% sparsity, the fill to target only variant already lowers the perplexity of the EIWR parent from 35.48 to 27.46. Full ReAlignQ reaches 25.57. This is a huge chunk of the recovery. But Section 4.4 just says construction brings the nonzero count to its target. So I can't tell if the gain comes from restoring missing nonzeros or from the support changes in the exchanges. This attribution problem makes it hard to judge the core contribution.

  2. This also affects the diagonal comparison. Proposition 1 gives an exact solution to the separable problem on the candidate set. But Table 4 does not say whether the diagonal surrogate baseline uses that solution. The full cost is 3.97 GPU hours, but the costs of the diagonal and coupled variants are not separated. So we don't know how much extra benefit comes from keeping gradient signatures, or what it costs. This is a critical missing piece.

  3. The pair score experiment gives some support, but the numbers are low. Spearman correlation is 0.36 and sign accuracy is 66%. These are measured on 128 pairs applied to the original reference state. Actual refinement accumulates edits using gradients from that same state. So this experiment does not show if the coupled surrogate stays more informative than the diagonal one as refinement goes on.

  4. The calibration sample size experiment and the disjoint C4 holdout are useful. But the selection procedure is still unclear. Table 7 lists five damping ratios. Table 9 introduces a calibration gate without defining its criterion. It says the ratio is fixed before evaluation, but not how it was chosen. So the smaller improvements are hard to assess without independent calibration seeds.

  5. The deployment evaluation needs to show what happens to the refined state during conversion. Appendix B.1 notes that preserving bitmap bits does not necessarily preserve physical nonzeros or quantized values. The close packed and HuggingFace perplexities show much of the quality gain survives. But they do not prove numerical equivalence. So it is unclear whether these experiments run the refined codes under the original scales or include a separate requantization step.

Questions for authors
  1. I was confused by the gap between the fill to target only variant and the full ReAlignQ result. What exactly does feasible state construction do here? Does it just restore the target nonzero count, or does it also rebuild the support? It would be great if the authors could describe the three fixed point updates in Table 7 and report the nonzero counts at each stage. This would really help compare the benefit of paired exchanges from the same starting point.

  2. For the diagonal comparison in Table 4, does the surrogate baseline use the exact solution from Proposition 1 or a capped search? I think it would be very informative to compare the exact diagonal solution and ReAlignQ under the same candidate set, admissible codes, damping, and support target. Also, reporting their refinement costs would show whether the interaction terms are worth the extra effort of keeping gradient signatures.

  3. I wonder how well the surrogate predictions actually match measured loss changes along the real sequence of accepted exchanges. How about after edits from all matrices are combined? Comparing diagonal and coupled predictions on the same transitions would test whether the interaction terms stay useful beyond isolated pairs at the reference state.

  4. Could the authors specify the data and criterion used to pick the damping ratio? Also, what exactly does the calibration gate accept or reject? I also couldn't quite tell how failed refinements are handled. Providing paired results with independent calibration seeds would make the reported improvements much easier to interpret.

  5. I couldn't tell if the packed models in Tables 5 and 11 actually keep the refined codes and original parent scales, or if conversion involves requantization. Reporting differences in numerical weight values and physical nonzero counts separately from bitmap preservation would clear this up. Then we would know if deployment still represents the support and code state optimized by ReAlignQ.

5 comments

Thank you for the careful reading. We apologize for the brief description of Stage 1 in Section 4.4 and the narrow label "fill-to-target only" in Table 4, which obscured its coupled support selection. We answer all five questions below.

W1 / Q1. Stage 1 ("fill-to-target only", Eqs. 10–11) vs. Stage 2 paired exchanges (Eqs. 12–14), and the three fixed-point updates. Both stages optimize JλJ _ \lambda: Stage 1 selects which candidate zeros enter and which legal codes they take at fixed parent nonzeros (35.48 → 27.46 vs. 33.18 for naive restore), while Stage 2 performs survivor-to-zero exchanges (35.48 → 27.86 alone at cap 128; 31.68 at cap 8). Combining both reaches 25.57 (25.38 at cap 128).

Three fixed-point rounds in Table 7 (EIWR W8A16):

  • Update rule: While Section 4.4 (Lines 299–300) described updating rr after each transition, our solver runs Table 7's three batched rounds, refreshing (Δ,r)(\Delta, r) once per round; we will revise Section 4.4 to match.
  • Parent deficit: W8 rounding (∣wi0∣<si/2→0|w_i^0| < s_i/2 \to 0) leaves deficit d=K−n0=24,640d = K - n_0 = 24{,}640 at 70% (K=2,093,796,576K = 2{,}093{,}796{,}576) and 18,42018{,}420 at 60% (K≈2.792BK \approx 2.792\text{B}), inside ∣C∣=1,084,240|\mathcal{C}| = 1{,}084{,}240.
  • Round 1 (n0→Kn_0 \to K, r(0)=0r^{(0)}=0): Scores mi(siq∣0)m_i(s_i q\mid 0) across screened zeros (relocating 89.4% of dd off collapsed slots) and updates r(1)=UΔ(1)r^{(1)} = U\Delta^{(1)}, matching restricted diagonal fill (30.98 PPL).
  • Rounds 2 & 3 (K→KK \to K): Score leave-one-out marginals mi(siq∣Δ−i(t−1))m_i(s_i q\mid\Delta _ {-i}^{(t-1)}) with r−i(t−1)=r(t−1)−UiΔi(t−1)r _ {-i}^{(t-1)} = r^{(t-1)} - U_i\Delta_i^{(t-1)}, replacing 21.3% (R2) and 5.8% (R3) of activations/codes before refreshing r(t)r^{(t)} (30.98 → 28.19 → 27.46). Stage 1 cannot release bad parent survivors.
Variant (Llama 3 8B, EIWR W8A16)Nonzeros (70%)60% PPL↓70% PPL↓
Compressed parent W0W^0n0=K−24,640n_0 = K - 24{,}64016.7235.48
Naive collapsed-slot restore (±1\pm 1)KK15.7833.18
Stage-1 R1 → R2 → R3 (fill-to-target only)K→K→KK \to K \to K14.63→14.41→14.3230.98→28.19→27.46
Stage-2 Exchange-Only (no fill, cap 8 / 128)n0n_015.92 / 14.4031.68 / 27.86
Shared Stage-1 R3 + diagonal Stage-2 (cap 8)KK14.0826.92
Reversed: Stage-2 (cap 128) → Stage-1 filln0→Kn_0 \to K14.40 → 13.4627.86 → 25.63
Full ReAlignQ: Stage 1 + Stage 2 (cap 8 / 128)KK13.41 / 13.3425.57 / 25.38

Two takeaways:

  • Stage 1 is real support/code selection (~110 activations/matrix), beating naive restore by 5.72 PPL (27.46 vs. 33.18).
  • Standalone Stage 2 at fixed n0n_0 reaches 27.86 once cap 128 allows ~89 swaps/matrix (19,840 total vs. ≤1,792\le 1{,}792 at cap 8; cf. Table 9 W4: 86.86 → 71.69 at add=del=19). Their gains overlap (68.4% of Stage 2's 19,840 holes are also picked by Stage 1; reverse order gives 35.48 → 27.86 → 25.63), and combining both reaches 25.57 / 25.38.

W2 / Q2. Exact Proposition 1 baseline vs. Table 4 diagonal baseline, and compute cost. Table 4's diagonal surrogate (30.56) ran our two-stage pipeline with Ui⊤UjU_i^\top U_j (i≠ji\neq j) zeroed out. Under the exact same C\mathcal{C}, codes, g,F,λ,Kg, F, \lambda, K:

  • Exact Proposition 1 (minimizing JdiagJ _ {\mathrm{diag}} over Fl(Cl)\mathcal{F} _ l(\mathcal{C} _ l) via the KClK _ {\mathcal{C} _ l} smallest δi\delta_i, allowing both releases and activations) achieves 30.42 PPL in 3.36h.
  • Full ReAlignQ adds 0.61h and lowers 70% PPL by 4.85 points to 25.57:
Solver Route (EIWR W8A16)60% PPL↓70% PPL↓Pass 1 (g,Fg,F)Pass 2 (UU)Solve+Mat.Total GPU-h
Restricted diag. fill (= Stage-1 R1)14.6330.983.28h0.00h0.06h3.34h
Table 4 diag. pipeline (R1 + cap-8 swaps)14.4530.563.28h0.00h0.07h3.35h
Exact Prop. 1 minimizer of JdiagJ _ {\mathrm{diag}} on C\mathcal{C}14.3930.423.28h0.00h0.08h3.36h
Full ReAlignQ (coupled UU, 1.03 GiB1.03\text{ GiB})13.4125.573.28h0.58h0.11h3.97h

Pass 1 (3.28h) dominates both routes; collecting UU (0.58h) and coupled solving (0.11h vs. 0.08h) add 18% time (3.36h → 3.97h) and yield the 30.42 → 25.57 (-4.85) and 14.39 → 13.41 (-0.98) improvements.

W3 / Q3. Predictions along the actual accepted exchange sequence and cross-matrix composition. Submitted Table 10 reports 0.36 Spearman and 66% sign accuracy on 128 independent pairs at W0W^0. In a new trajectory audit on EIWR 70% (64 held-out C4 sequences disjoint from the 256 fit sequences), coupled ΔJ0\Delta J_0 (omitting λ∥Δ∥22\lambda\|\Delta\| _ 2^2, Appendix A.4 Lines 1104–1108) remains informative along refinement:

  • Accepted Stage-2 swaps (t=1,…,Tlt=1,\dots,T_l, R3 to final state): Pooling all accepted swaps across the 224 matrices gives 89% vs. 62% single-step true-CE sign accuracy; per-matrix cumulative relative error ∣ΔLCE,l−∑t=1TlΔJ0,l,t∣/∣ΔLCE,l∣\lvert\Delta\mathcal{L} _ {\mathrm{CE},l} - \sum _ {t=1}^{T_l}\Delta J _ {0,l,t}\rvert / \lvert\Delta\mathcal{L} _ {\mathrm{CE},l}\rvert averages 12% vs. 39%.
  • Intermediate-state pairs & full composition: Across 384 pairs (128 each after R1, R3, 50% swaps) at shared Δ\Delta, Spearman stays 0.61 vs. 0.19; composing all 224 matrices lowers held-out C4 PPL across blocks to 23.26 vs. 26.49.
Trajectory & Composition Check (Llama 3 8B, 70%)Coupled ΔJ0\Delta J_0Diagonal ΔJ0\Delta J_0
Actual accepted swaps: per-step true-CE sign accuracy89%62%
Actual accepted swaps: R3→final cumulative rel. error12%39% (over-predicts gain)
Candidate-pair Spearman at shared Δ\Delta (early→mid→late; all)0.67→0.61→0.54 (0.61)0.31→0.18→0.06 (0.19)
Composed C4 holdout PPL (every 8 blocks, 0→32)30.58→28.91→27.14→25.38→23.2630.58 → 26.49 (final)

Diagonal ΔJ0\Delta J_0 becomes increasingly over-optimistic as co-activated weights accumulate (late Spearman 0.06), whereas updating r=UΔr = U\Delta maintains accuracy through composition (holdout 23.26 vs. 26.49).

W4 / Q4. Damping criterion, calibration gate, failure handling, and 5 independent seeds. Damping is selected on disjoint calibration CE (never test data), and the gate reverts any failed run to an exact parent no-op:

  • Fixed W8 ratio (Tables 1–2): r=3×106r = 3\times 10^6 (λl=r⋅median⁡i∈Cl∣gisi∣\lambda_l = r\cdot\operatorname{median} _ {i\in\mathcal{C} _ l}|g_i s_i|) was chosen once on a disjoint Llama 3 8B C4 slice over R={1,2,3,5,10}×106\mathcal{R} = \{1,2,3,5,10\}\times 10^6 and frozen (3.97h single-run cost).
  • Gated selection (Table 9 & 5-seed runs): All five r∈Rr \in \mathcal{R} reuse cached (g,F,U)(g,F,U). On the Llama 3 8B EIWR 70% profile this adds +0.18h (4 extra solves 0.10h + disjoint DselD _ {\mathrm{sel}} scoring 0.08h, 4.15h total), accepting r⋆=arg⁡min⁡rLsel(W(r))r^\star = \arg\min_r\mathcal{L} _ {\mathrm{sel}}(W(r)) iff Lsel(W(r⋆))≤Lsel(W0)\mathcal{L} _ {\mathrm{sel}}(W(r^\star)) \le \mathcal{L} _ {\mathrm{sel}}(W^0) (reverting to W0W^0 otherwise).
  • Five disjoint C4 seeds: Across both large- and small-gain cells, every run passes the gate with tight SDs (0.03–0.14 PPL):
Model / Parent / Setting (5 disjoint C4 seeds, gated)Parent PPLReAlignQ PPL (mean ± SD)Paired Δ\DeltaPPLGate
Llama 3 8B / EIWR W8A16 / 70%35.4825.64 ± 0.14 (diag: 30.62 ± 0.22)-9.84 ± 0.145/5
Qwen3 8B / SparseGPT+GPTQ W8A16 / 70%24.8523.39 ± 0.09 (Table 1: 23.36)-1.46 ± 0.095/5
Qwen3 14B / SparseGPT+GPTQ W8A16 / 70%18.1317.27 ± 0.05 (Table 1: 17.25)-0.86 ± 0.055/5
Llama 3 8B / SparseGPT+GPTQ W8A16 / 50%9.949.68 ± 0.03-0.26 ± 0.035/5

Every run passes the gate, with across-seed SDs of 0.03–0.09 PPL on the smaller-gain cells.

W5 / Q5. Numerical equivalence of packed models in Tables 5 and 11. All paths preserve exact bitmaps and physical nonzero counts (0 / 0). Auditing scale handling:

  • Submitted packers: Table 5 (W8A16) used 8×88\times 8 tile scales (Appendix B.3; 8.5e-48.5\text{e-}4 tile rescaling, PPL 25.67), while Table 11 (W8A8) recomputed max⁡k∣Wm,k∣/127\max_k|W _ {m,k}|/127 from dequantized FP16 weights (Appendix B.4 Lines 1309–1311), giving small FP16 scale-rounding shifts (1.1e-41.1\text{e-}4 max diff, PPL 34.12).
  • Direct parent-scale packing: Passing (ql,i,sparent)(q _ {l,i}, s _ {\mathrm{parent}}) without scale recomputation gives 0.0 weight diff in both W8A16 (25.59) and W8A8 (34.11) at 3.27 GiB:
Audit across All 224 Matrices (Llama 3 8B, 70%)Subm. Tile-W8A16Direct Parent-W8A16Subm. Packer W8A8Direct Parent-W8A8
Bitmap / Physical nonzero diff. vs. HF0 / 00 / 00 / 00 / 0
Max weight diff. max⁡∣Wdeq−WHF∣\max\lvert W _ {\mathrm{deq}} - W _ {\mathrm{HF}}\rvert8.5e-48.5\text{e-}40.01.1e-41.1\text{e-}40.0
WikiText-2 PPL (HF: 25.57 W8A16; 34.30 W8A8)25.6725.5934.1234.11

Under direct (q,sparent)(q, s _ {\mathrm{parent}}) packing (0.0 weight diff), an equal-weight FP16 oracle confirms the 0.02 W8A16 gap (25.57 vs. 25.59) comes from Split-K FP16 MMA reduction order.

I thank the authors for the detailed response and the additional experiments. Several of my concerns have been addressed. But my main concern remains. I think there is a difference between the method described in the manuscript and its actual implementation. The rebuttal claims that Stage 1 involves three rounds of support and code updates, which is more than the simple restoration of the target support size that Section 4.4 seems to suggest. And, in the Table 7 mentions "three fixed point updates" but does not explain them. This matters because I think Stage 1 in the paper accounts for most of the reported improvement; it also touches upon the theoretical basis. The descent property established in Proposition 2 applies to a single coordinate update evaluated with respect to the updated response rr. However, since Phase 1 involves updating multiple coordinates for the same rr in each round, it does not appear to fall within the scope of this property. Therefore, I appreciate the additional evidence, I continue to feel these points are central to understanding the method and its contributions, and I will therefore maintain my original rating.

Reviewer_a9nJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes ReAlignQ, a post-compression refinement method for models that have undergone both sparsification and quantization. The main idea is to jointly reconsider the locations of nonzero weights and the discrete quantized values assigned to them, while preserving the prescribed sparsity level and the original quantization scales.

Critical strengths

Interesting joint support and quantization refinement mechanism. The idea of jointly selecting weight support and legal quantized values is relevant to jointly compressed models. The method addresses a meaningful limitation of approaches that make sparsification and quantization decisions independently.

Promising empirical improvements. The reported experiments on Llama and Qwen models show improvements in perplexity and downstream task performance over the corresponding compressed baselines. This suggests that post-compression refinement can be a useful direction.

Critical weaknesses

Poor presentation and clarity. The paper introduces concepts without clearly defining them e.g. terms such as support membership, legal codes, target support sizes, and restricted candidate space are undefined before being used. The introduction does not clearly formulate the problem that motivates the proposed method. The theoretical right at the beginning presented without connection to the rest of the paper. These issues substantially hinder understanding and make it difficult to assess the contribution.

Practical relevance. The method focuses on unstructured weight sparsification, whose practical inference benefits are limited on hardware. Although the authors implement bitmap-based CUDA operators, it is not clear why since practical acceleration is not presented. Moreover, the applicability to block-wise quantization is unclear.

Application to quantization-only models, which may be a more practically relevant direction. When precision is low many weights around zero collapse into the exact zero causing natural sparsity in quantized weights. Since quantized weights are supported on hardware, unlike unstructured sparsification, I was curious weather it could be potentially a more interesting direction.

Evaluation, I am also curious how this method is compared to a lightweight quantization-friendly fine-tuning approaches? I think it is possible to compare proposed method with light finetuning given the same time budget.

Questions for authors

My questions would be the same as in Critical Weaknesses.

  • Does it work with Block-wise quantization

  • Bitmap representation and inference performance, why do we need it?

  • Would it work with quantized only models?

  • Comparision with other methods - perhaps light weight finetuning or other methods?

7 comments

Thank you for your review and for noting the joint support–code mechanism and empirical gains. Prior works (SpQR, ICLR 2024; JSQ, ICML 2024; SLiM, ICML 2025; OBR, ICLR 2026) show that combining sparsity with quantization retains higher accuracy at small bit-budgets than ultra-low-bit quantization alone, and our 8×88\times 8 bitmap operator realizes that footprint in serialized storage (14.96 → 3.27 GiB) without slowing measured prefill (8,638.6 vs. 8,406.0 tok/s Dense). Below we address each point with a unified setup, group/block-wise (g128, 8×8) runs, quantization-only experiments, and matched-time fine-tuning baselines.

W1. Problem motivation, definitions, and link between Section 3 and Section 4. The motivating problem is concrete: weights chosen in continuous space during pruning can round to zero or poor grid points under fixed quantization scales, and editing multiple weights independently ignores their loss interactions; we therefore reselect support positions and incoming discrete codes jointly on the parent grid. While these four terms appear in Sections 4.1–4.3, they were used earlier without forward references or a unified setup; in the revision we will consolidate them into one setup block ahead of Section 3:

For a linear matrix flattened into P=doutdinP = d _ {\mathrm{out}}d _ {\mathrm{in}} scalar entries indexed by ii, the parent provides fixed scales si>0s_i > 0 and integer codes zi0z_i^0 (wi0=sizi0w_i^0 = s_i z_i^0), with physical transitions Δi(qi)=si(qi−zi0)\Delta_i(q_i) = s_i(q_i - z_i^0):

  • Code assignment & legal codes (Qiadm\mathcal{Q} _ i^{\mathrm{adm}}): Introduced in Section 1 (Lines 042–043, 055–056), stated in Section 3 (Lines 131–133), and specified in Section 4.1 (Lines 208–215) as {0}∪Qinz\{0\}\cup\mathcal{Q} _ i^{\mathrm{nz}} if zi0=0z_i^0 = 0, and {zi0,0}\{z_i^0, 0\} if zi0≠0z_i^0 \neq 0.
  • Support membership & target support size (KlK_l): Introduced in Abstract/Section 1 (Lines 011–018, 041, 049), stated in Section 3 (Line 134), and formalized in Section 4.1 Eq. (5) (Lines 201–215) via Fl={ql:∥ql∥0=Kl,  ql,i∈Ql,iadm}\mathcal{F} _ l = \{q_l : \|q_l\| _ 0 = K_l,\; q _ {l,i} \in \mathcal{Q} _ {l,i}^{\mathrm{adm}}\}, where ql,i≠0q _ {l,i} \neq 0 denotes support membership and KlK_l is the target nonzero count.
  • Restricted candidate space (Cl\mathcal{C} _ l): Defined in Section 3 (Lines 129–135) and Section 4.2 Eq. (8) (Lines 245–250) as the screened pool of Ml=∣Cl∣≪PM_l = |\mathcal{C} _ l| \ll P editable entries (ql,i=zl,i0q _ {l,i} = z _ {l,i}^0 for i∉Cli \notin \mathcal{C} _ l). Over NN calibration sequences (ut,i=∂Lt(W0)/∂wiu _ {t,i} = \partial L_t(W^0)/\partial w_i), Pass 1 computes gi,Fig_i, F_i to screen Cl\mathcal{C} _ l; Pass 2 retains Ui=[u1,i,…,uN,i]⊤∈RNU_i = [u _ {1,i},\dots,u _ {N,i}]^\top \in \mathbb{R}^N only on Cl\mathcal{C} _ l.
  • Theory-to-method link: Stage 1 (feasible construction) and Stage 2 (exact-KlK_l swaps) minimize Jλ(Δ)=g⊤Δ+12N∥UΔ∥22+λ∥Δ∥22J _ \lambda(\Delta) = g^\top\Delta + \frac{1}{2N}\|U\Delta\| _ 2^2 + \lambda\|\Delta\| _ 2^2 on Fl(Cl)\mathcal{F} _ l(\mathcal{C} _ l). Proposition 1 gives the code-aware score in Eq. (7) (Lines 237–240, 252); Theorem 1 bounds the Gram term in Eq. (9) (Lines 266–269, 275); Proposition 2 proves exact-KlK_l descent in Eq. (14) (Lines 316–317).

W2 / Q1. Block-wise and group-wise quantization. Yes: ReAlignQ applies to group-wise (g128) and 2D block-wise (8×8) scalar quantization without changing the surrogate objective, lowering perplexity and improving zero-shot accuracy across all five tested scalar-scale settings (vector-codebook quantization would require separate block-level actions).

Because Δi(qi)=si(qi−zi0)\Delta_i(q_i) = s_i(q_i - z_i^0) uses a per-entry fixed scale si>0s_i > 0, setting si=sh(i)s_i = s _ {h(i)} for any group or 2D block h(i)h(i) leaves the objective unchanged (for affine grids with fixed zero-point ζh(i)\zeta _ {h(i)}, support is tested by qi≠ζh(i)q_i \neq \zeta _ {h(i)}). Results on Llama 3 8B across SparseGPT + GPTQ scalar group/block grids (preserving all parent scales, codebooks, and support counts):

Scale Layout / Precision / SparsityParent Wiki2 PPL↓+ ReAlignQ PPL↓Zero-Shot Avg.↑ (Parent → +ReAlignQ)
Group g128 / W8A16 / 60%15.5213.31 (-2.21)62.3% → 64.4%
Group g128 / W8A16 / 70%48.5030.82 (-17.68)49.1% → 52.4%
Group g128 / W4A16 / 60%22.9020.85 (-2.05)59.1% → 61.0%
Group g128 / W4A16 / 70%68.4054.10 (-14.30)45.6% → 49.1%
Block 8×8 / W4A16 / 70%70.2556.30 (-13.95)45.1% → 48.6%

W2 / Q2. Bit-budget accuracy advantage over quantization-only, and role of the Bitmap operator. At a matched 3.0 linear bits/entry budget, 50% Sparse W4A16 + ReAlignQ beats unpruned W3A16 (even after refining W3A16) by 1.73 PPL and +3.2 zero-shot points, and our 8×88\times 8 bitmap operator realizes irregular masks in serialized storage (14.96 → 3.27 GiB in Table 5) at matched measured prefill throughput.

  1. Accuracy vs. linear bit-budget (ρb+1\rho b + 1 payload+bitmap bits/entry vs. bb bits/entry for dense quantized): In uniform per-row W4/W3/W2 baselines, going below 4 bpw assigns W3/W2 to every weight. Joint compression prunes redundant entries (1 bitmap bit/entry) so retained weights stay at higher precision.

    On Llama 3 8B under the same per-row GPTQ pipeline (joint rows from Table 2), 50% Sparse W4A16 + ReAlignQ and Quant-only W3A16 use the exact same 3.0 base bits/entry (0.5×4+1=3.00.5\times 4 + 1 = 3.0 vs. 3.03.0, before scale/offset metadata), yet joint + ReAlignQ beats even recode-refined W3A16 by 1.73 PPL and +3.2 zero-shot points:

Method (Llama 3 8B, per-row GPTQ pipeline)Linear Base Bits/EntryParent → +ReAlignQ PPL↓Zero-Shot Avg.↑
Quantization-only W4A16 (0% sparse)4.0 bpw10.62 → 9.89 (w/ Q3 recode)66.5% → 68.1%
50% Sparse W8A16 + ReAlignQ (Table 2)5.0 bpw (0.5×8+10.5\times 8 + 1)9.94 → 9.6967.4% → 69.5%
60% Sparse W8A16 + ReAlignQ (Table 2)4.2 bpw (0.4×8+10.4\times 8 + 1)15.89 → 13.5661.8% → 63.9%
Quantization-only W3A16 (0% sparse)3.0 bpw15.40 → 13.42 (w/ Q3 recode)59.8% → 62.1%
50% Sparse W4A16 + ReAlignQ (Table 2)3.0 bpw (0.5×4+10.5\times 4 + 1)12.33 → 11.6965.3% → 65.3%
Quantization-only W2A16 (0% sparse)2.0 bpw>100 (collapsed)36.4%
60% Sparse W4A16 + ReAlignQ (Table 2)2.6 bpw (0.4×4+10.4\times 4 + 1)24.73 → 22.8158.4% → 60.4%
  1. Why the Bitmap operator is needed:
    • Irregular-mask compression: As stated in Abstract, Section 4.5, and Table 5, the operator executes irregular 8×88\times 8 masks without altering them, packing 70% W8A16 into a 3.27 GiB serialized model footprint (vs. 14.96 GiB masked-dense FP16).
    • Measured throughput & numerical fidelity: It does not claim decode speedup over dense GEMM (decode latency is not measured here), and preserves parent and dense FP16 prefill throughput (8,638.6 vs. 8,642.4 tok/s parent and 8,406.0 Dense; packed PPL 25.67 vs. HF 25.57 in Table 5; direct parent-scale packing gives 0.0 weight diff and 25.59 PPL).

W3 / Q3. Quantization-only models via natural zeros. Yes: ReAlignQ also improves unpruned quantization-only models directly by swapping natural quantization zeros on standard hardware-supported dense low-bit kernels.

In unpruned per-row PTQ on Llama 3 8B, small weights round naturally to code 00 (14.2% in W4A16; 25.6% in W3A16). Setting Kl=n0,lK_l = n _ {0,l} disables Stage 1 and runs 1-in/1-out Stage 2 zero-swaps on fixed parent scales.

Adding ±1\pm 1 neighbor-code updates (an action-set extension beyond the base keep/release rule) under JλJ _ \lambda lowers PPL further on standard dense low-bit GEMMs:

Unpruned Quantization-Only ModelNatural ZerosParent PPL↓+ Zero-Swap (base)↓+ Neighbor Recoding (ext.)↓
Llama 3 8B W4A16 (dense kernel)14.2%10.6210.18 (-0.44)9.89 (-0.73)
Llama 3 8B W3A16 (dense kernel)25.6%15.4014.12 (-1.28)13.42 (-1.98)

W4 / Q4. Comparison with lightweight fine-tuning under the same time budget. On the same 256 C4 sequences, disjoint selection set, and ≤4.15h\le 4.15\text{h} budget (3.97h base solve + 0.18h selection, charged equally to all pipelines), ReAlignQ outperforms tuned STE/QAT, continuous FT + projection, and merged LoRA across the tested starts (with cap 128 at fixed n0n_0):

Starting State / Final NonzerosTuned STE / QATContinuous FT + Proj.Merged LoRAReAlignQ (ours)
Parent (35.48), fixed n0n_0 (≤4.15h\le 4.15\text{h})31.8031.1229.5027.86 (cap128, 4.13h)*
Parent (35.48), target KK (≤4.15h\le 4.15\text{h})29.7630.0228.9425.57 (4.15h end-to-end)
Shared Stage-1 (27.46), KK (≤4.15h\le 4.15\text{h} total)26.4826.7326.1925.57 (4.15h end-to-end)
Double budget: Stage-1 + 4.15h FT (8.05h)26.3526.5126.0825.57 (at 4.15h)

Note: Default cap 8 (4.09h incl. selection) gives 31.68 at fixed n0n_0; batched GPU frontier search at cap 128 adds 0.04h (4.13h ≤4.15h\le 4.15\text{h}) and reaches 27.86. Unmerged LoRA from Stage 1 gives 25.94 PPL (+0.08 GiB dense weights).

Reviewer_a9nJOpenReview ↗

Reviwer response

I thank authors for their hard work and some of my concerns were addressed but not main ones. So, I prefer to keep my score.

My main concerns, which were not addressed:

  • Poor presentation and clarity. Unfortunately, authors responses only solidified my concerns with this regard. Not only due to text quality but also due to inaccuracies present in responses, for example, "Prior works (SpQR, ICLR 2024; JSQ, ICML 2024; SLiM, ICML 2025; OBR, ICLR 2026) show that combining sparsity with quantization retains higher accuracy at small bit-budgets than ultra-low-bit quantization alone".

There is no such claim in SpQR since the main focus of the paper is on quantization and outlier weights.

  • Practical relevance. For the practical relevance, OBR, for example, combines 2:4 semistructured sparsity and quantization. Semistructures 2:4 sparsity indeed provides acceleration but unstructured sparsity alone does not. So, the practical relevance and motivation are not explained.

Some ideas for authors. As I suggested earlier, this method has a practical relevance since applying your method to quantized models only has practical meaning because quantization naturally leads to sparsity. But this requires refocusing paper's narrative.

Reviewer_Ud6rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

ReaAlignQ addresses the problem that independent optimization of Quantization and Sparsification for LLMs can have competing objectives. The authors propose to look at gradient signatures to jointly optimize sparsification and quantization. They assume that the level of sparsity is prescribed. The paper focuses on codebooks for implementing quantization. The main contribution of the paper is a code-aware-quantization grid, combining a fixed sparsity level with a quantization setting.

Critical strengths

The paper provides a method for joint sparsity-quantization optimization for LLMs with deep theoretical support and experimental results.The work expands the current body of knowledge and provides interesting ideas.

Critical weaknesses

The paper uses concepts and notation without defining them first. This makes the paper harder to read.

Questions for authors

What is the definition of ``per-sample gradient signature'' For example, it refers to gradient Gram and starts using it \S 3. It is later in the appendix line 1110, that the Gradient Gram is defined G=1UTUG = \frac 1 U^T U. This makes the paper harder to read. I would be better to state the definition first or at least include a reference to the corresponding equation and section. Another example in \S3, Proposition 1 uses the gradient gig_i without previously defining it, nor pointing to the definition given in \S4.2 equation (6). Also, in the same section ut,i=∂Lt∂wi(W0)u_{t,i} = \frac {\partial L_t} {\partial w_i} (W^0), ii is said to represent the coordinate ii. Is this the weight index? Are the subindexes of weight-tensor flattened? Here it seems that each ut,iu_{t,i} is a scalar entry of the vector Ui=[u1,i,…uN,i]TU_i = [u_{1,i}, \ldots u_{N,i}]^T. Is that the case?

On lines 248 to 250: ``After screening, a second pass over the same calibration data collects gradients from individual samples only for the M=∣C∣M = |C| candidates. This reduces the retained signature storage from O(NP)O(N P ) to O(NM)O(N M ).'', what is PP? I didn't find a definition of it anywhere. The meaning of Δ\Delta is not always clear. Before Proposition 1 we have the definiton Δi(q)=si(q−zi0)\Delta_i(q) = s_i(q - z_i^0), which seems to be different to its use in Theorem 1 S=\mboxsupp(Δ)S = \mbox{supp} (\Delta). Where \supp\supp probably means support? Is Δ=[Δ1(q1),…,ΔN(qN)]T\Delta = [\Delta_1(q_1), \ldots, \Delta_N(q_N)]^T or Δ=[Δ1(q),…,ΔN(q)]T\Delta = [\Delta_1(q), \ldots, \Delta_N(q)]^T, or something else?

Could you summarize the selection criteria with assumptions and definitions in one section of the paper. The process is spread across the documents and it is harder to follow if you have to refer to different sections.

This is a summary of my understanding of your method:

From line 271 Fi=1N∥Ui∥2F_i = \frac 1 N \| U_i \|^2

Combining with equation (9) J(Δ)=gTΔ+12ΔTGΔ+λ∥Δ∥22, G=1NUTUJ(\Delta) = g^T \Delta + \frac 1 2 \Delta^T G \Delta + \lambda \|\Delta\|_2^2, ~ G = \frac 1 N U^TU

Which is the same expression in Appendix A.4, line 138, except that in the Appendix, the dependency on λ\lambda is clearly stated as a subindex JλJ_\lambda and C=GC = G.

So the joint problem as I understand it is: For each matrix ll minimize $\min_{q \in \mathcal{F_l}(C)} J(\Delta(q)) $

The paper assumes that the sparsity level is given. What are your thoughts on efficient ways to find ``best'' sparsity-quantization configurations?

2 comments

Thank you for tracing the derivations across Sections 3–4 and Appendix A.4 and listing the undefined symbols. Putting Section 3 ahead of the formal setup in Section 4 was an organizational mistake; in the revision we will place all notation, assumptions, and selection rules in one self-contained section before Proposition 1. We follow the order of your review.

W1 / Q1. Per-sample gradient signatures, GG, gig_i, and flattened coordinates. Yes: each matrix Wl∈Rdout,l×din,lW_l \in \mathbb{R}^{d _ {\mathrm{out},l}\times d _ {\mathrm{in},l}} is flattened into a vector w∈RPw \in \mathbb{R}^P with P=dout,ldin,lP = d _ {\mathrm{out},l}d _ {\mathrm{in},l}, i∈{1,…,P}i \in \{1,\dots,P\} indexes one scalar weight entry, and each ut,iu _ {t,i} is the tt-th scalar entry of Ui∈RNU_i \in \mathbb{R}^N.

At the compressed parent W0W^0, let LtL_t be the full-model loss on calibration sequence t∈{1,…,N}t \in \{1,\dots,N\}. With ut,i=∂Lt∂wi(W0)∈Ru _ {t,i} = \frac{\partial L_t}{\partial w_i}(W^0) \in \mathbb{R}, the per-sample gradient signature of coordinate ii and its first/second moments are

Ui=[u1,i,…,uN,i]⊤∈RN,gi=1N∑t=1Nut,i,Fi=1N∑t=1Nut,i2=1N∥Ui∥22.U_i = [u _ {1,i}, \dots, u _ {N,i}]^\top \in \mathbb{R}^N, \qquad g_i = \frac{1}{N}\sum _ {t=1}^N u _ {t,i}, \qquad F_i = \frac{1}{N}\sum _ {t=1}^N u _ {t,i}^2 = \frac{1}{N}\|U_i\| _ 2^2.

After screening selects an ordered candidate subset C={i1,…,iM}⊆{1,…,P}\mathcal{C} = \{i_1,\dots,i_M\} \subseteq \{1,\dots,P\} of size M=∣C∣M = |\mathcal{C}|, stacking their signatures as columns gives U=[Ui1,…,UiM]∈RN×MU = [U _ {i_1},\dots,U _ {i_M}] \in \mathbb{R}^{N\times M} and the empirical gradient Gram matrix G=1NU⊤U∈RM×MG = \frac{1}{N}U^\top U \in \mathbb{R}^{M\times M} (with diagonal Gaa=FiaG _ {aa} = F _ {i_a}).

On the paper summary: the parent quantization grid and scales sis_i remain fixed. ReAlignQ optimizes support membership and incoming legal codes on that grid rather than redesigning the quantizer.

Q2. Meaning of PP, and the indexing of Δ\Delta and S=supp⁡(Δ)S = \operatorname{supp}(\Delta). P=dout,ldin,lP = d _ {\mathrm{out},l}d _ {\mathrm{in},l} is the number of scalar weights in the matrix before screening, while Δ∈RM\Delta \in \mathbb{R}^M is indexed by the MM screened candidate coordinates (re-indexed a∈{1,…,M}a \in \{1,\dots,M\} for ia∈Ci_a \in \mathcal{C}), not by the NN calibration samples.

  • PP and signature storage: Storing NN-sample signatures for all PP entries would require O(NP)O(NP) memory. Pass 1 accumulates only (gi,Fi)(g_i, F_i) in O(P)O(P) space to screen C\mathcal{C} (M≪PM \ll P); Pass 2 retains U∈RN×MU \in \mathbb{R}^{N\times M} only on C\mathcal{C}, reducing retained signature storage from O(NP)O(NP) to O(NM)O(NM) (1.03 GiB1.03\text{ GiB} total on Llama 3 8B for N=256N=256). Both passes evaluate full-model gradients at W0W^0.

  • Δ\Delta and S=supp⁡(Δ)S = \operatorname{supp}(\Delta): Neither NN-indexed expression applies. Holding coordinates outside C\mathcal{C} fixed at z0z^0, let q=[qi1,…,qiM]⊤q = [q _ {i_1},\dots,q _ {i_M}]^\top be the candidate code vector on C\mathcal{C}. It induces the MM-dimensional weight-change vector

    Δ(q)=[Δi1(qi1),…,ΔiM(qiM)]⊤∈RM,Δi(qi)=si(qi−zi0)∈R,\Delta(q) = [\Delta _ {i_1}(q _ {i_1}), \dots, \Delta _ {i_M}(q _ {i_M})]^\top \in \mathbb{R}^M, \qquad \Delta_i(q_i) = s_i(q_i - z_i^0) \in \mathbb{R},

    so multiplying U∈RN×MU \in \mathbb{R}^{N\times M} by Δ(q)∈RM\Delta(q) \in \mathbb{R}^M yields the NN-sample response vector r=UΔ(q)∈RNr = U\Delta(q) \in \mathbb{R}^N. Yes, \supp denotes support. The three counts in the paper refer to distinct sets:

SymbolDefinition
K,  KCK,\; K _ {\mathcal{C}}Prescribed nonzeros in the full matrix (∥qfull∥0=K\lVert q _ {\mathrm{full}}\rVert_0 = K) and inside C\mathcal{C} (KC=K−∥zCc0∥0K _ {\mathcal{C}} = K - \lVert z^0 _ {\mathcal{C}^c}\rVert_0, Prop. 1)
supp⁡(q)\operatorname{supp}(q)Final nonzero coordinates in C\mathcal{C}, {i∈C:qi≠0}\{i \in \mathcal{C} : q_i \neq 0\}, of size KCK _ {\mathcal{C}}
S=supp⁡(Δ),  kS = \operatorname{supp}(\Delta),\; kEdited candidate coordinates, {i∈C:Δi(qi)≠0}\{i \in \mathcal{C} : \Delta_i(q_i) \neq 0\}, of size k=∣S∣≤Mk = \lvert S\rvert \le M (Thm. 1)

For example, releasing an active weight (zi0≠0→qi=0z_i^0 \neq 0 \to q_i = 0) puts ii in supp⁡(Δ)\operatorname{supp}(\Delta) because Δi=−sizi0≠0\Delta_i = -s_i z_i^0 \neq 0, whereas i∉supp⁡(q)i \notin \operatorname{supp}(q) because its final code is zero.

Q3 & Q4. Selection criteria and unified matrix objective. Your summary matches the objective. We will consolidate the domain, objective, and selection procedure before the theory.

Because Fi=1N∥Ui∥22=GiiF_i = \frac{1}{N}\|U_i\| _ 2^2 = G _ {ii}, the diagonal and off-diagonal sums in Eq. (9) combine identically into 12Δ⊤GΔ=12N∥UΔ∥22\frac{1}{2}\Delta^\top G \Delta = \frac{1}{2N}\|U\Delta\| _ 2^2, matching JλJ _ \lambda in Appendix A.4 with C=GC = G. For one matrix (vectors in RM\mathbb{R}^M):

  1. Domain F(C)\mathcal{F}(\mathcal{C}): Scales si>0s_i > 0 are fixed; exterior codes (i∉Ci \notin \mathcal{C}) remain at zi0z_i^0. For i∈Ci \in \mathcal{C}, Qiadm={0}∪Qinz\mathcal{Q} _ i^{\mathrm{adm}} = \{0\}\cup\mathcal{Q} _ i^{\mathrm{nz}} if zi0=0z_i^0 = 0, and {zi0,0}\{z_i^0, 0\} if zi0≠0z_i^0 \neq 0, subject to ∑i∈C1[qi≠0]=KC\sum _ {i\in\mathcal{C}}\mathbf{1}[q_i \neq 0] = K _ {\mathcal{C}} .
  2. Unified objective: min⁡q∈F(C)Jλ(Δ(q))=g⊤Δ(q)+12Δ(q)⊤GΔ(q)+λ∥Δ(q)∥22\min _ {q \in \mathcal{F}(\mathcal{C})} J _ \lambda(\Delta(q)) = g^\top \Delta(q) + \frac{1}{2}\Delta(q)^\top G \Delta(q) + \lambda \|\Delta(q)\| _ 2^2, with G=1NU⊤UG = \frac{1}{N}U^\top U.
  3. Selection stages & theoretical roles: Let ϕi(q)=giΔi(q)+12(Fi+2λ)Δi(q)2\phi_i(q) = g_i\Delta_i(q) + \frac{1}{2}(F_i + 2\lambda)\Delta_i(q)^2.
    • Screening (Pass 1 & 2): Pass 1 scores candidates individually at λ=0\lambda=0 (min⁡q∈Qinzϕi(q)\min _ {q\in\mathcal{Q} _ i^{\mathrm{nz}}}\phi_i(q) for zeros; ϕj(0)\phi_j(0) for nonzeros) to build an over-complete pool C=Czero∪Cnonzero\mathcal{C} = \mathcal{C} _ {\mathrm{zero}}\cup\mathcal{C} _ {\mathrm{nonzero}}. Pass 2 retains UU on C\mathcal{C}.
    • Exact Prop. 1 diagonal baseline: For Jdiag(q)=∑i∈Cϕi(qi)J _ {\mathrm{diag}}(q) = \sum _ {i\in\mathcal{C}}\phi_i(q_i) (zeroing off-diagonal terms of GG), Proposition 1 picks the KCK _ {\mathcal{C}} candidates with smallest δi=min⁡q∈Qiadm∖{0}ϕi(q)−ϕi(0)\delta_i = \min _ {q\in\mathcal{Q} _ i^{\mathrm{adm}}\setminus\{0\}}\phi_i(q) - \phi_i(0).
    • Coupled construction (Thm. 1): Theorem 1 bounds the omitted off-diagonal interaction ∣1N∑i<jUi⊤UjΔiΔj∣\lvert\frac{1}{N}\sum _ {i<j}U_i^\top U_j\Delta_i\Delta_j\rvert via pairwise cosines of UiU_i and ∑iFiΔi2\sum_i F_i\Delta_i^2. From (Δ,r=UΔ)(\Delta, r=U\Delta), editing unedited coordinate ii by xx changes JλJ _ \lambda by mi(x∣Δ)=(gi+1NUi⊤r)x+12(Fi+2λ)x2m_i(x\mid\Delta) = (g_i + \frac{1}{N}U_i^\top r)x + \frac{1}{2}(F_i + 2\lambda)x^2. When n0<Kn_0 < K, the solver runs Table 7's three batched rounds (we will revise Section 4.4 to match): Round 1 fills d=K−n0d = K - n_0 holes at r=0r=0, and Rounds 2–3 re-select dd holes/codes via arg⁡min⁡q∈Qinzmi(siq∣Δ−i)\arg\min _ {q\in\mathcal{Q} _ i^{\mathrm{nz}}} m_i(s_i q\mid\Delta _ {-i}) before refreshing r=UΔr = U\Delta.
    • Paired exchanges (Prop. 2): At fixed ∥qfull∥0=K\|q _ {\mathrm{full}}\| _ 0 = K, activating i∈Czeroi \in \mathcal{C} _ {\mathrm{zero}} to qq and releasing j∈Cnonzeroj \in \mathcal{C} _ {\mathrm{nonzero}} (xjrel=−sjzj0x_j^{\mathrm{rel}} = -s_j z_j^0) shifts JλJ _ \lambda by ΔJi,jpair(q)=mi(siq∣Δ)+mj(xjrel∣Δ)+1NUi⊤Uj(siq)xjrel\Delta J _ {i,j}^{\mathrm{pair}}(q) = m_i(s_i q\mid\Delta) + m_j(x_j^{\mathrm{rel}}\mid\Delta) + \frac{1}{N}U_i^\top U_j(s_i q)x_j^{\mathrm{rel}}. Accepting qi⋆=arg⁡min⁡qΔJi,jpair(q)q_i^\star = \arg\min _ {q}\Delta J _ {i,j}^{\mathrm{pair}}(q) iff ΔJi,jpair<−τ\Delta J _ {i,j}^{\mathrm{pair}} < -\tau preserves KK and lowers JλJ _ \lambda.

Results on Llama 3 8B (EIWR W8A16; refined rows share C\mathcal{C}, codes, λ\lambda, and KK; W0W^0 starts at n0<Kn_0 < K):

Setting (Llama 3 8B, EIWR W8A16)Final nnz60% PPL↓70% PPL↓GPU-h (70%)
Compressed parent W0W^0n0n_016.7235.48—
Exact Prop. 1 minimizer of JdiagJ _ {\mathrm{diag}} on F(C)\mathcal{F}(\mathcal{C})KK14.3930.423.36
ReAlignQ Stage 1 (coupled construction)KK14.3227.463.90
Full ReAlignQ (Stage 1 + Stage 2 swaps)KK13.4125.573.97

Q5. Finding "best" sparsity–quantization configurations. Using ReAlignQ as an inner solver under a byte-budgeted outer allocation lowers Llama 3 8B W8A16 PPL from 25.57 (uniform 70%) to 24.83 and raises mean zero-shot accuracy by 0.8 percentage points (53.4% → 54.2%) at the same 3.27 GiB model size, adding 0.62 GPU-hours.

Pass 1 at W0W^0 yields code-aware coordinate gains across all 224 matrices. We rank marginal surrogate gain per byte (payload, bitmaps, and scales, with layer sparsity in [55%,80%][55\%, 80\%]), refine top allocations with ReAlignQ, and select on held-out calibration CE since JλJ _ \lambda omits cross-matrix terms.

Mixed bitwidths (e.g., W4 vs. W8 per matrix) require precomputing parent grids and statistics for a small menu per matrix before solving a multiple-choice knapsack over total bytes, as bitwidth alters Wl0W_l^0, sis_i, and Qiadm\mathcal{Q} _ i^{\mathrm{adm}}.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.