- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ReAlignQ, a post-compression refinement method for jointly sparse and quantized LLMs. It performs support-preserving weight exchanges while selecting legal quantized codes for newly activated weights, using gradient-based screening and an empirical gradient-Gram surrogate to model candidate interactions. Experiments show improvements without changing the parent quantization scales or sparsity budgets.
Critical strengths
- ReAlignQ addresses a practical mismatch between support selection and quantized code assignment under fixed compression constraints.
- I appreciate the paper’s theoretical analysis and its implementation of operators tailored to the refined sparse-quantized models. Although the current evaluation does not demonstrate substantial practical acceleration, these contributions remain valuable from a research perspective.
- Empirical results show improvements
Critical weaknesses
- I think it's necessary to compare with dense models' performance. At 70% sparsity, the compressed models already suffer substantial performance degradation, which may limit their practical usability. I therefore encourage the authors to provide examples of actual model responses.
- The practical value of this post-compression setting should be better motivated. Moreover, the refinement stage introduces additional computational cost, which is not sufficiently characterized in the experiments.
Questions for authors
See above
4 comments
W2. Practical value of the post-compression setting, and full characterization of refinement cost. ReAlignQ improves quality across the four tested compression pipelines without altering parent scales or adding deployment parameters. At 3.0 linear bits/entry, 50% Sparse W4A16 + ReAlignQ outperforms unpruned W3A16 by 1.73 PPL, and its one-time 3.97 GPU-hour offline solve beats matched-time fine-tuning across the tested starting states.
- Why post-compression refinement matters in practice:
- Broad compatibility: ReAlignQ refines JSQ, OBR-GPTQ, EIWR, and SparseGPT+GPTQ checkpoints while keeping parent scales, sparsity budget, and prefill throughput (8,638.6 vs. 8,642.4 tok/s parent, 8,406.0 Dense, in 3.27 GiB, Table 5).
- Unpruned quantization-only models: Via natural zeros in per-row W4A16/W3A16, base 1-in/1-out zero-swaps lower PPL (10.62 → 10.18, 15.40 → 14.12); adding neighbor-recoding reaches 9.89 and 13.42 on standard dense kernels.
- Matched 3.0-bpw linear bit-budget: In uniform per-row W4/W3/W2 baselines, going below 4 bpw assigns W3/W2 to every weight, whereas joint compression prunes redundant entries (1 bitmap bit/entry) so retained weights keep higher precision. On Llama 3 8B, 50% Sparse W4A16 + ReAlignQ and Quant-only W3A16 both use 3.0 linear base bits/entry ( vs. ), yet joint + ReAlignQ beats even recode-refined W3A16 by 1.73 PPL and +3.2 zero-shot points (W2A16 at 2.0 bpw collapses >100 PPL):
| Method (Llama 3 8B, per-row GPTQ pipeline) | Linear Base Bits/Entry | Parent → Refined PPL↓ | Zero-Shot Avg.↑ |
|---|---|---|---|
| Quantization-only W4A16 (base swap: 10.18; +recode: 9.89) | 4.0 bpw | 10.62 → 9.89 | 66.5% → 68.1% |
| 50% Sparse W8A16 + ReAlignQ (Table 2) | 5.0 bpw () | 9.94 → 9.69 | 67.4% → 69.5% |
| Quantization-only W3A16 (base swap: 14.12; +recode: 13.42) | 3.0 bpw | 15.40 → 13.42 | 59.8% → 62.1% |
| 50% Sparse W4A16 + ReAlignQ (Table 2) | 3.0 bpw () | 12.33 → 11.69 | 65.3% → 65.3% |
-
Stage-wise breakdown of refinement cost & comparison with same-budget fine-tuning: On Llama 3 8B (EIWR W8A16, 70%, 256 C4 sequences), the 3.97 GPU-hour offline solver cost (peak 37,152 MiB; 1.03 GiB signatures discarded after calibration) breaks down into:
- Pass 1 (full-model gradient statistics + screening): 3.28 GPU-h (82.6%);
- Pass 2 (retaining candidate signatures on entries): 0.58 GPU-h (14.6%);
- In-memory coupled solver + materialization: 0.11 GPU-h (0.04h Stage 1 + 0.07h Stage 2 & materialization, 2.8%; vs. 3.36h and 30.42 PPL for exact Prop. 1 diagonal without Pass 2). Adding 5-ratio validation selection takes 0.18 GPU-h (4.15h end-to-end).
Under the identical 256 C4 sequences, disjoint selection set, and end-to-end budget, ReAlignQ outperforms tuned STE/QAT, continuous FT + projection, and merged LoRA across tested starting states (with an expanded swap cap in the fixed- row):
| Starting State / Final Nonzeros (Llama 3 8B, 70%) | Tuned STE / QAT | Continuous FT + Proj. | Merged LoRA | ReAlignQ (ours) |
|---|---|---|---|---|
| Parent (35.48), fixed () | 31.80 | 31.12 | 29.50 | 27.86 (cap128, 4.13h)* |
| Parent (35.48), target () | 29.76 | 30.02 | 28.94 | 25.57 (4.15h end-to-end) |
| Shared Stage-1 (27.46), ( total) | 26.48 | 26.73 | 26.19 | 25.57 (4.15h end-to-end) |
| Double budget: Stage-1 + 4.15h FT (8.05h) | 26.35 | 26.51 | 26.08 | 25.57 (at 4.15h) |
Note: Default cap 8 (4.09h incl. selection) gives 31.68 at fixed ; batched GPU frontier search at cap 128 adds 0.04h (4.13h ) and reaches 27.86. Unmerged LoRA from Stage 1 gives 25.94 PPL (+0.08 GiB extra dense weights). Five disjoint C4 seeds further confirm stability (25.64 ± 0.14 PPL).
We will incorporate these Dense comparisons, TriviaQA/continuation evaluations, footprint analyses, and fine-tuning baselines into the revision.
I really appreciate authors' effort during rebuttal. My concerns have been adequately addressed. I will update my score.
Thank you for your careful review, constructive feedback, and for raising your score. We are glad that our clarifications and additional experiments have addressed your concerns. We will incorporate these results and presentation improvements into the revised manuscript.
Thank you for your positive assessment of our problem formulation, theoretical analysis, and bitmap operators. Below we address both concerns with same-evaluator Dense FP16 comparisons across 50%–70% sparsity, actual model continuations, bit-budget accuracy comparisons, and a full breakdown of refinement cost vs. matched-time fine-tuning.
W1. Comparison with the dense model across sparsity levels, and actual model responses. Under the same 6-task lm-eval 0-shot protocol, 50% W8A16 + ReAlignQ retains 93.7% of Dense FP16 accuracy (69.5% vs. 74.2%) at 5.0 linear bits/entry. At 70% sparsity (3.4 linear bits/entry; 3.27 vs. 14.96 GiB serialized footprint on Llama 3 8B in Table 5, and 66.2% vs. 74.4% Dense on Qwen3-14B), ReAlignQ lifts 0-shot closed-book TriviaQA EM on Llama 3 8B from 28.6% to 41.8% (+13.2 points) and improves two illustrative continuations below.
Note: Linear payload + bitmap bits/entry (ρb+1 at density ρ and b-bit codes; 16.0 bpw for FP16), before scale/offset metadata; 14.96 GiB and 3.27 GiB are whole-model serialized footprints from Table 5.
50%–60% sparsity (and 70% on 14B, keeping 89% of Dense accuracy) serves memory-constrained deployment, while 70% on 8B stress-tests severe compression (retaining a clear gap to Dense, yet recovering up to 20.16 PPL over the parent).
Greedy continuations (first ~30 words shown) on Llama 3 8B (70% EIWR W8A16) illustrate this shift:
In these two examples, the 70% parent contradicts the premise ("only treats a disease") or repeats "the host's immune system", whereas the refined continuations follow the prompt premise ("only prevents future infections", bacterial growth/infection control).