- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReFiT, a post-hoc reconstruction method for recovering the quality degradation caused by block-shared FFN activation sparsity in LLMs. The method uses calibration data to fit the existing down-projection weights through anchored ridge regression, reconstructing the missing FFN outputs from retained activations. The resulting correction is absorbed into the original projection weights, requiring no additional matrix multiplication during inference. Experiments on LLaMA2-7B, LLaMA3.2-1B/3B, and Qwen3-4B demonstrate improvements in WikiText-2 perplexity and LongBench performance, particularly under high sparsity.
Critical strengths
- Simple and practical approach. ReFiT requires neither gradient-based retraining nor additional inference-time compensation branches. Its closed-form formulation makes it straightforward to apply to existing sparse FFN implementations.
- Consistent empirical improvements. The method demonstrates substantial quality recovery across multiple models, sparsity ratios, and block sizes. The improvements at high sparsity are particularly encouraging.
- Useful error analysis. The decomposition of reconstruction errors into block-mean and within-block components provides insights into the limitations of block-shared activation sparsity. The authors also include ablations to investigate token-dependent corrections.
Critical weaknesses
- Limited methodological novelty and insufficient discussion of closely related work. The core method is essentially an anchored ridge regression that reconstructs missing outputs from retained activations, followed by folding the correction into existing weights. Similar principles have been explored in previous post-hoc reconstruction and compensation methods.
In particular, the paper does not discuss several closely related works:
- GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks. https://arxiv.org/abs/2602.23795
- Reconstruct the Pruned Model without Any Retraining. https://arxiv.org/abs/2407.13331
- Restoring Pruned Large Language Models via Lost Component Compensation. https://arxiv.org/abs/2510.21834
GRAIL is especially relevant, as it also uses calibration-based linear reconstruction and absorbs the resulting compensation into existing consumer weights without additional inference operations. Although these methods address different compression settings, the conceptual and mathematical overlap deserves a more thorough discussion. The authors should clarify what methodological contribution ReFiT introduces beyond applying existing linear reconstruction techniques to block-shared activation sparsity.
-
Missing evaluation of actual prefill acceleration. The primary motivation of the paper is efficient LLM prefill. However, the experiments focus almost exclusively on quality recovery, without reporting actual prefill latency, time-to-first-token (TTFT), or throughput. Although ReFiT does not introduce additional matrix multiplications, this alone does not establish the end-to-end efficiency of the resulting inference system. In particular, the practical implications of sparse kernel execution, dynamic neuron selection, and maintaining different fitted weights for different sparsity configurations remain unclear. A measured latency–quality trade-off would substantially strengthen the practical contribution.
-
Limited evaluation on standard zero-shot benchmarks. The evaluation primarily relies on WikiText-2 perplexity and LongBench. Standard zero-shot benchmarks are missing. Furthermore, calibration and perplexity evaluation both use WikiText-2, although with separate training and test splits. Cross-dataset evaluation would provide stronger evidence of generalization.
-
Token-dependent recovery appears relatively limited. Although recovering token-dependent errors is a central motivation, the reported within-block error reduction is approximately 15% in the representative setting. Moreover, the appendix indicates that much of this improvement is concentrated in the final layer, with substantially smaller recovery after excluding it. This raises questions about how broadly the proposed mechanism recovers token-specific information across the network.
Questions for authors
- How does ReFiT differ mathematically and conceptually from existing post-hoc linear reconstruction methods, particularly GRAIL? Could the authors clarify the specific novelty beyond the application to block-shared activation sparsity?
- Can the authors provide actual prefill latency or TTFT measurements, ideally together with a quality–latency Pareto comparison?
- Can the authors evaluate ReFiT on standard zero-shot benchmarks, such as HellaSwag, PIQA, ARC, and WinoGrande?
- How does ReFiT perform when (T=1)? Is the relative reconstruction gain larger under block-shared sparsity than under token-wise sparsity?
- Why is within-block error recovery relatively limited and concentrated in the final layer? How much does this mechanism contribute to the observed end-to-end quality improvements?
4 comments
W2 & Q2. Measured prefill acceleration and hardware implications.
We benchmarked a block-sparse Triton kernel, including dynamic predictor and top-k selection, over block sizes 8, 32, and 128.
| Block size | Sparsity | Sparse w/o ReFiT, ms | Sparse w/ ReFiT, ms | Speedup vs dense, w/o / w/ | ReFiT / Sparse |
|---|---|---|---|---|---|
| 8 | 50% | 1095.90 | — | 0.31× / — | — |
| 8 | 70% | 528.57 | — | 0.64× / — | — |
| 8 | 90% | 290.98 | — | 1.16× / — | — |
| 32 | 50% | 422.38 | — | 0.80× / — | — |
| 32 | 70% | 301.28 | 301.20 | 1.12× / 1.12× | 0.9997× |
| 32 | 90% | 226.41 | — | 1.49× / — | — |
| 128 | 50% | 281.86 | 281.98 | 1.20× / 1.20× | 1.0004× |
| 128 | 70% | 251.31 | 251.89 | 1.35× / 1.34× | 1.0023× |
| 128 | 90% | 214.77 | 215.73 | 1.57× / 1.56× | 1.0045× |
Table R2. Llama2-7B model-prefill latency: A100-SXM4-40GB, fp16, batch 1, 3,500 tokens. Medians use five repeats for block size 8 and ten otherwise. Speedups use the paired dense measurement for each arm; dashes denote unmeasured ReFiT configurations.
- Larger blocks and higher sparsity improve execution efficiency over this grid. At , ReFiT-Sparse is 1.20–1.56× faster than dense; smaller blocks require higher sparsity to offset overhead. Across 12 matched Sparse/ReFiT-Sparse comparisons, including shorter prompts, the latency ratio is 0.999–1.009 (median 1.002).
- Dynamic selection, irregular weight access, limited reuse at small blocks, and unchanged attention/dense computation limit speedup relative to the FFN arithmetic reduction. Sparse accelerators such as SLIM (arXiv:2507.09201) fetch only active neurons, motivating our complementary objective: recover accuracy at fixed sparsity without extra online correction operations.
- For Llama2-7B, calibration takes approximately 30 minutes on an A100 (4 minutes for 16 sequences), plus a 20-second solve, under shared GPU load. Fits are operating-point-specific; a dynamic server can precompute supported configurations. Each 16-bit down-projection copy occupies about 2.9 GB temporarily; but it is then folded to during inference, removing the need for a separate correction matrix.
W3 & Q3. Zero-shot evaluation and prefill-heavy downstream tasks.
To test whether gains extend beyond severely degraded models, we added (1) zero-shot reasoning and MMLU, and (2) prefill-heavy classification evaluations across several models. All use WikiText-2-calibrated weights without target-task fitting.
(1) zero-shot reasoning and MMLU
| Evaluation | Model | Block / sparsity | Dense | Sparse | + ReFiT |
|---|---|---|---|---|---|
| Zero-shot mean (3 tasks) | Llama2-7B | 8 / 50% | 58.55 | 52.23 | 53.08 |
| Zero-shot mean (3 tasks) | Llama2-7B | 8 / 70% | 58.55 | 43.81 | 46.06 |
| Zero-shot mean (3 tasks) | Llama2-7B | 8 / 90% | 58.55 | 32.34 | 34.59 |
| Zero-shot mean (3 tasks) | Llama2-7B | 32 / 50% | 58.55 | 52.10 | 52.75 |
| Zero-shot mean (3 tasks) | Llama2-7B | 32 / 70% | 58.55 | 43.84 | 45.92 |
| Zero-shot mean (3 tasks) | Llama2-7B | 32 / 90% | 58.55 | 32.83 | 34.71 |
| MMLU | Llama-3.2-1B-Instruct | 128 / 90% | 46.13 | 44.37 | 45.58 |
| MMLU | Llama-3.2-3B-Instruct | 128 / 90% | 60.72 | 57.78 | 59.86 |
Table R3. Additional downstream accuracy (%). The zero-shot mean equally weights HellaSwag, ARC-Easy, and ARC-Challenge on their full evaluation sets (10,042/2,376/1,172 examples), calculated before rounding. MMLU is 5-shot with 14,042 questions. Zero-shot contexts have no dense prefix after BOS; MMLU retains the first/last 128 prompt tokens as dense. Answer continuations remain dense.
- ReFiT improves the three-task zero-shot mean in all six configurations, including 50% sparsity. On MMLU, the sparse models remain close to dense accuracy, and ReFiT recovers much of the remaining loss for both Llama-3.2 models.
(2) prefill-heavy classification
| Benchmark | Model | Dense | Sparse | + ReFiT |
|---|---|---|---|---|
| Banking77 | Llama2-7B | 70.60 | 35.67 | 52.73 |
| Banking77 | Llama-3.2-1B-Instruct | 61.33 | 40.00 | 45.40 |
| Banking77 | Llama-3.2-3B-Instruct | 65.80 | 44.33 | 54.67 |
| Banking77 | Qwen3-4B | 74.73 | 41.13 | 48.93 |
| CLINC150 | Llama2-7B | 82.60 | 65.60 | 70.20 |
| CLINC150 | Llama-3.2-1B-Instruct | 73.80 | 59.67 | 61.47 |
| CLINC150 | Llama-3.2-3B-Instruct | 80.93 | 65.33 | 68.80 |
| CLINC150 | Qwen3-4B | 85.80 | 68.47 | 71.80 |
Table R4. Prefill-heavy classification accuracy (%), block size 128, 70% sparsity. Prompts contain one demonstration per intent (77/150), with the first/last 128 prompt tokens and answer continuations dense. Confirmation sets are disjoint from exploratory selection: 1,500 examples per cell, except Llama2-7B CLINC150 (500).
- Both prefill-heavy tasks improve across all four models. Llama2-7B Banking77 uses roughly 1,827-token prompts, with 84% processed sparsely. At 50% sparsity, separate Banking77 confirmation runs also gain 5.0/4.1/5.5 points on Llama2-7B/Llama-3.2-3B/Qwen3-4B.
We also changed the calibration corpus while fixing WikiText-2 test evaluation:
| Calibration | PPL, 50% ↓ | PPL, 70% ↓ | PPL, 90% ↓ |
|---|---|---|---|
| No ReFiT | 8.913 | 20.814 | 615.90 |
| WikiText-2 | 8.053 | 14.626 | 82.14 |
| C4 | 8.285 | 15.529 | 82.26 |
| MiniPile | 8.333 | 15.820 | 91.69 |
Table R5. Calibration transfer: Llama2-7B, block size 128, 128 sequences, common A100 setup.
- C4 and MiniPile improve PPL at all three sparsities: at 90%, the two fits reduce PPL from 615.90 to 82.26 and 91.69. Thus, substantial recovery transfers from C4 and MiniPile calibration to WikiText-2 evaluation.
W4 & Q5. Contribution of token-dependent correction and multiple layers.
We isolate the contribution of token-dependent correction by retaining only the block-mean component of ReFiT, and test the importance of applying ReFiT across layers by restricting it to the final layer.
| Correction | PPL, 50% ↓ | PPL, 70% ↓ | PPL, 90% ↓ |
|---|---|---|---|
| None | 8.913 | 20.814 | 615.90 |
| Calibration-mean replacement | 8.577 | 20.737 | 531.03 |
| Block-mean component | 8.269 | 15.958 | 133.27 |
| Final-layer-only | 8.738 | 20.520 | 522.80 |
| Full ReFiT | 8.053 | 14.626 | 82.14 |
Table R6. Llama2-7B correction ablation, diagnostic PPL ↓, block size 128. The block-mean arm averages the input-dependent ReFiT correction within each block; all arms use the same A100 evaluation setup.
- Full ReFiT gives the best PPL at every tested sparsity. The block-mean component already helps substantially, and retaining token-dependent correction improves it further. Calibration-mean replacement, , uses calibration-average activations and provides less recovery than correction based on the current block's retained activations. Final-layer-only refitting recovers much less PPL than full ReFiT.
Q4. Token-wise sparsity () versus block sharing.
We added a control and compare it with using actual-activation selection (dense activations supply exact neuron scores).
| Sparsity | : PPL w/o → w/ ReFiT | : PPL w/o → w/ ReFiT | : PPL w/o → w/ ReFiT |
|---|---|---|---|
| 50% | 5.514 → 5.510 | 6.320 → 6.129 | 6.603 → 6.301 |
| 70% | 5.715 → 5.643 | 7.846 → 6.963 | 9.039 → 7.510 |
| 90% | 7.795 → 6.471 | 26.224 → 10.739 | 39.010 → 15.936 |
Table R7. Token-wise and block-shared sparsity on Llama2-7B. Dense PPL = 5.4735. is causal; block-shared actual-activation selection yields diagnostic PPL.
- ReFiT improves PPL even at . The absolute gain grows with block size, while the fraction recovered is comparable. At 90% sparsity, the PPL reduction is 1.32, 15.49, and 23.07 for , 32, and 128, and ReFiT closes more than half of the PPL gap to dense in all three cases (57%, 75%, 69%). Block sharing therefore leaves more error to repair, and ReFiT repairs a similar share of it.
We thank the reviewer for the detailed feedback. We clarify the original-weight anchor, report actual prefill measurements and expanded downstream results, and test the roles of token-dependent correction and block sharing. Sparse means the model without ReFiT.
Thanks a lot for the constructive comment; this comparison helped us clarify our contribution. The original-weight anchor (W0) makes reconstruction preserve pretrained information when dynamic-mask calibration provides little evidence for a neuron.
GRAIL (arXiv:2602.23795) reconstructs compressed activations and absorbs the reconstruction into consumer weights. Its equivalent consumer-weight objective uses zero-centered ridge. Writing Y for dense target outputs and Z for the consumer's retained inputs, the two regularizers are:
\text{GRAIL-form:}\quad &\min_W\|Y-WZ\|_F^2+\mu\|W\|_F^2,\\\\ \text{ReFiT:}\quad &\min_W\|Y-WZ\|_F^2+\mu\|W-W_0\|_F^2. \end{aligned}$$ Here $W_0$ is the original down-projection. ReFiT uses full-width, zero-filled $Z$ from input-dependent block masks. Let $G=ZZ^\top$, $C=YZ^\top$, and $\mu=\mu_{\mathrm{rel}}\operatorname{mean}(\operatorname{diag}G)$. The normal equations give: $$\widehat W_{\mathrm{GRAIL}}=C(G+\mu I)^{-1},\qquad \widehat W_{\mathrm{ReFiT}}=(C+\mu W_0)(G+\mu I)^{-1}.$$ Equivalently, with $E=Y-W_0Z$, $$\widehat W_{\mathrm{ReFiT}}=W_0+\underbrace{EZ^\top(G+\mu I)^{-1}}_{\text{regularized residual correction}}.$$ The anchor regularizes the correction to pretrained weights. We collect $G,C$ on the original sparse model with a frozen predictor, solve once per layer, and fold the update into its down-projection. This distinction has measurable consequences. The kept set varies across token blocks, so one matrix must serve a distribution of masks. In calibration, **0.1%, 2.3%, and 29%** of FFN feature columns are never observed with nonzero sparse activations at 50%, 70%, and 90% sparsity. For such a column $j$, $Ge_j=0$ and $Ce_j=0$, giving $$\widehat W_{\mathrm{GRAIL }}e_j=0,\qquad \widehat W_{\mathrm{ReFiT }}e_j=W_0e_j.$$ The anchor therefore preserves that neuron's original contribution if it is selected on a future input. Static pruning uses the same retained coordinate set for every sample; dynamic block masks introduce the additional possibility of retaining a previously unobserved coordinate. We isolate this effect by comparing both anchors at ReFiT's default strength and strengths in GRAIL's reported range, using identical calibration statistics within each operating point. | Relative ridge strength | 50%: ReFiT / GRAIL | 70%: ReFiT / GRAIL | 90%: ReFiT / GRAIL | |---|---:|---:|---:| | 1 (ReFiT setting) | **8.053** / 27.480 | **14.626** / 45.171 | **82.14** / 199.88 | | 0.1 | **8.093** / 11.608 | **14.466** / 20.830 | **81.11** / 118.53 | | 5e-3 (GRAIL range*) | **8.180** / 9.172 | **14.630** / 16.448 | **88.17** / 96.30 | | 1e-3 (GRAIL range*) | **8.164** / 8.556 | **14.571** / 15.371 | **97.74** / 100.99 | | w/o ReFiT | 8.913 | 20.814 | 615.90 | **Table R1. Effect of the original-weight anchor, diagnostic PPL ↓:** Llama2-7B, block size 128. This controlled anchor ablation uses our dynamic masks; every variant updates only the down-projections. *These strengths lie within GRAIL's reported range of $10^{-4}$–$5\times10^{-3}$. - **ReFit's original-weight anchor wins all 12 matched comparisons**, including weak regularization in GRAIL's range. Our unique contribution is **formulation and demonstrated importance of anchor for a shared projection serving dynamic masks, including neurons unobserved during calibration.** - We will also discuss the reviewer's other references: LIAR (arXiv:2407.13331) reconstructs structurally pruned models by linear interpolation of preserved weights; RestoreLCC (arXiv:2510.21834) identifies and reinjects lost attention components. **ReFiT targets dynamically omitted FFN activations with an anchored down-projection fit.**