acceptodds
Under review as a conference paper at ICLR 2027

ReFiT: Recovering Block-Sparse FFN Quality at No Extra Inference Cost

Abstract

Long-context tasks require processing many input tokens before generation, making efficient LLM prefill increasingly important. Sharing FFN neuron sets across tokens preserves efficient batched computation, but limits adaptation to each token and degrades quality. Individual-neuron energy scores omit interactions between discarded contributions, while a block-constant correction cannot recover token-dependent errors. We propose ReFiT, a closed-form refit of the FFN down-projection weights to reconstruct missing output from surviving activations under a fixed selection policy. The update performs ridge regression of the discarded output on retained activations, regularized toward the original weights. These activations remain token-specific under a shared mask, allowing one fitted down-projection matrix to produce different corrections for different tokens.

Primary area:
generative models
Keywords:
contextual sparsity, efficient LLM inference, FFN sparsity, block-shared masks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vZ9COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReFiT, a post-hoc reconstruction method for recovering the quality degradation caused by block-shared FFN activation sparsity in LLMs. The method uses calibration data to fit the existing down-projection weights through anchored ridge regression, reconstructing the missing FFN outputs from retained activations. The resulting correction is absorbed into the original projection weights, requiring no additional matrix multiplication during inference. Experiments on LLaMA2-7B, LLaMA3.2-1B/3B, and Qwen3-4B demonstrate improvements in WikiText-2 perplexity and LongBench performance, particularly under high sparsity.

Critical strengths
  1. Simple and practical approach. ReFiT requires neither gradient-based retraining nor additional inference-time compensation branches. Its closed-form formulation makes it straightforward to apply to existing sparse FFN implementations.
  2. Consistent empirical improvements. The method demonstrates substantial quality recovery across multiple models, sparsity ratios, and block sizes. The improvements at high sparsity are particularly encouraging.
  3. Useful error analysis. The decomposition of reconstruction errors into block-mean and within-block components provides insights into the limitations of block-shared activation sparsity. The authors also include ablations to investigate token-dependent corrections.
Critical weaknesses
  1. Limited methodological novelty and insufficient discussion of closely related work. The core method is essentially an anchored ridge regression that reconstructs missing outputs from retained activations, followed by folding the correction into existing weights. Similar principles have been explored in previous post-hoc reconstruction and compensation methods.

In particular, the paper does not discuss several closely related works:

GRAIL is especially relevant, as it also uses calibration-based linear reconstruction and absorbs the resulting compensation into existing consumer weights without additional inference operations. Although these methods address different compression settings, the conceptual and mathematical overlap deserves a more thorough discussion. The authors should clarify what methodological contribution ReFiT introduces beyond applying existing linear reconstruction techniques to block-shared activation sparsity.

  1. Missing evaluation of actual prefill acceleration. The primary motivation of the paper is efficient LLM prefill. However, the experiments focus almost exclusively on quality recovery, without reporting actual prefill latency, time-to-first-token (TTFT), or throughput. Although ReFiT does not introduce additional matrix multiplications, this alone does not establish the end-to-end efficiency of the resulting inference system. In particular, the practical implications of sparse kernel execution, dynamic neuron selection, and maintaining different fitted weights for different sparsity configurations remain unclear. A measured latency–quality trade-off would substantially strengthen the practical contribution.

  2. Limited evaluation on standard zero-shot benchmarks. The evaluation primarily relies on WikiText-2 perplexity and LongBench. Standard zero-shot benchmarks are missing. Furthermore, calibration and perplexity evaluation both use WikiText-2, although with separate training and test splits. Cross-dataset evaluation would provide stronger evidence of generalization.

  3. Token-dependent recovery appears relatively limited. Although recovering token-dependent errors is a central motivation, the reported within-block error reduction is approximately 15% in the representative setting. Moreover, the appendix indicates that much of this improvement is concentrated in the final layer, with substantially smaller recovery after excluding it. This raises questions about how broadly the proposed mechanism recovers token-specific information across the network.

Questions for authors
  1. How does ReFiT differ mathematically and conceptually from existing post-hoc linear reconstruction methods, particularly GRAIL? Could the authors clarify the specific novelty beyond the application to block-shared activation sparsity?
  2. Can the authors provide actual prefill latency or TTFT measurements, ideally together with a quality–latency Pareto comparison?
  3. Can the authors evaluate ReFiT on standard zero-shot benchmarks, such as HellaSwag, PIQA, ARC, and WinoGrande?
  4. How does ReFiT perform when (T=1)? Is the relative reconstruction gain larger under block-shared sparsity than under token-wise sparsity?
  5. Why is within-block error recovery relatively limited and concentrated in the final layer? How much does this mechanism contribute to the observed end-to-end quality improvements?
4 comments

We thank the reviewer for the detailed feedback. We clarify the original-weight anchor, report actual prefill measurements and expanded downstream results, and test the roles of token-dependent correction and block sharing. Sparse means the model without ReFiT.


W1 & Q1. Relation to GRAIL and the methodological contribution.


Thanks a lot for the constructive comment; this comparison helped us clarify our contribution. The original-weight anchor (W0W_0) makes reconstruction preserve pretrained information when dynamic-mask calibration provides little evidence for a neuron.

GRAIL (arXiv:2602.23795) reconstructs compressed activations and absorbs the reconstruction into consumer weights. Its equivalent consumer-weight objective uses zero-centered ridge. Writing YY for dense target outputs and ZZ for the consumer's retained inputs, the two regularizers are:

\text{GRAIL-form:}\quad &\min_W\|Y-WZ\|_F^2+\mu\|W\|_F^2,\\\\ \text{ReFiT:}\quad &\min_W\|Y-WZ\|_F^2+\mu\|W-W_0\|_F^2. \end{aligned}$$ Here $W_0$ is the original down-projection. ReFiT uses full-width, zero-filled $Z$ from input-dependent block masks. Let $G=ZZ^\top$, $C=YZ^\top$, and $\mu=\mu_{\mathrm{rel}}\operatorname{mean}(\operatorname{diag}G)$. The normal equations give: $$\widehat W_{\mathrm{GRAIL}}=C(G+\mu I)^{-1},\qquad \widehat W_{\mathrm{ReFiT}}=(C+\mu W_0)(G+\mu I)^{-1}.$$ Equivalently, with $E=Y-W_0Z$, $$\widehat W_{\mathrm{ReFiT}}=W_0+\underbrace{EZ^\top(G+\mu I)^{-1}}_{\text{regularized residual correction}}.$$ The anchor regularizes the correction to pretrained weights. We collect $G,C$ on the original sparse model with a frozen predictor, solve once per layer, and fold the update into its down-projection. This distinction has measurable consequences. The kept set varies across token blocks, so one matrix must serve a distribution of masks. In calibration, **0.1%, 2.3%, and 29%** of FFN feature columns are never observed with nonzero sparse activations at 50%, 70%, and 90% sparsity. For such a column $j$, $Ge_j=0$ and $Ce_j=0$, giving $$\widehat W_{\mathrm{GRAIL }}e_j=0,\qquad \widehat W_{\mathrm{ReFiT }}e_j=W_0e_j.$$ The anchor therefore preserves that neuron's original contribution if it is selected on a future input. Static pruning uses the same retained coordinate set for every sample; dynamic block masks introduce the additional possibility of retaining a previously unobserved coordinate. We isolate this effect by comparing both anchors at ReFiT's default strength and strengths in GRAIL's reported range, using identical calibration statistics within each operating point. | Relative ridge strength | 50%: ReFiT / GRAIL | 70%: ReFiT / GRAIL | 90%: ReFiT / GRAIL | |---|---:|---:|---:| | 1 (ReFiT setting) | **8.053** / 27.480 | **14.626** / 45.171 | **82.14** / 199.88 | | 0.1 | **8.093** / 11.608 | **14.466** / 20.830 | **81.11** / 118.53 | | 5e-3 (GRAIL range*) | **8.180** / 9.172 | **14.630** / 16.448 | **88.17** / 96.30 | | 1e-3 (GRAIL range*) | **8.164** / 8.556 | **14.571** / 15.371 | **97.74** / 100.99 | | w/o ReFiT | 8.913 | 20.814 | 615.90 | **Table R1. Effect of the original-weight anchor, diagnostic PPL ↓:** Llama2-7B, block size 128. This controlled anchor ablation uses our dynamic masks; every variant updates only the down-projections. *These strengths lie within GRAIL's reported range of $10^{-4}$–$5\times10^{-3}$. - **ReFit's original-weight anchor wins all 12 matched comparisons**, including weak regularization in GRAIL's range. Our unique contribution is **formulation and demonstrated importance of anchor for a shared projection serving dynamic masks, including neurons unobserved during calibration.** - We will also discuss the reviewer's other references: LIAR (arXiv:2407.13331) reconstructs structurally pruned models by linear interpolation of preserved weights; RestoreLCC (arXiv:2510.21834) identifies and reinjects lost attention components. **ReFiT targets dynamically omitted FFN activations with an anchored down-projection fit.**

W2 & Q2. Measured prefill acceleration and hardware implications.


We benchmarked a block-sparse Triton kernel, including dynamic predictor and top-k selection, over block sizes 8, 32, and 128.

Block sizeSparsitySparse w/o ReFiT, msSparse w/ ReFiT, msSpeedup vs dense, w/o / w/ReFiT / Sparse
850%1095.90—0.31× / ——
870%528.57—0.64× / ——
890%290.98—1.16× / ——
3250%422.38—0.80× / ——
3270%301.28301.201.12× / 1.12×0.9997×
3290%226.41—1.49× / ——
12850%281.86281.981.20× / 1.20×1.0004×
12870%251.31251.891.35× / 1.34×1.0023×
12890%214.77215.731.57× / 1.56×1.0045×

Table R2. Llama2-7B model-prefill latency: A100-SXM4-40GB, fp16, batch 1, 3,500 tokens. Medians use five repeats for block size 8 and ten otherwise. Speedups use the paired dense measurement for each arm; dashes denote unmeasured ReFiT configurations.

  • Larger blocks and higher sparsity improve execution efficiency over this grid. At T=128T=128, ReFiT-Sparse is 1.20–1.56× faster than dense; smaller blocks require higher sparsity to offset overhead. Across 12 matched Sparse/ReFiT-Sparse comparisons, including shorter prompts, the latency ratio is 0.999–1.009 (median 1.002).
  • Dynamic selection, irregular weight access, limited reuse at small blocks, and unchanged attention/dense computation limit speedup relative to the FFN arithmetic reduction. Sparse accelerators such as SLIM (arXiv:2507.09201) fetch only active neurons, motivating our complementary objective: recover accuracy at fixed sparsity without extra online correction operations.
  • For Llama2-7B, calibration takes approximately 30 minutes on an A100 (4 minutes for 16 sequences), plus a 20-second solve, under shared GPU load. Fits are operating-point-specific; a dynamic server can precompute supported configurations. Each 16-bit down-projection copy occupies about 2.9 GB temporarily; but it is then folded to WdW_d during inference, removing the need for a separate correction matrix.

 


W3 & Q3. Zero-shot evaluation and prefill-heavy downstream tasks.


To test whether gains extend beyond severely degraded models, we added (1) zero-shot reasoning and MMLU, and (2) prefill-heavy classification evaluations across several models. All use WikiText-2-calibrated weights without target-task fitting.

(1) zero-shot reasoning and MMLU

EvaluationModelBlock / sparsityDenseSparse+ ReFiT
Zero-shot mean (3 tasks)Llama2-7B8 / 50%58.5552.2353.08
Zero-shot mean (3 tasks)Llama2-7B8 / 70%58.5543.8146.06
Zero-shot mean (3 tasks)Llama2-7B8 / 90%58.5532.3434.59
Zero-shot mean (3 tasks)Llama2-7B32 / 50%58.5552.1052.75
Zero-shot mean (3 tasks)Llama2-7B32 / 70%58.5543.8445.92
Zero-shot mean (3 tasks)Llama2-7B32 / 90%58.5532.8334.71
MMLULlama-3.2-1B-Instruct128 / 90%46.1344.3745.58
MMLULlama-3.2-3B-Instruct128 / 90%60.7257.7859.86

Table R3. Additional downstream accuracy (%). The zero-shot mean equally weights HellaSwag, ARC-Easy, and ARC-Challenge on their full evaluation sets (10,042/2,376/1,172 examples), calculated before rounding. MMLU is 5-shot with 14,042 questions. Zero-shot contexts have no dense prefix after BOS; MMLU retains the first/last 128 prompt tokens as dense. Answer continuations remain dense.

  • ReFiT improves the three-task zero-shot mean in all six configurations, including 50% sparsity. On MMLU, the sparse models remain close to dense accuracy, and ReFiT recovers much of the remaining loss for both Llama-3.2 models.

 

(2) prefill-heavy classification

BenchmarkModelDenseSparse+ ReFiT
Banking77Llama2-7B70.6035.6752.73
Banking77Llama-3.2-1B-Instruct61.3340.0045.40
Banking77Llama-3.2-3B-Instruct65.8044.3354.67
Banking77Qwen3-4B74.7341.1348.93
CLINC150Llama2-7B82.6065.6070.20
CLINC150Llama-3.2-1B-Instruct73.8059.6761.47
CLINC150Llama-3.2-3B-Instruct80.9365.3368.80
CLINC150Qwen3-4B85.8068.4771.80

Table R4. Prefill-heavy classification accuracy (%), block size 128, 70% sparsity. Prompts contain one demonstration per intent (77/150), with the first/last 128 prompt tokens and answer continuations dense. Confirmation sets are disjoint from exploratory selection: 1,500 examples per cell, except Llama2-7B CLINC150 (500).

  • Both prefill-heavy tasks improve across all four models. Llama2-7B Banking77 uses roughly 1,827-token prompts, with 84% processed sparsely. At 50% sparsity, separate Banking77 confirmation runs also gain 5.0/4.1/5.5 points on Llama2-7B/Llama-3.2-3B/Qwen3-4B.

 

We also changed the calibration corpus while fixing WikiText-2 test evaluation:

CalibrationPPL, 50% ↓PPL, 70% ↓PPL, 90% ↓
No ReFiT8.91320.814615.90
WikiText-28.05314.62682.14
C48.28515.52982.26
MiniPile8.33315.82091.69

Table R5. Calibration transfer: Llama2-7B, block size 128, 128 sequences, common A100 setup.

  • C4 and MiniPile improve PPL at all three sparsities: at 90%, the two fits reduce PPL from 615.90 to 82.26 and 91.69. Thus, substantial recovery transfers from C4 and MiniPile calibration to WikiText-2 evaluation.

 


W4 & Q5. Contribution of token-dependent correction and multiple layers.


We isolate the contribution of token-dependent correction by retaining only the block-mean component of ReFiT, and test the importance of applying ReFiT across layers by restricting it to the final layer.

CorrectionPPL, 50% ↓PPL, 70% ↓PPL, 90% ↓
None8.91320.814615.90
Calibration-mean replacement8.57720.737531.03
Block-mean component8.26915.958133.27
Final-layer-only8.73820.520522.80
Full ReFiT8.05314.62682.14

Table R6. Llama2-7B correction ablation, diagnostic PPL ↓, block size 128. The block-mean arm averages the input-dependent ReFiT correction within each block; all arms use the same A100 evaluation setup.

  • Full ReFiT gives the best PPL at every tested sparsity. The block-mean component already helps substantially, and retaining token-dependent correction improves it further. Calibration-mean replacement, W0[(1−mB)⊙aˉcal]W_0[(1-m_B)\odot\bar a_{\mathrm{cal}}], uses calibration-average activations and provides less recovery than correction based on the current block's retained activations. Final-layer-only refitting recovers much less PPL than full ReFiT.

 


Q4. Token-wise sparsity (T=1T=1) versus block sharing.


We added a T=1T=1 control and compare it with T=32/128T=32/128 using actual-activation selection (dense activations supply exact neuron scores).

SparsityT=1T=1: PPL w/o → w/ ReFiTT=32T=32: PPL w/o → w/ ReFiTT=128T=128: PPL w/o → w/ ReFiT
50%5.514 → 5.5106.320 → 6.1296.603 → 6.301
70%5.715 → 5.6437.846 → 6.9639.039 → 7.510
90%7.795 → 6.47126.224 → 10.73939.010 → 15.936

Table R7. Token-wise and block-shared sparsity on Llama2-7B. Dense PPL = 5.4735. T=1T=1 is causal; block-shared actual-activation selection yields diagnostic PPL.

  • ReFiT improves PPL even at T=1T=1. The absolute gain grows with block size, while the fraction recovered is comparable. At 90% sparsity, the PPL reduction is 1.32, 15.49, and 23.07 for T=1T=1, 32, and 128, and ReFiT closes more than half of the PPL gap to dense in all three cases (57%, 75%, 69%). Block sharing therefore leaves more error to repair, and ReFiT repairs a similar share of it.
Reviewer_LogsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses quality loss in block-shared FFN activation sparsity for LLM prefill, where one neuron mask is shared across a token block. It studies two settings: FastForward-style intermediate sparsity and LaRoSA-style input sparsity. The proposed method refits the down-projection by ridge regression, adding no inference-time compensation. The authors motivate it with exact error decompositions and local reconstruction controls. Experiments show improved PPL and LongBench scores across four models.

Critical strengths

Simple, closed-form compensation with no inference overhead. The fit is cheap, folds into the down-projection, and yields large gains at high sparsity, which makes it practical.

Critical weaknesses

The central claim (token-dependent correction) is weakly supported. Within-block recovery is only ~15%, mostly from the final layer, and ReFiT does not beat the per-block best constant at T=16. Most of the gain may come from correcting a mean offset, so the novelty is limited unless the token-dependent component is shown to matter.

Baselines are limited. ReFiT should be compared with alternatives such as a fitted bias vector, mean compensation, and low-rank compensation. If the authors claim complementarity, combination experiments should be reported.

Large gains appear only where the model remains unusable (s=0.9, PPL 65–90), while at s=0.5 the LongBench gain is negligible (+0.05).

Since the motivation is efficient prefill, latency and throughput measurements are necessary.

Questions for authors
  1. Calibration uses WikiText-2 train and PPL is evaluated on WikiText-2 test. Do the gains hold when calibrating on out-of-domain data (e.g., C4)?

  2. How much of the LongBench gain remains if only the final layer is refit?

3 comments

We thank the reviewer for the constructive assessment. We address the four weakness paragraphs as W1–W4, with new ablations, compensation combinations, downstream results, and prefill measurements. Sparse means the model without ReFiT.

 


W1, W2 (mean/bias baselines) & Q2. What drives recovery, and is it concentrated in the final layer?


To distinguish mean compensation, token-dependent correction, and layer placement, we compare calibration-mean replacement, the block-mean component of ReFiT, final-layer-only refitting, and full ReFiT.

CorrectionPPL, 50% ↓PPL, 70% ↓PPL, 90% ↓
None8.91320.814615.90
Calibration-mean replacement8.57720.737531.03
Block-mean component8.26915.958133.27
Final-layer-only8.73820.520522.80
Full ReFiT8.05314.62682.14

Table R1. Llama2-7B correction ablation, diagnostic PPL ↓, block size 128. The block-mean arm averages the input-dependent ReFiT correction within each block; all arms use the same A100 evaluation setup.

  • Full ReFiT gives the best PPL at every tested sparsity. The block-mean component already helps substantially, and retaining token-dependent correction improves it further. Calibration-mean replacement, W0[(1−mB)⊙aˉcal]W_0[(1-m_B)\odot\bar a_{\mathrm{cal}}], uses calibration-average activations and provides less recovery than correction based on the current block's retained activations. Final-layer-only refitting recovers much less PPL than full ReFiT.
  • In the case of LongBench, final-layer-only refitting retains none of the gain: at block size 128 and 70% sparsity, the score is 23.74 without ReFiT, 23.74 with final-layer-only refitting, and 25.87 with full ReFiT. The final-layer FFN output at a sparse prefill position feeds only that position's logits, which are discarded, and it does not enter the KV cache that generation reads; accordingly, all 750 generations we checked (four tasks) are identical to those without ReFiT.

 


W2 (low-rank compensation). Can ReFiT complement an auxiliary branch?


The low-rank branch estimates dropped-neuron outputs using approximations of the gate, up-, and down-projections. Higher rank trades extra computation for better accuracy. To combine it with ReFiT, we fit the down-projection against dense output minus branch output.

Actual-activation selection is an oracle predictor for neuron scores computed from dense activations, used here as a diagnostic reference; the predictor estimates those scores without evaluating all neurons.

Branch rankActual-activation: BranchBranch + ReFiTPredictor: BranchBranch + ReFiT
34415.7513.81 (−12%)125.6676.98 (−39%)
68812.6511.45 (−9%)76.6160.14 (−21%)
13768.597.87 (−8%)22.4321.40 (−5%)

Table R2. Branch versus branch + ReFiT: Llama2-7B, block size 128, 90% sparsity, diagnostic PPL ↓ under the archived branch-composition protocol. Parentheses show relative PPL reductions. Rank-344/1376 fits share calibration states from the rank-1376 cascade; rank 688 uses its own cascade.

  • ReFiT lowers PPL at all three ranks under both selection policies. The branch incurs rank-dependent computation; ReFiT is folded into WdW_d and adds no further inference operations.

 


W3. Are gains useful beyond severely degraded models?


To test whether gains extend beyond severely degraded models, we added (1) zero-shot reasoning and MMLU, and (2) prefill-heavy classification evaluations across several models. All use WikiText-2-calibrated weights without target-task fitting.

(1) zero-shot reasoning and MMLU

EvaluationModelBlock / sparsityDenseSparse+ ReFiT
Zero-shot mean (3 tasks)Llama2-7B8 / 50%58.5552.2353.08
Zero-shot mean (3 tasks)Llama2-7B8 / 70%58.5543.8146.06
Zero-shot mean (3 tasks)Llama2-7B8 / 90%58.5532.3434.59
Zero-shot mean (3 tasks)Llama2-7B32 / 50%58.5552.1052.75
Zero-shot mean (3 tasks)Llama2-7B32 / 70%58.5543.8445.92
Zero-shot mean (3 tasks)Llama2-7B32 / 90%58.5532.8334.71
MMLULlama-3.2-1B-Instruct128 / 90%46.1344.3745.58
MMLULlama-3.2-3B-Instruct128 / 90%60.7257.7859.86

Table R3. Additional downstream accuracy (%). The zero-shot mean equally weights HellaSwag, ARC-Easy, and ARC-Challenge on their full evaluation sets (10,042/2,376/1,172 examples), calculated before rounding. MMLU is 5-shot with 14,042 questions. Zero-shot contexts have no dense prefix after BOS; MMLU retains the first/last 128 prompt tokens as dense. Answer continuations remain dense.

  • ReFiT improves the three-task zero-shot mean in all six configurations, including 50% sparsity. On MMLU, the sparse models remain close to dense accuracy, and ReFiT recovers much of the remaining loss for both Llama-3.2 models.

 

(2) prefill-heavy classification

BenchmarkModelDenseSparse+ ReFiT
Banking77Llama2-7B70.6035.6752.73
Banking77Llama-3.2-1B-Instruct61.3340.0045.40
Banking77Llama-3.2-3B-Instruct65.8044.3354.67
Banking77Qwen3-4B74.7341.1348.93
CLINC150Llama2-7B82.6065.6070.20
CLINC150Llama-3.2-1B-Instruct73.8059.6761.47
CLINC150Llama-3.2-3B-Instruct80.9365.3368.80
CLINC150Qwen3-4B85.8068.4771.80

Table R4. Prefill-heavy classification accuracy (%), block size 128, 70% sparsity. Prompts contain one demonstration per intent (77/150), with the first/last 128 prompt tokens and answer continuations dense. Confirmation sets are disjoint from exploratory selection: 1,500 examples per cell, except Llama2-7B CLINC150 (500).

  • Both prefill-heavy tasks improve across all four models. Llama2-7B Banking77 uses roughly 1,827-token prompts, with 84% processed sparsely. At 50% sparsity, separate Banking77 confirmation runs also gain 5.0/4.1/5.5 points on Llama2-7B/Llama-3.2-3B/Qwen3-4B. Thus, the small cited LongBench gain does not characterize all moderate-sparsity tasks.

 


W4. Actual prefill efficiency.


We measured original and refitted weights using the same block-sparse Triton kernel, including predictor and top-k costs. ReFiT preserves tensor shapes and layout and adds no correction operator or ReFiT-specific kernel tuning.

Block sizeSparsitySparse w/o ReFiT, msSparse w/ ReFiT, msSpeedup vs dense, w/o / w/ReFiT / Sparse
850%1095.90—0.31× / ——
870%528.57—0.64× / ——
890%290.98—1.16× / ——
3250%422.38—0.80× / ——
3270%301.28301.201.12× / 1.12×0.9997×
3290%226.41—1.49× / ——
12850%281.86281.981.20× / 1.20×1.0004×
12870%251.31251.891.35× / 1.34×1.0023×
12890%214.77215.731.57× / 1.56×1.0045×

Table R5. Llama2-7B model-prefill latency: A100-SXM4-40GB, fp16, batch 1, 3,500 tokens. Medians use five repeats for block size 8 and ten otherwise. Speedups use the paired dense measurement for each arm; dashes denote unmeasured ReFiT configurations.

  • Larger blocks and higher sparsity improve execution efficiency over this grid. At T=128T=128, ReFiT-Sparse is 1.20–1.56× faster than dense; smaller blocks require higher sparsity to offset overhead. Across 12 matched Sparse/ReFiT-Sparse comparisons, including shorter prompts, the latency ratio is 0.999–1.009 (median 1.002).
  • Dynamic selection, irregular weight access, limited reuse at small blocks, and unchanged attention/dense computation limit speedup relative to the FFN arithmetic reduction. Sparse accelerators such as SLIM (arXiv:2507.09201) fetch only active neurons, motivating our complementary objective: recover accuracy at fixed sparsity without extra online correction operations.

 


Q1. Out-of-domain calibration.


Recovery persists with out-of-domain calibration. We calibrated on C4 and MiniPile while fixing evaluation to WikiText-2 test, then varied WikiText-2 seeds and sample counts to assess whether recovery depends on a particular calibration sample.

CalibrationSeedSequencesPPL, 50% ↓PPL, 70% ↓PPL, 90% ↓
No ReFiT——8.91320.814615.90
WikiText-2421288.05314.62682.14
WikiText-211288.06314.66990.33
WikiText-221288.05014.61890.27
C4421288.28515.52982.26
MiniPile421288.33315.82091.69
WikiText-24232—14.76882.76
WikiText-24216—14.92980.25

Table R6. Calibration transfer and sample efficiency: Llama2-7B, block size 128, diagnostic PPL on a common A100 setup. Each sequence contains 2,048 tokens; dashes denote unmeasured settings.

  • (Calibration Data) Both C4 and MiniPile substantially improve PPL at every tested sparsity. (Seed) WikiText-2 seed changes have little effect at 50% and 70% sparsity. Even at 90%, all tested calibration choices reduce PPL from 615.90 to approximately 80–92. (Calibration sample count) Using only 16 sequences gives results close to the 128-sequence fit, supporting practical calibration with limited data. The downstream gains in Tables R3–R4 further demonstrate transfer from WikiText-2 calibration to different tasks without target-task fitting.
Reviewer_tTPDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the quality degradation caused by block-shared activation sparsity in LLM FFN layers during prefill acceleration. When a shared neuron support is used across a block of tokens (as in FastForward or LaRoSA), tokens with different activation patterns are forced to use the same channels, leading to discarded output that varies per token. ReFiT proposes a closed-form refit of the FFN down-projection weights that regresses the discarded output (tail) on the retained sparse activations via ridge regression, regularized toward the original weights. The key insight is that although the support mask is shared, the activation values z_t vary across tokens, so a single fitted matrix can produce token-dependent corrections. The update folds into the existing down-projection matrix, adding no auxiliary inference branch or matrix multiplication. The paper provides an exact mean/within-block error decomposition showing that within-block variation accounts for 69-91% of local error, and that cross-term interactions (8-24% of error) are missed by individual-energy-based approaches. Experiments across four models (LLaMA2-7B, LLaMA3-8B, Mistral-7B, Qwen1.5-7B) and 24 operating points show that ReFiT improves diagnostic perplexity at all operating points, and on LLaMA2-7B at 90% sparsity with 128-token blocks, LongBench improves from 12.90 to 21.11, recovering 51.8% of the gap to dense quality.

Critical strengths

The core idea is elegant and practically appealing: by recognizing that retained activation values carry token-specific information even under a shared mask, ReFiT can produce token-dependent corrections through a single matrix that replaces the existing down-projection — zero added inference cost. The error decomposition (Section 3.3) is a valuable diagnostic contribution, cleanly separating mean-block and within-block components and demonstrating that cross-term interactions matter. The ridge regression formulation with anchoring to original weights (Eq. 8-9) is well-justified and has a clean closed-form solution. The controlled experimental methodology is commendable: predictor parameters are frozen before and after ReFiT, paper-reference implementation checks precede the comparisons, and the shuffle control (Table 2) cleverly demonstrates that token-specific alignment, not just average correction, drives the benefit. The evaluation spans two sparsity families (FastForward-based and LaRoSA-based), multiple block sizes, sparsities, and four model families, providing broad coverage.

Critical weaknesses

(1) The quality recovery is incomplete. Even after ReFiT, at 90% sparsity with 128-token blocks, the LongBench score of 21.11 remains far from the dense model's performance. The paper frames this as "recovering 51.8% of the gap," but the remaining 48.2% gap is substantial, particularly at the high sparsities where acceleration matters most. The paper does not discuss whether this residual gap is fundamental or could be closed by complementary techniques. (2) ReFiT requires calibration data to fit the down-projection matrix (Eq. 9 involves sums over calibration blocks). While the paper fixes the predictor and only fits the down-projection, this still means the method is not calibration-free, and the fitted matrix is specific to the operating point (block size, sparsity, model). Changing any of these requires re-fitting, which could limit deployability in dynamic serving settings. (3) The evaluation is primarily on diagnostic perplexity (WikiText-2) with LongBench as the only downstream task. Perplexity is a sensitive but imperfect proxy for generation quality, and the LongBench evaluation is limited to one model (LLaMA2-7B). Showing results on more diverse downstream tasks and models would strengthen the claims. (4) The method's interaction with the upstream sparsity predictor is not fully explored. The paper fixes the predictor before fitting ReFiT, but in practice, a better predictor would change the activation patterns and potentially the effectiveness of ReFiT. Whether ReFiT and predictor improvement are complementary or conflicting is not addressed. (5) The paper claims "no extra inference cost," but the fitted down-projection matrix differs from the original. If the sparse computation path uses specialized kernels optimized for the original weight layout, deploying a different matrix may require kernel re-tuning or could affect numerical stability in ways not captured by perplexity. (6) The input-sparsity results (Section 5, "16 of 18 tested input-sparsity pairs") are less comprehensive than the intermediate-sparsity results, and the two negative cases are not analyzed in depth. Understanding when and why ReFiT fails to help would be valuable.

Questions for authors

Q1. The LongBench evaluation is limited to a single model (LLaMA2-7B) at a single operating point (90% sparsity, 128-token blocks). Can you provide LongBench results for additional models (LLaMA3-8B, Mistral-7B, Qwen1.5-7B) and at least one other sparsity/block-size combination? Does the 51.8% gap-closure rate hold across settings? Q2. ReFiT requires calibration data to fit the down-projection matrix. How sensitive is the fitted matrix to the choice of calibration data? If the calibration distribution differs from the deployment distribution, how much does the recovery degrade? Have you measured the transfer gap? Q3. The residual 48.2% quality gap at the best operating point is substantial. Can you decompose this residual — how much is attributable to within-block variation that the linear refit cannot capture, how much to cross-term interactions, and how much to the fundamental information loss from sparsification? Is the gap closable with complementary techniques (e.g., lightweight auxiliary branches)? Q4. The two negative input-sparsity cases (2 of 18) are not analyzed. What are the specific configurations where ReFiT fails, and what is the underlying reason? Is there a detectable condition (e.g., activation pattern properties) that predicts when ReFiT will not help? Q5. The method folds the refit into the existing down-projection matrix, but the sparse computation path may use specialized kernels. Have you tested whether the modified matrix affects kernel performance (e.g., memory access patterns, numerical stability) in a real serving system? Is there any throughput difference beyond the perplexity results? Q6. The error decomposition shows cross-term interactions account for 8–24% of local error. How would an individual-energy-based approach need to be modified to account for these interactions? Is there a formulation that captures cross-terms without sacrificing the closed-form solution?

3 comments

We thank the reviewer for the constructive feedback. Below we combine expanded evaluations with direct answers on calibration, predictor interaction, residual error, and sparse-kernel execution. Sparse means the model without ReFiT.


W3 & Q1. Generalization across tasks, models, and operating points.


Our paper already evaluated LongBench at nine Llama2-7B operating points (T=8,32,128T=8,32,128; 50/70/90% sparsity) and on Llama-3.2-1B/3B-Instruct and Qwen3-4B at T=128T=128 and all three sparsities (Tables 3–4).

Within the rebuttal period, we additionally tested (1) zero-shot reasoning and MMLU, and (2) prefill-heavy classification, all with WikiText-2-calibrated weights and no target-task fitting.

(1) zero-shot reasoning and MMLU

EvaluationModelBlock / sparsityDenseSparse+ ReFiT
Zero-shot mean (3 tasks)Llama2-7B8 / 50%58.5552.2353.08
Zero-shot mean (3 tasks)Llama2-7B8 / 70%58.5543.8146.06
Zero-shot mean (3 tasks)Llama2-7B8 / 90%58.5532.3434.59
Zero-shot mean (3 tasks)Llama2-7B32 / 50%58.5552.1052.75
Zero-shot mean (3 tasks)Llama2-7B32 / 70%58.5543.8445.92
Zero-shot mean (3 tasks)Llama2-7B32 / 90%58.5532.8334.71
MMLULlama-3.2-1B-Instruct128 / 90%46.1344.3745.58
MMLULlama-3.2-3B-Instruct128 / 90%60.7257.7859.86

Table R1. Additional downstream accuracy (%). The zero-shot mean equally weights HellaSwag, ARC-Easy, and ARC-Challenge on their full evaluation sets (10,042/2,376/1,172 examples), calculated before rounding. MMLU is 5-shot with 14,042 questions. Zero-shot contexts have no dense prefix after BOS; MMLU retains the first/last 128 prompt tokens as dense. Answer continuations remain dense.

  • ReFiT improves the three-task zero-shot mean in all six configurations and improves MMLU accuracy for both Llama-3.2 models at 90% sparsity.

 

(2) prefill-heavy classification

BenchmarkModelDenseSparse+ ReFiT
Banking77Llama2-7B70.6035.6752.73
Banking77Llama-3.2-1B-Instruct61.3340.0045.40
Banking77Llama-3.2-3B-Instruct65.8044.3354.67
Banking77Qwen3-4B74.7341.1348.93
CLINC150Llama2-7B82.6065.6070.20
CLINC150Llama-3.2-1B-Instruct73.8059.6761.47
CLINC150Llama-3.2-3B-Instruct80.9365.3368.80
CLINC150Qwen3-4B85.8068.4771.80

Table R2. Prefill-heavy classification accuracy (%), block size 128, 70% sparsity. Prompts contain one demonstration per intent (77/150), with the first/last 128 prompt tokens and answer continuations dense. Confirmation sets are disjoint from exploratory selection: 1,500 examples per cell, except Llama2-7B CLINC150 (500).

  • Both prefill-heavy tasks improve across all four models. Llama2-7B Banking77 uses roughly 1,827-token prompts, with 84% processed sparsely. At 50% sparsity, separate Banking77 confirmation runs also gain 5.0/4.1/5.5 points on Llama2-7B/Llama-3.2-3B/Qwen3-4B.
  • The 51.8% LongBench gap closure is setting-specific: the submission spans approximately 6–71% recovery overall and 37–55% across the four models at T=128T=128, 70% sparsity.

 


W2 & Q2. Calibration sensitivity and deployment overhead.


Recovery persists across calibration corpora, sample counts, and seeds. Table R3 holds WikiText-2 test evaluation fixed.

CalibrationSeedSequencesPPL, 50% ↓PPL, 70% ↓PPL, 90% ↓
No ReFiT——8.91320.814615.90
WikiText-2421288.05314.62682.14
WikiText-211288.06314.66990.33
WikiText-221288.05014.61890.27
C4421288.28515.52982.26
MiniPile421288.33315.82091.69
WikiText-24232—14.76882.76
WikiText-24216—14.92980.25

Table R3. Calibration transfer and sample efficiency: Llama2-7B, block size 128, diagnostic PPL on a common A100 setup. Each sequence contains 2,048 tokens; dashes denote unmeasured settings.

  • (Seed) Seed changes barely affect PPL at 50/70% sparsity. (Calibration Data) Both C4 and MiniPile calibration also improve PPL at every tested sparsity.. (Calibration sample count) At 90%, all calibration choices reduce PPL from 615.90 to approximately 80–92; 16 sequences remain close to the 128-sequence fit.
  • For Llama2-7B, calibration takes approximately 30 minutes on an A100 (4 minutes for 16 sequences), plus a 20-second solve, under shared GPU load. Fits are operating-point-specific; a dynamic server can precompute supported configurations. Each 16-bit down-projection copy occupies about 2.9 GB temporarily; but it is then folded to WdW_d during inference, removing the need for a separate correction matrix.

 


W5 & Q5. Actual sparse-kernel performance.


We measured original and refitted weights using the same block-sparse Triton kernel, including predictor and top-k costs. ReFiT preserves tensor shapes and layout and adds no correction operator or ReFiT-specific kernel tuning.

Block sizeSparsitySparse w/o ReFiT, msSparse w/ ReFiT, msSpeedup vs dense, w/o / w/ReFiT / Sparse
850%1095.90—0.31× / ——
870%528.57—0.64× / ——
890%290.98—1.16× / ——
3250%422.38—0.80× / ——
3270%301.28301.201.12× / 1.12×0.9997×
3290%226.41—1.49× / ——
12850%281.86281.981.20× / 1.20×1.0004×
12870%251.31251.891.35× / 1.34×1.0023×
12890%214.77215.731.57× / 1.56×1.0045×

Table R4. Llama2-7B model-prefill latency: A100-SXM4-40GB, fp16, batch 1, 3,500 tokens. Medians use five repeats for block size 8 and ten otherwise. Speedups use the paired dense measurement for each arm; dashes denote unmeasured ReFiT configurations.

  • Larger blocks and higher sparsity improve execution efficiency over this grid. At T=128T=128, ReFiT-Sparse is 1.20–1.56× faster than dense; smaller blocks require higher sparsity to offset overhead. Across 12 matched Sparse/ReFiT-Sparse comparisons, including shorter prompts, the latency ratio is 0.999–1.009 (median 1.002).
  • Dynamic selection, irregular weight access, limited reuse at small blocks, and unchanged attention/dense computation limit speedup relative to the FFN arithmetic reduction. Sparse accelerators such as SLIM (arXiv:2507.09201) fetch only active neurons, motivating our complementary objective: recover accuracy at fixed sparsity without extra online correction operations.

 


W1, W4 & Q3. What limits the remaining gap, and the interaction with the predictor.


To test predictor interaction, we compare the learned predictor with actual-activation selection, an oracle predictor for neuron scores computed on-the-fly from dense activations. We use dense activations only as an oracle reference to isolate score-prediction error.

Selection policyReFiTPPL, 50% ↓PPL, 70% ↓PPL, 90% ↓
PredictorNo8.9120.81614.03
PredictorYes8.0514.6381.37
Actual-activation diagnosticNo6.609.0439.01
Actual-activation diagnosticYes6.307.5115.94

Table R5. Selection–refitting interaction: Llama2-7B, block size 128, WikiText-2 diagnostic PPL, using the submission's Tables 10–11. ReFiT is fitted separately for each policy. The submission's 90% predictor measurement used B200; Table R3 reports the A100 calibration rerun.

  • ReFiT improves both selection policies at every tested sparsity, including 39.01→15.94 PPL with actual-activation selection at 90%. Thus, improved selection and reconstruction are complementary. (The remaining gap may be further reduced with complementary methods, as our auxiliary-branch experiments suggest below. Quantifying how each source of error affects LongBench task performance requires analyzing their nonlinear interactions, which we are considering for future work.)

The low-rank branch estimates dropped-neuron outputs via low-rank projection approximations, trading compute for accuracy. We fit ReFiT on the remaining residual so their combined output matches the dense FFN.

Branch rankActual-activation: BranchBranch + ReFiTPredictor: BranchBranch + ReFiT
34415.7513.81 (−12%)125.6676.98 (−39%)
68812.6511.45 (−9%)76.6160.14 (−21%)
13768.597.87 (−8%)22.4321.40 (−5%)

Table R6. Branch versus branch + ReFiT: Llama2-7B, block size 128, 90% sparsity, diagnostic PPL ↓ under the archived branch-composition protocol. Parentheses show relative PPL reductions. Rank-344/1376 fits share calibration states from the rank-1376 cascade; rank 688 uses its own cascade.

  • ReFiT lowers PPL at all three ranks under both selection policies. The branch incurs rank-dependent computation; ReFiT is folded into WdW_d and adds no further inference operations.

 


W6 & Q4. Negative input-sparsity cases.


The two negative cases are UGD input sparsity at 50%, T=8/16T=8/16, with PPL increases of only 0.016/0.010. Keeping the same policy and block sizes, 70/90% sparsity produces clear improvements:

Block size50%: PPL w/o → w/ ReFiT70%: PPL w/o → w/ ReFiT90%: PPL w/o → w/ ReFiT
88.9761 → 8.991938.6390 → 35.47762556.80 → 1722.43
1611.1253 → 11.135760.1925 → 52.09843060.89 → 2762.99

Table R7. The two input-sparsity cases and their higher-sparsity counterparts: Llama2-7B, UGD input sparsity, diagnostic PPL ↓, down-only ReFiT with the same fixed recipe.

  • The gain is larger at higher sparsity in these matched configurations. Still, we conjecture that input sparsity perturbs the FFN before its nonlinearity, so a down-only linear fit need not improve every setting.

 


Q6. Cross-terms and the closed-form solution.


Thank you for this insightful question. Cross-terms can be included while retaining a closed-form solution for reconstruction, but not generally for selection.

Selection. With qi∈{0,1}q_i\in\lbrace 0,1\rbrace marking a dropped neuron whose contribution over a block is wiai⊤w_i a_i^\top, the dropped output energy is (Appendix C):

= q^\top Qq, \qquad Q_{ij}=(w_i^\top w_j)(a_i^\top a_j).$$ The individual-energy score uses only the diagonal of $Q$, for which keeping the top-$K$ neurons by sorting is optimal. Including off-diagonal terms makes selection a cardinality-constrained binary quadratic problem, requiring combinatorial optimization or approximate search. Appendix C tests a fixed neighborhood of 64 candidate swaps per block: a lower-error mask exists in all 8,192 sampled block–layer observations at each sparsity, but the best candidate reduces error by only 0.09% ($s=0.7$) and 0.12% ($s=0.9$) on average. *Reconstruction.* ReFiT keeps the mask and fits the full residual $R=Y-W_0Z$ on the surviving activations $Z$: $$\min_{\Delta W}\lVert R-\Delta WZ\rVert_F^2+\mu\lVert\Delta W\rVert_F^2 \quad\Longrightarrow\quad \Delta W=RZ^\top(ZZ^\top+\mu I)^{-1}.$$ This is Eq. 9 written for $\Delta W=\widetilde W-W_0$. For fixed layer inputs, $R$ is the sum above, so the objective contains the interactions among dropped neurons and the solution stays closed-form. It does not solve interaction-aware selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.