acceptodds
Under review as a conference paper at ICLR 2027

ReComp: Low-Rank Residual Compensation for Contextual Sparsity in Language Models

Abstract

Contextual sparsity, or dynamically computing a subset of neurons during inference, reduces multilayer perceptron (MLP) cost in Large Language Models (LLMs), but discards the output of the rest of the neurons, which is individually small but collectively substantial. We show that this output is predictable rather than lost, and evaluate whether part of a fixed compute budget is better spent on reconstructing it than computing more neurons. For a given token at layer , ReComp uses a low-rank neuron-importance predictor that selects the top- neurons to compute by importance. Then, a low-rank affine map estimates the combined output of the skipped neurons from the MLP input, fit via reduced-rank ridge regression (ridge-RRR) and distilled against the dense model with the objective having a KL-divergence term and an optional Jacobian-lens term. Across five models from 0.5B to 14B parameters, reconstruction beats spending the same amount of computation on more neurons at every budget we test with our selector. For our predictor, the compensator helps even before distillation. After distillation, it also improves other selectors; for example, when added to TEAL, it lowers relative perplexity on Qwen2.5-7B from 1.42 to 1.16 at the same arithmetic. At 25% of compressed-layer MLP arithmetic, ReComp lowers relative WikiText-2 perplexity on Qwen2.5-7B from 1.44 for the best of TEAL, WINA, and R-Sparse to 1.16, and beats all three on every 7-8B model tested.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models (LLMs), efficient inference, contextual sparsity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BnZyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper proposed to use a low rank affine map to compensate the lost info of contextual sparsity
  2. Different from existing approaches which neglects the discarded neurons, this work tries to compensate the lost info by reconstructing the discarded output.
  3. Relevant literatures has applied similar recompensation in the quantization literature to compensate quantization loss, which motivates this paper.
  4. Experimental results show that the proposed method outperforms other comparison methods in tight budget settings.
Critical strengths
  1. The paper proposed an interesting problem to study instead of using dense MLP layers for prediction, one can use low rank predictions and compensators to reconstruct the output. The idea is novel and worth exploration.
  2. Experimental results show that the proposed method outperforms other comparison methods in tight budget settings.
Critical weaknesses
  1. Real world latency is not only determined by FLOPs, but more importantly could we fuse kernels to better use GPU. The introduced branches likely make the deployment more complicated even though the configuration is iso-flops.
  2. More refinement on writing to make it easier to follow would be helpful
  3. At higher budget B, it loses significantly to other comparison methods in Figure 2 for PPL and Figure 4 for MMLU. Any explanation why it loses
  4. Would be good to have a method comparison section to compare the methodology difference with baselines
Questions for authors
  1. What’s the difference between comparison methods of TEAL vs the proposed methods?
  2. While measuring PPL is an interesting metric, it would be good to show how does the method compare on various downstream tasks?
4 comments
AuthorsOpenReview ↗

Response to Reviewer BnZy

We thank the reviewer for finding the idea novel and worth exploring. We have addressed each one of your concerns below.

Q1: Real world latency is not only determined by FLOPs, but more importantly could we fuse kernels to better use GPU. The introduced branches likely make the deployment more complicated even though the configuration is iso-flops.

Thank you for highlighting this important point. We have built and measured a fused kernel (Section 3.7). Each layer runs as a single kernel launch, where the predictor scores the neurons, a 4096-bin histogram picks the top-kk without sorting, the selected rows are gathered, and then the compensator is applied. The compensator runs inside that launch as qq linear neurons that are always on, so it adds no separate branch. Each decoding step is one CUDA graph, and ReComp replaces the MLP module without any other changes. On end-to-end decoding, this fused kernel is 1.34–1.45× faster than dense at B=0.25B=0.25 on five 7–14B models, with perplexity within 0.1% of the reference (Appendix A.1).

Q2: At higher budget B, it loses significantly to other comparison methods in Figure 2 for PPL and Figure 4 for MMLU. Any explanation why it loses?

Thanks for the great question. At higher budget B, ReComp loses to other comparison methods because of selection, not reconstruction (Sections 4.3–4.4). The following table, the relative WikiText-2 perplexity of Qwen2.5-7B at B=0.50B=0.50 demonstrates this:

SelectionCompensatorRel. ppl
Oracle top-kk (a ceiling)–1.009
Our rank-64 predictor–1.365
Our rank-64 predictor (ReComp)✓1.070
TEAL (exact magnitudes)–1.042
TEAL (TEAL$^\dagger$+C, ours)✓1.027

Since BB getting larger means more neurons needing to be selected, which neurons are kept matters more than what is being reconstructed, and our cheap predictor selects worse than exact magnitudes. Our compensator is not the limit since on TEAL's selection, it is the best method in the row, and a method with our compensator is best in 11 of 12 model–budget rows on the 7–8B models (Table 1). This is the same for MMLU, since oracle top-kk is within 1.7 points of dense on Qwen2.5-0.5B at B=0.35B=0.35. Conversely, the advantage grows as BB shrinks, since at B=0.10B=0.10 on Qwen2.5-7B, ReComp reaches 1.31 against 9.7–10.5 for the baselines.

Q3: Would be good to have a method comparison section to compare the methodology difference with baselines, and what’s the difference between comparison methods of TEAL vs the proposed methods?

Thank you for the thoughtful suggestion and question. We have added a method comparison section in Section 4.1 and a method table.

MethodSelectionSkipped neuronsPreparation
TEAL∣xi∣\lvert x_i\rvert above a threshold, per elementdiscardedcalibration
WINA∣xi∣ ∥W:,i∥\lvert x_i\rvert\,\lVert W_{:,i}\rVert above a thresholddiscardedcalibration
R-Sparseinput magnitudestatic low-rank copy of WWSVD
ReComprank-64 predictor of each neuron's output norm, top-kkreconstructed by an input-conditioned rank-qq mapclosed-form fit, distillation

Our co-design of a selector and reconstructor works since a low-rank selector misses neurons as a systematic function of xx, so its omitted output is 1.4–2.3× more predictable than TEAL's (Section 4.4). Additionally, at equal omitted energy the compensator removes 40% of it against 17% of TEAL's.

Q4: It would be good to show how does the method compare on various downstream tasks?

Thanks for the important question. We have now made all four 7–8B models have six zero-shot tasks and MMLU at every budget. At B=0.25B=0.25, ReComp has the best six-task average on three of four models and is best on HellaSwag, PIQA and WinoGrande on all four, though it loses for MMLU on three. Generation with dense verification is lossless (GSM8K, MATH-500, and AIME benchmarks).

We hope our responses address your concerns, and we would be grateful to answer any further questions.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer BnZy,

As the discussion period is nearing its end, we kindly ask if you could consider whether our newly added content and experiments have addressed your concerns. If so, we humbly ask that you re-evaluate our work. As for our writing, Section 4 has been rewritten and reorganized, and all revised portions of our manuscript are marked in blue. We are happy to provide any further clarifications that might be helpful to you. Thank you again for your time and consideration.

Sincerely,

Authors

Thanks for doing the extra experiments on latency, and the quality comparison on downstream tasks. It addressed some of my main concerns. I am happy to raise the score to 3.

AuthorsOpenReview ↗

Response to Reviewer BnZy

Thank you again for your time and constructive feedback. We very much appreciate the opportunity to improve our work based on your thoughts and review.

Reviewer_e24fOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method named ReComp for making an existing model cheaper to run, which reduces MLP computation by selecting a subset of neurons and using a low-rank correction to estimate the output of the skipped neurons. Experiments on five language models show that this can improve perplexity compared with computing more neurons at the same cost, although the benefits on downstream tasks are mixed.

Critical strengths

The authors test several model families (Qwen2.5, Mistral, and Llama 3.1) and sizes, which gives more confidence that the improvements are not specific to one model.

Critical weaknesses
  1. The overall speed benefit remains unclear. The reported 2.9× speedup is measured for a single MLP layer at batch size one, rather than the whole model. At batch size 16, the tested kernel is slower than dense computation.

  2. The authors report improvements in perplexity (Table 1), but these do not consistently lead to better task accuracy. ReComp often has lower accuracy than competing methods at similar computation budgets (Table 11), making its practical advantage less convincing.

Questions for authors

Can the authors provide full-model inference timings against the baselines at similar computation budgets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer e24f

We thank the reviewer for their constructive review and comments. We have addressed each one of your concerns below.

Q1: Can the authors provide full-model inference timings against the baselines at similar computation budgets?

Thank you for your request. Yes, we have provided full-model inference timings against baselines at similar computation budgets in Section 4.6 and Appendix A.1. These runs are all done on one RTX 5090 at batch size 1 with a 1,024-token cache, and are compared with the dense model running in the same CUDA graph engine. The dense baseline already is 1.12–1.20× faster than Hugging Face generate, so the speedups are not inflated by a slow reference. The following is a table of the speed relative to dense (×):

MethodBBQwen2.5-7BLlama-3.1-8BMistral-7BQwen2.5-14BQwen3-8B
ReComp0.251.451.401.431.411.34
TEAL0.251.511.471.491.441.38
ReComp0.501.261.231.241.231.20
TEAL0.501.291.261.271.251.23

At an equal budget, TEAL's kernel is ≈4\approx 4% faster since ReComp also reads its predictor and compensator. ReComp and TEAL, however, do not give equal quality, since at B=0.25B=0.25, TEAL's perplexity increase over dense is 1.6–5.9× larger than ReComp's, both on WikiText-2. At equal perplexity, they are similar at B=0.25B=0.25, TEAL is 5–10% faster at B=0.50B=0.50, and below B=0.25B=0.25 ReComp is faster:

ModelReComp BBReCompTEAL at equal WikiText-2 pplReComp / TEAL
Qwen2.5-7B0.101.611.461.10
Qwen2.5-7B0.151.541.441.07
Qwen3-8B0.101.441.371.05
Qwen3-8B0.151.401.341.05

The authors of WINA note that their Triton kernel corresponds to TEAL's kernel's latency, so TEAL's rows also give WINA's speed.

Lossless generation is where ReComp is ahead in every setting (Section 4.7). This is because ReComp is 1.48–1.70× faster than dense on GSM8K (TEAL is 1.44–1.61×). On Qwen3-8B with thinking, it is 1.30× on MATH-500 and 1.11× on AIME, where every training-free drafter is slower than dense (<1.00×). All drafters use one shared engine, acceptance rule and output shortlist. The baselines were tuned more than ReComp since they had 8–13 settings each against 4, with the fastest on 150 questions reported on 300 others.

Q2: At batch size 16, the tested kernel is slower than dense computation.

Thank you for the thoughtful observation. We agree with this, and the paper states this small-batch limitation. The rows of a batch select different neurons, which means that even our union kernel (reads each selected neuron once) gives per-layer speedups of 2.99×, 1.58×, 1.03× and 0.58× at batches 1, 2, 4 and 8, respectively. This is true for any neuron-level method, including prominent methods such as TEAL, Deja Vu, and PowerInfer.

Q3: The authors report improvements in perplexity (Table 1), but these do not consistently lead to better task accuracy. ReComp often has lower accuracy than competing methods at similar computation budgets (Table 11), making its practical advantage less convincing.

  • At B=0.25B=0.25, which was missing from the original Table 11, ReComp has the best six-task average of TEAL, WINA and R-Sparse on Qwen2.5-7B (71.1 vs 70.4), Mistral-7B (69.5 vs 68.1) and Qwen3-8B (69.8 vs 68.2), and is the best on HellaSwag, PIQA and WinoGrande on all four models.
  • It trails on MMLU on three models, and at B≥0.35B\ge0.35, due to selection. Oracle top-kk keeps MMLU within 1.7 points of dense on Qwen2.5-0.5B at B=0.35B=0.35.
  • With dense verification during generation, the output matches dense (token-identical under greedy up to floating-point ties, same distribution under sampling).

We hope our responses address your concerns, and we would be grateful to answer any further questions.

I appreciate the authors' responses and the extra experiments. I find that the speed gains over existing methods are modest in several settings, but the extra experiments still demonstrate useful improvements in terms of perplexity and generation efficiency. I would like to raise my score to 3.

AuthorsOpenReview ↗

Response to Reviewer e24f

Thank you again for your time and constructive feedback. We very much appreciate the opportunity to improve our work based on your thoughts and review, and we will make the speed comparison with existing methods explicit in the final version.

Reviewer_q46cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They uses a low-rank neuron-importance predictor that selects the top- neurons to compute by importance and uses a low-rank affine map to estimate the combined output of the skipped neurons from the MLP input, which is fitted via reduced-rank ridge regression (ridge-RRR) and distilled against the dense model with the objective having a KL-divergence term and an optional Jacobian-lens term. At 25% of compressed-layer MLP arithmetic, ReComp lowers relative WikiText-2 perplexity on Qwen2.5-7B from 1.44 for the best of TEAL, WINA, and R-Sparse to 1.16, and beats all three on every 7-8B model tested.

Critical strengths
  1. This article is easy to understand and provides executable code.
  2. The experimental model in this article is comprehensive and the results are convincing.
Critical weaknesses
  1. The data provided in the main table is too limited and the term "compressed/dense" in the caption is confusing. If it is intended to represent division, I think it would be more appropriate to use the percentage form. I think the original data is also acceptable.

  2. Table 1-12 has not been appropriately bolded. For PPL, small differences are difficult to explain the validity of the method. Could the author provide more Benchmarks to support the validity of the method? Refer to Table 1 in [1-2].

[1] Chen, Xinrui, et al. "A simple linear patch revives layer-pruned large language models." Advances in Neural Information Processing Systems 38 (2026): 154484-154509.

[2] Shopkhoev, Dmitriy, et al. "Replaceme: Network simplification via depth pruning and transformer block linearization." Advances in Neural Information Processing Systems 38 (2026): 19487-19517.

Questions for authors
  1. Can the author provide the experimental results of the Reasoning model (eg. Qwen 3-8B)? And more challenging dataset results such as the AIME, PIQA, ARC datasets?
  2. The main table suggests expanding the data and highlighting the advantages of the existing methods. From Table 1,2,3,11, it seems that the success rate of the proposed method is not high.
2 comments
AuthorsOpenReview ↗

Response to Reviewer q46c

We thank the reviewer for their constructive suggestions and have followed each of them.

Q1: Experimental results of the Reasoning model (eg. Qwen 3-8B)? And more challenging dataset results such as the AIME, PIQA, ARC datasets?

Thank you for your suggestion. We have added Qwen3-8B to the full pipeline and done the necessary comparisons against TEAL, WINA, and R-Sparse. At B=0.25B=0.25, ReComp’s relative perplexity on Qwen3-8B is 1.071 versus 1.420–1.530 on WikiText-2, and its six-task average is 69.8 against 68.2. Here are some results with thinking enabled and dense verification:

Qwen3-8BMATH-500AIME 2024AIME 2025
Accuracy, dense55.051.738.3
Accuracy, ReComp drafter53.850.036.7
Speed vs dense, ReComp1.30×1.11×1.11×
Speed vs dense, TEAL0.95×0.81×0.80×

Accuracies are the averages of two sampled runs. After adding verification (speculative decoding), the output is distributed exactly as the dense model’s samples, so the differences are sampling noise. Additionally, PIQA and ARC are in the new downstream table.

Q2: The data provided in the main table is too limited and the term "compressed/dense" in the caption is confusing. If it is intended to represent division, I think it would be more appropriate to use the percentage form. Table 1-12 has not been appropriately bolded.

Thank you for the thoughtful observation. The main table now covers all 6 of the models we tested at every budget that we ran for ReComp and its baselines where they were run. The caption also states that 1.000 is dense and 1.158 means that the perplexity is 15.8% higher, and gives every dense perplexity there also. A new appendix figure also shows the same thing in percentage form; for example, at B=0.25B=0.25, ReComp's increase is 16% versus 44% for the best baseline on Qwen2.5-7B. Additionally, every table now marks the best results in bold and the second-best results with an underline.

Q3: For PPL, small differences are difficult to explain the validity of the method. Could the author provide more Benchmarks to support the validity of the method? Refer to Table 1 in Chen, Xinrui, et al and Shopkhoev, Dmitriy, et al.

Thanks for the great suggestion. The new main-text downstream table in Section 4.5 uses this format. Six-task average and retained performance (RP) at B=0.25B=0.25 can be summarized here:

ModelReComp Avg.RPBest of TEAL / WINA / R-SparseRP
Qwen2.5-7B71.193.9%70.493.0%
Mistral-7B69.595.3%68.193.4%
Qwen3-8B69.894.2%68.292.0%
Llama-3.1-8B69.091.0%69.691.7%

ReComp is best on HellaSwag, PIQA, and WinoGrande on all four models. We have also added five-shot MMLU at every budget in the appendix.

Q4: From Table 1,2,3,11, it seems that the success rate of the proposed method is not high.

Thank you for the thoughtful observation.

  • For perplexity, when B=0.25B=0.25, ReComp is better than TEAL, WINA, and R-Sparse on all four 7–8B models and both datasets. At B=0.10B=0.10, ReComp is at 1.21–1.31 while the baselines reach 9.7–27.
  • A method with our compensator is best in 11 of 12 rows on the 7–8B models (14 of 15 overall). Where our predictor loses at B=0.50B=0.50 is because of selection, and a method with our compensator is still best on three of the four models.
  • The downstream results at B=0.25B=0.25 are the best six-task average on three of four models.
  • During Lossless generation, ReComp is the fastest compressed drafter in every setting, and our measured speed is 1.34–1.45× faster than dense end-to-end.

We hope our responses address your concerns, and we would be grateful to answer any further questions.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer q46c,

As the discussion period is nearing its end, we kindly ask if you could consider whether our newly added content and experiments have addressed your concerns. If so, we humbly ask that you re-evaluate our work. We are happy to provide any further clarifications that might be helpful to you. Thank you again for your time and consideration.

Sincerely,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all the reviewers again for their insightful feedback, which has greatly improved the quality of our paper. Here, we provide a general response, as well as individual responses to each reviewer. We have also uploaded a revised version of our manuscript highlighting some changes that are marked in blue, and the updated code.

Our method is unchanged, as we keep the same rank-64 predictor, rank-384 compensator, closed-form fit, distillation loss and budget accounting. Our main additions and revisions are as follows:

  • ReComp achieves 1.34-1.45$\times$ faster end-to-end decoding than the dense model (B=0.25B=0.25, batch size 1) using a fused Triton kernel and CUDA graph engine (Sections 3.7 & 4.6).
  • We implement lossless generation, in which ReComp drafts tokens and a dense forward pass verifies them, so the resulting output is the dense model’s (Sections 3.8 & 4.7). To compare ReComp against its baselines, every drafter runs in the same engine with the same acceptance rule and output shortlist. The table below summarizes lossless generation results, where each number is the speed relative to the dense model:
DrafterQwen2.5-7B GSM8KLlama-3.1-8B GSM8KMistral-7B GSM8KQwen3-8B MATH-500Qwen3-8B AIME24
ReComp (ours)1.701.481.611.301.11
TEAL1.611.441.570.950.81
WINA1.481.361.500.89–
R-Sparse1.361.311.390.77–
Layer skipping0.990.970.990.74–
  • We show that at budgets below 25%, ReComp clearly beats baselines in terms of relative WikiText-2 perplexity at the same arithmetic for every method:
ModelBBReCompTEALWINAR-Sparse
Qwen2.5-7B0.101.3110.5010.109.66
Qwen2.5-7B0.151.234.384.183.67
Qwen3-8B0.101.2124.2424.9627.23
Qwen3-8B0.151.145.245.346.12

At equal perplexity, ReComp at these budgets decodes 3–10% faster than TEAL, which needs 2.1–3.0× more MLP arithmetic to match ReComp.

  • We have added a reasoning model, Qwen3-8B, through the full pipeline and comparisons, as per Reviewer q46c's request.
  • A downstream accuracy table, in the format suggested by Reviewer q46c. For all four 7–8B models, it includes six zero-shot tasks, average and retention of dense, and five-shot MMLU.
  • ReComp trails at B=0.50B=0.50 due to selection, not compensation. Our compensator on TEAL's selection (TEAL$^\dagger$+C) lowers TEAL's perplexity at every budget and is the best method in most rows at B≥0.35B\ge0.35.
  • We have also made refinements to the presentation, with a method-comparison table, perplexities of dense models in every caption, consistent bolding and underlining, polished writing, and a rewritten Section 4.

We thank all reviewers again, look forward to further discussions, and would be grateful if ratings could be updated where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.