acceptodds
Under review as a conference paper at ICLR 2027

RoShift: Lazy Shift and Weight-aware Rotation for Activation Sparsity in LLM Inference

Abstract

Activation sparsity accelerates LLM decoding by skipping unnecessary weight reads and computation. However, SiLU-based LLMs exhibit little natural sparsity, and imposing high sparsity can substantially degrade accuracy. We propose RoShift, a training-free method that transforms activations into sparsity-friendly representations. RoShift introduces lazy shift, which recenters SiLU activation distributions around zero and defers correction to the end of the FFN to obtain additional sparsity. It also reduces sparsification-induced output error through rotation that jointly accounts for activation and weight information. By fusing the corrections required for shifting and rotation, RoShift translates these gains into decoding speedup. At 50% and 60% sparsity, RoShift improves average accuracy over TEAL by 1.60 and 3.29 percentage points, respectively, across five models. It achieves up to decoding speedup over the dense model on an RTX 5090 at 50% sparsity and yields a better accuracy-speedup Pareto frontier than TEAL.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
activation sparsity, decoding acceleration, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zTbKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RoShift, which combines lazy shift with activation- and weight-aware rotation. Lazy shift recenters post-SiLU activations and defers the corresponding correction to the end of the FFN, creating additional sparsity that both the up and down projections can exploit. Activation- and weight-aware rotation jointly incorporates activation statistics and weight information to construct rotations that reduce sparsification-induced output error. RoShift further fuses the associated correction computations to reduce overhead and employs efficient sparse kernels to translate these computational savings into practical decoding acceleration.

Critical strengths
  • Lazy shift offers a concrete improvement to gated FFNs, supported by direct ablation evidence. It recenters post-SiLU activations and reformulates the corresponding correction as a linear computation with precomputable weights, allowing both the up and down projections to exploit the resulting sparsity.

  • The quality improvements at high sparsity are consistently supported across models and tasks. Table 2 and the appendices cover 5 models and 11 tasks, with relevant baselines including TEAL, LaRoSA, and WINA. At target sparsity levels of 50% and 60% with non-uniform allocation, RoShift improves average accuracy over TEAL by 1.60 and 3.29 percentage points, respectively, outperforming it in 50 and 54 of the 55 model-task combinations. These results indicate that the gains are not primarily driven by isolated models or tasks and support the claim that RoShift better preserves model quality under aggressive sparsification.

  • The paper translates its methodological improvements into measured decoding acceleration and provides evidence of the accuracy-speed trade-off. RoShift addresses the overhead introduced by representation transfromations through correction-transition fusion, caching and sparse kernels.

Critical weaknesses
  • The paper contains several redundant or inconsistent definitions. What exactly does "WA-Top-K" in Table 4 denote? This operation is not introduced in the preceding method section. What distinguishes the full RoShift method drom the "Lazy shift + WA-Top-K" configuration? Similarly, line 251 defines TlT_l as the inter-layer transition matrix: does this refer to Ql⊤Ql+1Q_{l} ^\top Q_{l+1}? Please clarify these definitions and use consistent mathematical notation throughout the paper.

  • Why is LaRoSA omitted from the speedup comparison? Compared with LaRoSA’s rotation-based approach, RoShift introduces additional computation for shift correction, yet the paper does not clearly quantify its impact on decoding speed. Could the authors include LaRoSA as a baseline under matched experimental conditions and provide a controlled ablation that isolates the latency overhead of shift correction and the extent to which the proposed fusion mitigates it?

  • How is the condition that "the relative difference between the actual RMS and its representative is within 5%" enforced in Correction Fusion? My understanding is that checking this condition and handling cases that exceed the tolerance require dedicated kernel-level logic, but I could not find these implementation details in the appendix. Could the authors clarify how this mechanism is implemented?

  • The description of activation- and weight- rotation is difficult to follow and relies heavily on the appendix, leaving the main text insufficiently self-contained. At line 220, although the single-axis output-contribution variance, $e_q = q Sq $, provides a meaningful motivation, the mixed-statistics eigenbasis is not an exact solution to this objective, nor does the derivation establish specific distributional properties of the rotated activations. What statistics does this rotation change relative to PCA, and how do those changes reduce actual sparsification error? The paper lacks paired comparisons under identical inputs, such as covariance structure, cumulative variance or energy, near-zero probability, or output-contribution distributions. Table 4 supports the empirical advantage of AW-Rot over A-Rot, but higher accuracy alone does not explain the statistical mechanism behind that advantage.

  • After introducing the additional correction weights for recentering, my understanding is that RoShift’s effective computational sparsity should account for operations absent from TEAL and LaRoSA. However, I could not find a clear explanation of how this overhead is incorporated into the sparsity calculation. Specifically, at the nominal 40% sparsity reported in Table 1, what are the actual effective sparsity levels of TEAL, LaRoSA, and RoShift when all additional computations are included?

Questions for authors
  • Could the autors clarify the definition of WA-Top-K, the difference between full RoShift and "Lazy shift + WA-Top-K", and whether T_l = Q_l ^\top Q_{l+1}? Please use consistent notation and introduce all components explicitly in the main text.

  • Could the authors include LaRoSA in the decoding-speed comparison under matched hardware, workloads, and sparsity settings? Please separately quantify the latency overhead introduced by shift correction and the overhead reduction achieved through correction fusion.

  • How is the 5% relative RMS tolerance enforced in the implementation? Please explain how representatives are selected, how the condition is checked, what fallback is used when the tolerance is exceeded, and whether these operations are included in the reported latency.

  • Could the authors provide a self-contained description of how the activation- and weight-aware rotation is contructed and explain its statistical advantages over PCA? Please provide paired diagnostics under identical inputs and sparsification settings to show which statistics change the how those changes relate to reduced sparsification error, beyond the accuracy comparisons in Table 4.

  • How is effective computational sparsity defined after accounting for shift correction, rotation transitions, and other additional operations? At the reported nominal 40% sparsity, what effective spasity does each of TEAL, LaRoSA, and RoShift achieve under a common accounting convention? Please specify the fomula, denominator, and included operations.

I would appreciate the authors' responses to these questions and will consider my final score in light of their clarifications.

5 comments

Thank you for your constructive feedback. We address your comments and questions below.

Q1: Could the autors clarify the definition of WA-Top-K, the difference between full RoShift and "Lazy shift + WA-Top-K", and whether T_l = Q_l ^\top Q_{l+1}? Please use consistent notation and introduce all components explicitly in the main text.

Thank you for pointing out the unclear definitions in our paper.

WA-Top-K stands for Weight-Aware Top-K, which selects components based on the importance score ∣Xi∣⋅∥Wi,:∥2|X_i| \cdot \|W_{i,:}\|_2, considering both activation magnitudes and the corresponding weight norms. However, for Up output sparsity, we use standard Top-K instead.

Lazy Shift + WA-Top-K is an ablation configuration of RoShift that excludes only the rotation component.

Transition Matrix TlT_l is indeed defined as Tl=Ql⊤Ql+1T_l = Q_l^\top Q_{l+1}, as the reviewer pointed out.

We have revised several parts of the manuscript to clarify these definitions and ensure consistent mathematical notation throughout the paper.

Q2: Could the authors include LaRoSA in the decoding-speed comparison under matched hardware, workloads, and sparsity settings? Please separately quantify the latency overhead introduced by shift correction and the overhead reduction achieved through correction fusion.

Although LaRoSA reports end-to-end decoding speedups in its paper, its publicly released code contains only PyTorch code for accuracy evaluation and does not include implementations for decoding acceleration, such as Sparse GEMV kernels. We therefore omitted LaRoSA from the original speedup comparison, as noted in the original manuscript. We have clarified this rationale in the revised manuscript.

To address the reviewer’s concern, we implemented LaRoSA’s sparsification method (denoted by ‘LaRoSA’) using RoShift's GPU kernel backend and measured its decoding latency. However, while RoShift's kernel supports only B128 Blockwise Top-K, LaRoSA uses a kernel that fuses Top-K selection with sparse GEMV in its original paper. Furthermore, differences may exist in sparse mask generation and GEMV implementation. Therefore, the measured ‘LaRoSA’ performance below may differ from that of its original optimized implementation.

To separately quantify the overhead introduced by shift correction and the reduction achieved through correction fusion, we evaluated RoShift under the following configurations. For this ablation, all configurations use a dense LM-head.

  • Correction Off: Shift correction is disabled. This is a timing-only ablation used to measure correction overhead.
  • Unfused Serial: Correction operations are executed separately in a serial manner.
  • Fused Serial: Correction and transition operations are fused, and they are executed in serial with the FFN.
  • Fused Parallel: The final RoShift configuration, applying both correction fusion and parallel execution with the sparse FFN.

The following table presents the end-to-end decoding latency measured under the same experimental conditions: RTX 5090 GPU, batch size 1, 50% uniform sparsity, and a dense LM-head. All values are reported in ms/token, where lower is better.

ModelDense (Paper)‘LaRoSA’ (RoShift Backend)Correction OffUnfused SerialFused SerialFused Parallel
Llama3.1-8B9.946.916.917.646.956.75
Llama2-7B9.566.396.317.076.386.18
Mistral-7B10.566.426.377.116.446.23
Qwen2.5-7B10.346.346.306.796.386.19
Qwen3-8B10.267.297.578.307.597.35

With the final Fused Parallel configuration, RoShift achieved lower decoding latency than LaRoSA on four out of five models.

Shift Correction Overhead and Fusion Effects

ModelCorrection Overhead (ms)Fusion Reduction (ms)Parallel Reduction (ms)
Llama3.1-8B0.730.690.20
Llama2-7B0.760.690.20
Mistral-7B0.740.670.21
Qwen2.5-7B0.490.410.19
Qwen3-8B0.730.710.24

The table above quantifies the overhead of shift correction, the latency reduction from fusing correction and transition operations, and the additional benefit of multi-stream parallel execution.

Executing shift correction separately introduces an additional overhead of 0.49–0.76 ms/token. However, correction fusion reduces latency by 0.41–0.71 ms/token, mitigating most of this additional overhead. Furthermore, multi-stream parallel execution provides an additional latency reduction of 0.19–0.24 ms/token.

Overall, although RoShift adds computation for shift correction, correction fusion and parallel execution mitigate most of the overhead. As a result, RoShift achieves competitive decoding latency compared with LaRoSA implemented under the shared kernel backend.

We have also incorporated the above experimental results and analysis into Appendix 11 of the revised manuscript.

Q3: How is the 5% relative RMS tolerance enforced in the implementation? Please explain how representatives are selected, how the condition is checked, what fallback is used when the tolerance is exceeded, and whether these operations are included in the reported latency.

During inference, we compare the actual RMS value against all representatives and select the closest one. We then compute the relative error between the actual RMS value and the selected representative and set a Boolean flag if the error exceeds 5%, indicating that fallback is required.

Based on this Boolean flag, tl.where selects the pointer to either the precomputed Fused Matrix or the Transition Matrix, while the mask in tl.load ensures that the Correction Matrix is additionally loaded only when fallback occurs. All these operations are handled within a single Triton kernel.

All these operations are included in the reported end-to-end decoding latency. We have added this explanation and further implementation details to Appendix 8.2 in the revised manuscript.

Q4: Could the authors provide a self-contained description of how the activation- and weight-aware rotation is contructed and explain its statistical advantages over PCA? Please provide paired diagnostics under identical inputs and sparsification settings to show which statistics change the how those changes relate to reduced sparsification error, beyond the accuracy comparisons in Table 4.

To better explain the statistical mechanism of Activation- and Weight-Aware Rotation (AW-Rot), we conducted additional experiments comparing No Rotation, Activation-only Rotation (PCA), and AW-Rot under identical inputs and sparsification settings.

We evaluated a total of 128 projections (Q/K/V/Gate across all 32 layers) of Llama3.1-8B. We used the same 2,048 tokens sampled from the mixed calibration dataset and applied 50% sparsity with B128 Weight-aware Top-K to both methods. Lazy Shift was disabled to isolate the effect of rotation.

In the table below, Leading 50% refers to the first half of the coordinates, ordered by eigenvalues for PCA and AW-Rot, and by their original order for No Rotation. These are distinct from the coordinates actually selected by Top-K. Output NMSE (Normalized Mean Squared Error) is calculated as the squared output error normalized by the squared norm of the dense output for each projection, then averaged across all 128 projections.

Diagnostic (50% Sparsity)No RotationPCAAW-Rot
Leading 50% Activation Variance49.51%83.69%80.49%
Leading 50% Weight Energy49.97%58.44%78.92%
Leading 50% Output Contribution Variance49.59%90.77%94.85%
Removed Contribution Proxy4.41%1.62%0.89%
Output NMSE (%)2.11%1.33%0.70%

The results show that AW-Rot does not concentrate activation variance more effectively than PCA, but it achieves greater concentration of weight energy and output contribution variance. Furthermore, the removed output contribution proxy decreased from 1.62% to 0.89%, while the actual output NMSE decreased by 47.75% compared with PCA.

PCA constructs its rotation to maximize activation variance concentration without considering the corresponding weight energy. In contrast, AW-Rot constructs its rotation by jointly considering activation variance and weight energy. As a result, we observed that output contributions became more concentrated in the leading axes, accompanied by reductions in both the removed output contribution proxy and the actual sparsification error.

We have also clarified the construction of AW-Rot and its statistical differences from PCA in Section 3.3 of the revised manuscript. We would appreciate it if the reviewer could refer to the revised section.

Q5: How is effective computational sparsity defined after accounting for shift correction, rotation transitions, and other additional operations? At the reported nominal 40% sparsity, what effective spasity does each of TEAL, LaRoSA, and RoShift achieve under a common accounting convention? Please specify the fomula, denominator, and included operations.

We define effective computational sparsity based on the number of GEMV multiply-accumulate (MAC) operations performed in the seven Transformer linear projections, the LM-head, and the additional transition and correction matrix operations. Non-GEMV operations, such as attention score computation and RMS normalization, are excluded.

The denominator of effective computational sparsity is the total number of GEMV MACs performed by the Transformer projections and LM-head in the dense model. The formula is defined as follows:

Seffective=1−(MACTransformer+MACLMHead+MACAdditional)/(MACDense,Transformer+MACDense,LMHead)S_{effective} = 1 - (MAC_{Transformer} + MAC_{LMHead} + MAC_{Additional}) / (MAC_{Dense,Transformer} + MAC_{Dense,LMHead})

In RoShift, correction fusion combines the transition and correction operations into a single GEMV. However, when the RMS tolerance condition is not satisfied, the fallback executes the two operations separately, introducing additional GEMV MACs proportional to the fallback rate.

RoShift's fallback rate is below 10% in most cases. Assuming a fallback rate of 10%, the effective computational sparsity of each method at 40% nominal sparsity is shown below for each model.

MethodLlama-3.1-8BLlama-2-7BMistral-7BQwen2.5-7BQwen3-8B
TEAL37.20%39.21%39.26%36.92%36.71%
WINA37.20%39.21%39.26%36.92%36.71%
LaRoSA30.05%31.08%31.71%31.83%28.73%
RoShift (Dense LM Head)29.33%30.27%30.96%31.32%27.93%
RoShift (Sparse LM Head)35.14%31.77%32.28%37.11%33.81%

Please note, however, that effective computational sparsity does not directly reflect actual GPU execution efficiency. RoShift improves GPU execution efficiency by reusing Up output sparsity in Down, maintaining a fixed workload per block through Blockwise Top-K, and executing correction operations in parallel with the sparse FFN. Therefore, RoShift can achieve higher decoding speedups even at the same effective computational sparsity.

Reviewer_h992OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SiLU-based models rarely produce exact zeros, so magnitude pruning tends to fall apart at high sparsity. This paper works in that setting, targeting memory-bound batch-size-1 decoding. RoShift combines three pieces. Lazy shift subtracts a calibrated per-channel median from the post-SiLU gate and corrects for it exactly at the FFN output with a precomputed, input-independent matrix. Selecting Up-output and Down-input channels on the recentered magnitudes lets both projections use the extra sparsity without restoring activations mid-FFN. Activation- and weight-aware rotation builds a per-layer orthogonal basis from a mixture of six trace-normalized statistics (attention and FFN covariances plus Q/K/V/Gate Gram matrices), motivated by per-axis output contribution variance; the authors note plainly that this is a workable surrogate, not an exact optimizer. Around these two ideas a fair amount of systems work is done: fusing the inter-layer transition with the shift correction through RMS-representative matrices with exact fallback, blockwise Top-K, index-based sparse GEMV, dual CUDA streams, a rotated three-stage LM-head with early termination, and perplexity-driven searches over sparsity allocation and rotation mixtures. Evaluation covers five 7-8B base models on 11 benchmarks at 40/50/60% Transformer MAC sparsity. RoShift leads mean accuracy in 13 of 15 settings, beating TEAL by 1.60 points at 50% and 3.29 points at 60%. End-to-end FP16 batch-1 decoding reaches up to 1.72x at 50% uniform sparsity on an RTX 5090, with a better accuracy-speedup Pareto than TEAL.

Critical strengths
  1. Lazy shift is the paper's best idea, and it is both exact and well isolated. Centering itself is not new: SCAP did it for non-GLU FFNs (https://neurips2024-enlsp.github.io/papers/paper_7.pdf) — but showing the SwiGLU shift term reduces to a precomputable correction, so Up and Down can both sparsify on recentered magnitudes, is a neat step. On Llama-3.1-8B at 50%/60% uniform sparsity, weight-aware Top-K goes from 52.97/42.57 to 58.95/53.28 once lazy shift is added, and RoShift beats TEAL in 54 of 55 model-benchmark pairs at 60%.
  2. This is a measured systems paper rather than a sparsity simulation. Batch-1 throughput is reported with correction, transition, and head costs included, alongside a stepwise speedup breakdown, a memory-latency tradeoff curve, and explicit overhead accounting, with code provided. TEAL remains the only public end-to-end baseline, with LaRoSA reporting 1.30x wall-clock at 40% on LLaMA2-7B. Results on both a desktop GPU and an edge device are great for the paper.
Critical weaknesses
  1. The accuracy-speedup claim over TEAL is not shown under an equal-LM-head comparison. At matched 50% uniform Transformer sparsity, the paper's own ablation has RoShift without its sparse head running slower than TEAL: 1.47x vs. 1.55x on Llama-3.1-8B and 1.42x vs. 1.46x on Qwen3-8B — and RoShift only moves ahead once head sparsification is added (1.59x/1.51x, Fig. 6). That step offsets a 14.04% correction overhead with a 10.49% head saving while TEAL keeps a dense head. To be clear, the head itself is a solid result: skipping 71-83% of head MACs where plain input sparsification manages 2-3% is nothing to dismiss. The problem is bundling. Table 3 and the Pareto combine a Transformer-sparsity claim with a head-sparsity advantage TEAL never gets, using a head construction (offline rotation plus staged pruning) that could in principle help any baseline. The accuracy table has a milder version of the same issue, with dense correction work excluded from the budget. Until a head-controlled Pareto is shown, the practical conclusion is not established.
  2. Accuracy gains come with friendlier tuning attached. RoShift searches allocation and rotation mixtures against full-model perplexity on its own 32-sequence mix with math and code content, roughly 5 GPU-hours per model and sparsity level, while baselines stay on their published recipes, and the largest gaps appear on generation tasks in those same domains. The unconstrained-allocation novelty claim should also acknowledge concurrent non-monotonic allocation work such as WiSparse, which reports non-monotonic block sensitivity and weight-aware mixed allocation (https://arxiv.org/abs/2602.14452). With evaluation limited to 7-8B base models at batch size 1 and no seeds or intervals, it’s hard to tell how much of the margin is the representation and how much is in-domain search. Same-calibration baselines would help resolve it.
Questions for authors
  1. Could you add Pareto curves for RoShift with a dense head and, if feasible, TEAL with the same rotated three-stage head? A per-model split of Transformer-path versus head speedup at 50% uniform sparsity would answer it. If RoShift still leads on both, the head counts as a strength.
  2. Do the sparsity targets cover correction, transition, and fused-matrix work? Accuracy at matched executed MACs and bytes read per token, corrections included, would show whether the headline margins survive equal-cost comparison.
  3. Would you run TEAL and WINA under your calibration mix (or RoShift under C4 only) and report held-out perplexity for every method? I mainly want to know whether the high-sparsity generation-task margins hold.
  4. Plain Top-K in the ablation trails TEAL by about 5 points on the same model and sparsity. Is that Up-output vs. input sparsity, Top-K vs. thresholding, or achieved sparsity? It matters for how much of the lazy-shift gain to attribute to the shift itself.
5 comments

Thank you for your constructive feedback. We address your comments and questions below.

Q1: Could you add Pareto curves for RoShift with a dense head and, if feasible, TEAL with the same rotated three-stage head? A per-model split of Transformer-path versus head speedup at 50% uniform sparsity would answer it. If RoShift still leads on both, the head counts as a strength.

We have added the requested Pareto curves to Appendix 12. RoShift achieves a superior accuracy–speedup Pareto frontier over TEAL even without LM-head sparsification.

Specifically, we compared both methods under two matched settings: (1) both using a dense LM-head and (2) both using the same rotated three-stage sparse LM-head. RoShift consistently maintained its advantage in both settings.

We further quantify these accuracy–speedup trade-offs using the Hypervolume indicator [1, 2]. The reference point is set to 1.00× speedup and 1 percentage point (pp) below the minimum accuracy among the compared configurations. Higher values indicate better performance. We would appreciate it if the reviewer could refer to Appendix 12 for the actual Pareto curves. All decoding speedups used in the Pareto analysis were measured on an RTX 5090 GPU at batch size 1.

MethodLM-HeadLlama3.1-8BMistral-7BQwen3-8B
TEALDense5.686.092.55
RoShiftDense7.267.483.25
TEALSparse6.956.243.54
RoShiftSparse8.967.603.93

Finally, the table below shows the respective contributions of the Transformer path and LM-head to decoding acceleration at 50% uniform sparsity. Transformer Savings represents the gross latency reduction in the Transformer path, excluding RoShift-specific overhead.

ModelTransformer Savings (ms)Additional LM-Head Savings (ms)Speedup (Dense Head → Sparse Head)Share of Component Savings (Transformer / Head)
Llama3.1-8B3.160.481.47× → 1.58×86.83% / 13.17%
Mistral-7B4.310.111.69× → 1.72×97.59% / 2.41%
Qwen3-8B2.950.471.41× → 1.51×86.12% / 13.88%

[1] E. Zitzler and L. Thiele, "Multiobjective Optimization Using Evolutionary Algorithms—A Comparative Case Study," PPSN V, 1998. https://doi.org/10.1007/BFb0056872

[2] H. Ishibuchi et al., "How to Specify a Reference Point in Hypervolume Calculation for Fair Performance Comparison," Evolutionary Computation, 2018. https://doi.org/10.1162/evco_a_00226

Q2: Do the sparsity targets cover correction, transition, and fused-matrix work? Accuracy at matched executed MACs and bytes read per token, corrections included, would show whether the headline margins survive equal-cost comparison.

When comparing RoShift against TEAL at the same Bytes Read per Token as TEAL's 50% sparsity, RoShift achieved higher average benchmark accuracy on four out of five models.

The target sparsity defined in our paper is based on the seven Transformer projections. In contrast, for this equal-cost comparison, we matched the number of GEMV MAC operations, including Correction, Transition, and Fused Matrix operations. For batch-size-1 GEMV with weights of the same precision, the number of MACs is proportional to the amount of weight data read, allowing us to compare both costs. Operations such as the LM-head, RMS Normalization, and Attention Score computation are excluded from this comparison. We conservatively assume a 10% Correction Fallback Rate.

The table below compares performance under non-uniform sparsity, with a dense LM-head for all methods and Bytes Read accounting for Correction, Transition, and Fused Matrix operations. At matched Bytes Read per Token, RoShift achieved an average accuracy improvement of 0.33 pp over TEAL.

MethodLlama3.1-8BLlama2-7BMistral-7BQwen2.5-7BQwen3-8BAvg
TEAL Actual Sparsity48.24%49.43%49.42%48.18%47.89%-
RoShift Target Sparsity56.70%58.55%57.88%54.24%57.46%-
TEAL Avg. Accuracy59.2144.3456.9570.4673.48960.89
RoShift Avg. Accuracy59.3743.7758.1671.2973.49461.22

Additionally, the table below compares the decoding speedups at the corresponding uniform sparsity levels. This is a separate experiment from the non-uniform sparsity accuracy comparison presented above.

MethodLlama3.1-8BLlama2-7BMistral-7BQwen2.5-7BQwen3-8B
TEAL Actual Sparsity48.25%49.44%49.42%48.18%47.87%
RoShift Uniform Sparsity57.03%58.59%57.81%53.91%57.81%
TEAL Speedup1.55x1.57x1.62x1.60x1.43x
RoShift Speedup1.65x1.65x1.70x1.60x1.55x

Please note that matching Bytes Read per Token does not necessarily imply equal GPU execution efficiency. RoShift improves GPU execution efficiency by reusing Up output sparsity in Down, maintaining a fixed workload per block through Blockwise Top-K, and executing Correction operations in parallel with the sparse FFN. These structural advantages enable RoShift to achieve higher decoding acceleration through more efficient GPU execution, beyond simply reducing memory reads.

We have also incorporated the above experimental results and analysis into Appendix 9.7 of the revised manuscript.

Q3: Would you run TEAL and WINA under your calibration mix (or RoShift under C4 only) and report held-out perplexity for every method? I mainly want to know whether the high-sparsity generation-task margins hold.

To investigate the impact of calibration datasets on performance at high sparsity, we evaluated the held-out perplexity (PPL) of TEAL, WINA, and RoShift using different calibration datasets. All PPL measurements were conducted on a held-out mixed dataset.

First, we compared TEAL and RoShift using both Alpaca and Mixed calibration datasets at 60% uniform sparsity. TEAL's Alpaca dataset contains 20,480 tokens, while RoShift's Mixed dataset contains 131,072 tokens. Both methods showed relatively small PPL changes when switching calibration datasets, and RoShift consistently maintained lower PPL than TEAL under both calibration settings.

For Mistral-7B, TEAL's threshold-based sparsification appears to be particularly sensitive to the attention sink effect at the initial token [3]. When the first token was excluded from sparsification, PPL decreased substantially from 48.16 to 5.46 with Alpaca calibration and from 49.75 to 5.57 with Mixed calibration.

MethodLlama3.1-8BLlama2-7BMistral-7BQwen2.5-7BQwen3-8BAvg
Dense4.995.924.605.005.155.13
TEAL (Alpaca-20k)7.4312.6948.166.476.6016.27
TEAL (Mixed)7.8313.1249.756.536.6316.77
RoShift (Mixed)6.086.955.295.595.765.93
RoShift (Alpaca-20k)6.126.884.985.685.785.89

WINA uses calibration data only for sparsity allocation search, so we conducted the comparison at 60% non-uniform sparsity. WINA also uses Alpaca for calibration, but its dataset contains only approximately 2,000 tokens, substantially fewer than TEAL's 20,480 tokens.

MethodLlama3.1-8BLlama2-7BMistral-7BQwen2.5-7BQwen3-8BAvg
Dense4.995.924.605.005.155.13
WINA (Alpaca-2k)6.696.795.255.986.156.17
WINA (Mixed)6.736.665.395.996.126.18
RoShift (Mixed)5.926.795.725.245.615.86
RoShift (C4-only)6.037.276.105.265.626.06
RoShift (Alpaca-2k)8.797.135.415.655.746.55

Additionally, the C4-only calibration dataset consists of the C4 portion of RoShift's Mixed dataset, containing 65,536 tokens. RoShift achieved average PPLs of 5.86 with Mixed, 6.06 with C4-only, and 6.55 with WINA's approximately 2,000-token Alpaca dataset. These results further suggest that calibration dataset size may affect RoShift's rotation construction.

Switching WINA's calibration dataset from Alpaca to Mixed did not consistently improve its PPL, with the five-model average remaining nearly unchanged (6.171 vs. 6.176). Under the same Mixed calibration dataset, RoShift maintained a lower average PPL than WINA (5.86 vs. 6.18).

Additionally, RoShift maintained its accuracy advantage over TEAL and LaRoSA on generation benchmarks when all methods used the same Mixed calibration dataset. Detailed results are provided in Appendix 9.6.

[3] G. Xiao et al., "Efficient Streaming Language Models with Attention Sinks," ICLR, 2024. https://openreview.net/forum?id=NG7sS51zVF

Q4: Plain Top-K in the ablation trails TEAL by about 5 points on the same model and sparsity. Is that Up-output vs. input sparsity, Top-K vs. thresholding, or achieved sparsity? It matters for how much of the lazy-shift gain to attribute to the shift itself.

The Plain Top-K baseline in our ablation uses Up-output sparsity with B128 Blockwise Top-K. This configuration was chosen to isolate the effect of Lazy Shift under the same sparsification settings as RoShift. For B128 Top-K, the achieved sparsity is exactly 50%. In contrast, the Up-output Threshold configuration achieves an average sparsity of 52.96% across 11 benchmarks, which may also contribute to the observed accuracy difference. The additional ablation results for each setting are presented below.

MethodLlama3.1-8B
TEAL (uniform)57.44 (-6.43)
Table4 Plain Top-K52.44 (-11.43)
TEAL + B128 input Top-K55.44 (-8.43)
TEAL + Up-output sparsity (threshold)54.04 (-9.83)

Compared with TEAL, applying B128 Input Top-K reduces accuracy by 2.00 pp, while applying Threshold-based Up-output Sparsity reduces accuracy by 3.40 pp. Therefore, the 5.00 pp accuracy gap between Plain Top-K and TEAL reflects the effects of both the sparsification location and the selection method.

As the reviewer correctly pointed out, the experimental settings of the original ablation were not clearly described. We have clarified these settings in the revised manuscript.

Reviewer_dRSqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RoShift is a training-free method for activation-sparse LLM decoding. It combines lazy shift, which recenters post-SiLU activations and postpones the corresponding correction until the FFN output, with activation- and weight-aware rotations. Correction fusion and specialized sparse kernels reduce the execution overhead. Experiments on five models demonstrate improved accuracy preservation over existing sparsification methods and practical decoding acceleration on both desktop and edge GPUs.

Critical strengths

Lazy shift addresses a concrete limitation of applying activation recentering to gated FFNs. The proposed decomposition allows both Up and Down projections to exploit the resulting sparsity. Table 4 demonstrates substantial standalone gains from this component and complementary improvements from weight-aware rotation.

The paper connects algorithmic improvements to practical execution. It reports measured decoding acceleration on two hardware platforms, accuracy–speedup comparisons against TEAL, and ablations covering correction fusion, parallel execution, and LM-head sparsification. The discussion of additional memory and low-bit quantization limitations is also useful.

Critical weaknesses

RoShift and the baselines use different calibration datasets and search procedures. Consequently, the accuracy comparisons do not clearly separate the benefits of the proposed transformations from those of calibration data and configuration optimization.

The effects of B128 blockwise selection and approximate correction fusion are mainly evaluated through PPL on Llama-3.1-8B. Their downstream accuracy impact across the five models is insufficiently established.

Throughput is measured using only four fixed teacher-forced continuations, each containing 128 decoding steps. This limited workload does not adequately characterize input-dependent correction fallbacks or latency variability during free-running generation.

Questions for authors

How do the accuracy comparisons change when all methods use the same calibration data and comparable search budgets?

Do the accuracy results in Figure 4 use the exact timed configuration, including B128 selection and approximate correction fusion? How much accuracy do these approximations lose relative to global Top-K and exact correction?

How do throughput and correction-fallback rates vary across prompts and longer free-running generations?

How does the approximately 6 GiB of additional matrix storage translate into the reported 29.31% peak-VRAM increase? What are the absolute peak-memory values for Dense and RoShift?

4 comments

Thank you for your constructive feedback. We address your comments and questions below.

Q1: How do the accuracy comparisons change when all methods use the same calibration data and comparable search budgets?

When all compared methods were evaluated using the same calibration dataset as RoShift, RoShift consistently maintained its accuracy advantage.

Since RoShift's sparsity allocation search builds upon WAS, which itself extends TEAL's allocation strategy, these methods have different search spaces and optimization procedures, making it challenging to directly match their search budgets. We therefore compared the average accuracy across 11 benchmarks on five models at 60% uniform sparsity, eliminating differences arising from sparsity allocation search. The table below presents the results using RoShift's mixed calibration dataset for all methods. WINA was excluded from this comparison because it does not require calibration under uniform sparsity.

MethodLlama3.1-8BLlama2-7BMistral-7BQwen2.5-7BQwen3-8BAvg
Dense63.8747.1459.6773.2876.4364.08
TEAL49.20 (-14.67)39.54 (-7.61)51.09 (-8.58)64.37 (-8.91)68.81 (-7.62)54.60 (-9.48)
LaRoSA50.46 (-13.41)41.01 (-6.13)50.42 (-9.25)63.03 (-10.25)65.60 (-10.83)54.10 (-9.98)
RoShift57.06 (-6.81)43.31 (-3.83)56.33 (-3.34)67.37 (-5.91)71.40 (-5.03)59.09 (-4.99)

At 60% uniform sparsity, RoShift achieved average accuracy improvements of 4.49 pp over TEAL and 4.99 pp over LaRoSA across the five models. Additionally, we repeated this comparison on Llama-3.1-8B at 60% non-uniform sparsity under two calibration settings: (1) using each method’s original calibration dataset and (2) using RoShift’s calibration dataset for all methods. The results are shown below.

MethodOriginal CalibrationRoShift Mixed CalibrationChange (pp)
Dense63.8763.87-
TEAL53.44 (-10.43)52.75 (-11.12)-0.69
LaRoSA47.49 (-16.38)51.39 (-12.48)+3.90
WINA50.62 (-13.25)51.03 (-12.84)+0.41
RoShift58.54 (-5.33)58.54 (-5.33)0.00

Notably, despite the substantial improvement in LaRoSA's accuracy, RoShift maintained an accuracy of 58.54%, outperforming LaRoSA by 7.15 pp and TEAL by 5.79 pp. These results demonstrate that RoShift retains its accuracy advantage even when all methods use the same calibration dataset.

Q2: Do the accuracy results in Figure 4 use the exact timed configuration, including B128 selection and approximate correction fusion? How much accuracy do these approximations lose relative to global Top-K and exact correction?

The accuracy results in Figure 4 use the same configuration as the decoding-speed measurements, including both B128 Blockwise Top-K and Approximate Correction Fusion. These two optimizations also have minimal impact on benchmark accuracy.

To evaluate the accuracy impact of these two optimizations, we compared the configuration with both optimizations enabled against Global Top-K + Exact Correction across 11 benchmarks at 60% uniform sparsity. As a result, the average accuracy across the five models decreased by only 0.23 percentage points, from 59.32% to 59.09%. Detailed results are presented in the table below.

MethodLlama-3.1-8BLlama-2-7BMistral-7BQwen2.5-7BQwen3-8BAvg
Dense63.8747.1459.6773.2876.4364.08
B128 + Approximate Correction57.06 (-6.81)43.31 (-3.83)56.33 (-3.34)67.37 (-5.91)71.40 (-5.03)59.09 (-4.99)
Global Top-K + Exact Correction57.42 (-6.46)43.78 (-3.36)56.22 (-3.45)67.57 (-5.71)71.60 (-4.83)59.32 (-4.76)

Q3: How do throughput and correction-fallback rates vary across prompts and longer free-running generations?

RoShift maintains consistent decoding speedup and stable fallback rates as the generation length increases up to 2K tokens.

We evaluated free-running generation on 16 prompts, the four original prompts plus three additional prompts from each of the four generation benchmarks used in our paper. All speedups are measured relative to the dense baseline. All decoding speeds were measured on an RTX 5090 GPU at a batch size of 1 and 50% uniform sparsity.

Generated Tokens1282565121,0242,048
Speedup1.57×1.58×1.57×1.56×1.56×
Fallback5.52%6.39%6.44%7.48%7.30%

The reported results are averaged over three repeated measurements for each configuration. As the generation length increases, RoShift maintains a stable decoding speedup of 1.56-1.58×, while the average fallback rate also remains below 10%, ranging from 5.52% to 7.48%.

Please note that we used teacher-forced continuations in the original paper because it is difficult to precisely control TEAL's achieved sparsity during free-running generation.

Q4: How does the approximately 6 GiB of additional matrix storage translate into the reported 29.31% peak-VRAM increase? What are the absolute peak-memory values for Dense and RoShift?

The measured peak memory usage for all five models is reported in the table below. Measurements were conducted with batch size 1, a prefill length of 4K tokens, one decoding step, and 50% uniform sparsity for RoShift.

Memory MetricLlama3.1-8BLlama2-7BMistral-7BQwen2.5-7BQwen3-8B
Dense Weights (GiB)14.9612.5513.4914.1915.26
A: Dense Peak (GiB)20.4819.6219.0118.8620.85
RoShift Peak (GiB)26.4925.6325.0222.8727.60
B: Additional Memory (GiB)6.006.006.004.016.75
B/A: Increase (%)29.31%30.59%31.57%21.27%32.40%

For Llama-3.1-8B, the reported 29.31% increase is calculated relative to the Dense model's peak memory usage of 20.48 GiB.

We have incorporated these additional experiments and analyses into Appendices 9.5, 9.6, 13 and 14 of the revised manuscript.

Thank you for the additional experiments. The shared-calibration comparison and the accuracy evaluation of the timed configuration across five models address my main concerns. The extra VRAM remains a practical limitation, and a direct comparison with TEAL during free-running generation would strengthen the efficiency claims. I am maintaining my rating of 3.

We sincerely thank you for your thoughtful and constructive feedback. We are pleased that our additional experiments and revisions have addressed your main concerns.

We appreciate your comments on the additional VRAM overhead and the comparison with TEAL under free-running generation. We acknowledge that the increased VRAM usage is a practical limitation. At the same time, we would also like to note that RoShift’s additional memory overhead is adjustable, allowing a trade-off among memory consumption, accuracy, and inference performance. We present the comparison with TEAL under free-running generation below, where RoShift retains its speed advantage over TEAL, consistent with the results obtained under teacher-forced continuation. We will further clarify both of them in the revised manuscript.

The table below presents the free-running generation results of TEAL and RoShift under the same experimental settings as our previous evaluation. All experiments were conducted on an RTX 5090 GPU with a batch size of 1, using 16 prompts. All speedups are measured relative to the dense baseline.

Decode Length1282565121,0242,048
Target Sparsity50.00%50.00%50.00%50.00%50.00%
TEAL Actual Sparsity48.77%48.99%49.58%50.29%51.01%
TEAL Speedup1.49×1.50×1.51×1.52×1.53×
RoShift Speedup1.57×1.58×1.57×1.56×1.56×

RoShift retains decoding speedups comparable to or higher than TEAL. Please note that, since TEAL uses fixed thresholds, its actual sparsity can vary depending on the input prompts and generated tokens. In this results, TEAL's actual sparsity deviated from the target sparsity by up to 1.23 percentage points (pp).

We would also like to briefly clarify why we used teacher-forced continuations in our original draft. Ideally, decoding speed should be measured on all inputs used for accuracy evaluation (free-running generation benchmarks). However, due to the large number of evaluation prompts and the computational cost of decoding-speed measurements (batch size 1), speed is generally evaluated using a limited set of representative prompts. Prior work, including TEAL and LaRoSA, also measured decoding speed using a limited number of inputs rather than the entire accuracy evaluation set.

To ensure a fair comparison between accuracy and decoding speed in our original draft, we adopted teacher-forced continuations and calibrated TEAL's thresholds so that the actual sparsity achieved during speed measurements matched that observed in the accuracy benchmarks. In free-running generation, however, the generated sequences may differ, so the calibrated thresholds do not necessarily guarantee the intended sparsity.

We plan to incorporate these experimental results and analyses into the revised manuscript.

Thank you again for your constructive feedback and consideration!

Comments on the paper

1 comment

We sincerely thank the reviewers for the careful reading of our paper and the thoughtful, constructive feedback. The comments have been very helpful in improving the clarity and rigor of our work.

We have provided detailed explanations and additional experimental results in response to each reviewer’s questions. We hope these responses address their concerns. We have also incorporated the additional experiments and analyses into the revised manuscript, primarily in the appendix, to further strengthen the paper.

We have highlighted the changes made to the revised manuscript relative to the original submission in blue.

In the main text, we clarified the definitions of WA-Top-K and the inter-layer transition matrix, expanded the mathematical derivation and statistical analysis of activation- and weight-aware rotation, and added a comparison with PCA (Section 3.3 and Table 1). We also clarified the experimental settings of the ablation study (Section 4.6) and the rationale for selecting decoding-speed baselines (Section 4.1).

In addition, we have added the following Appendix sections in response to the reviewers' feedback.

  • Appendix A.8.2: RMS Tolerance and Fallback Implementation
  • Appendix A.9.5: Accuracy Impact of B128 Selection and Approximate Correction
  • Appendix A.9.6: Baseline Comparisons with Shared Calibration Data
  • Appendix A.9.7: Correction-Inclusive MAC-Budget Comparison
  • Appendix A.11: LaRoSA Comparison and Shift Correction Analysis
  • Appendix A.12: End-to-End Accuracy–Speedup Trade-offs under Different LM-Head Settings
  • Appendix A.13: Peak GPU Memory Usage
  • Appendix A.14: Decoding Speed and Fallback in Free-Running Generation

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.