- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RoShift, which combines lazy shift with activation- and weight-aware rotation. Lazy shift recenters post-SiLU activations and defers the corresponding correction to the end of the FFN, creating additional sparsity that both the up and down projections can exploit. Activation- and weight-aware rotation jointly incorporates activation statistics and weight information to construct rotations that reduce sparsification-induced output error. RoShift further fuses the associated correction computations to reduce overhead and employs efficient sparse kernels to translate these computational savings into practical decoding acceleration.
Critical strengths
-
Lazy shift offers a concrete improvement to gated FFNs, supported by direct ablation evidence. It recenters post-SiLU activations and reformulates the corresponding correction as a linear computation with precomputable weights, allowing both the up and down projections to exploit the resulting sparsity.
-
The quality improvements at high sparsity are consistently supported across models and tasks. Table 2 and the appendices cover 5 models and 11 tasks, with relevant baselines including TEAL, LaRoSA, and WINA. At target sparsity levels of 50% and 60% with non-uniform allocation, RoShift improves average accuracy over TEAL by 1.60 and 3.29 percentage points, respectively, outperforming it in 50 and 54 of the 55 model-task combinations. These results indicate that the gains are not primarily driven by isolated models or tasks and support the claim that RoShift better preserves model quality under aggressive sparsification.
-
The paper translates its methodological improvements into measured decoding acceleration and provides evidence of the accuracy-speed trade-off. RoShift addresses the overhead introduced by representation transfromations through correction-transition fusion, caching and sparse kernels.
Critical weaknesses
-
The paper contains several redundant or inconsistent definitions. What exactly does "WA-Top-K" in Table 4 denote? This operation is not introduced in the preceding method section. What distinguishes the full RoShift method drom the "Lazy shift + WA-Top-K" configuration? Similarly, line 251 defines as the inter-layer transition matrix: does this refer to ? Please clarify these definitions and use consistent mathematical notation throughout the paper.
-
Why is LaRoSA omitted from the speedup comparison? Compared with LaRoSA’s rotation-based approach, RoShift introduces additional computation for shift correction, yet the paper does not clearly quantify its impact on decoding speed. Could the authors include LaRoSA as a baseline under matched experimental conditions and provide a controlled ablation that isolates the latency overhead of shift correction and the extent to which the proposed fusion mitigates it?
-
How is the condition that "the relative difference between the actual RMS and its representative is within 5%" enforced in Correction Fusion? My understanding is that checking this condition and handling cases that exceed the tolerance require dedicated kernel-level logic, but I could not find these implementation details in the appendix. Could the authors clarify how this mechanism is implemented?
-
The description of activation- and weight- rotation is difficult to follow and relies heavily on the appendix, leaving the main text insufficiently self-contained. At line 220, although the single-axis output-contribution variance, $e_q = q Sq $, provides a meaningful motivation, the mixed-statistics eigenbasis is not an exact solution to this objective, nor does the derivation establish specific distributional properties of the rotated activations. What statistics does this rotation change relative to PCA, and how do those changes reduce actual sparsification error? The paper lacks paired comparisons under identical inputs, such as covariance structure, cumulative variance or energy, near-zero probability, or output-contribution distributions. Table 4 supports the empirical advantage of AW-Rot over A-Rot, but higher accuracy alone does not explain the statistical mechanism behind that advantage.
-
After introducing the additional correction weights for recentering, my understanding is that RoShift’s effective computational sparsity should account for operations absent from TEAL and LaRoSA. However, I could not find a clear explanation of how this overhead is incorporated into the sparsity calculation. Specifically, at the nominal 40% sparsity reported in Table 1, what are the actual effective sparsity levels of TEAL, LaRoSA, and RoShift when all additional computations are included?
Questions for authors
-
Could the autors clarify the definition of WA-Top-K, the difference between full RoShift and "Lazy shift + WA-Top-K", and whether T_l = Q_l ^\top Q_{l+1}? Please use consistent notation and introduce all components explicitly in the main text.
-
Could the authors include LaRoSA in the decoding-speed comparison under matched hardware, workloads, and sparsity settings? Please separately quantify the latency overhead introduced by shift correction and the overhead reduction achieved through correction fusion.
-
How is the 5% relative RMS tolerance enforced in the implementation? Please explain how representatives are selected, how the condition is checked, what fallback is used when the tolerance is exceeded, and whether these operations are included in the reported latency.
-
Could the authors provide a self-contained description of how the activation- and weight-aware rotation is contructed and explain its statistical advantages over PCA? Please provide paired diagnostics under identical inputs and sparsification settings to show which statistics change the how those changes relate to reduced sparsification error, beyond the accuracy comparisons in Table 4.
-
How is effective computational sparsity defined after accounting for shift correction, rotation transitions, and other additional operations? At the reported nominal 40% sparsity, what effective spasity does each of TEAL, LaRoSA, and RoShift achieve under a common accounting convention? Please specify the fomula, denominator, and included operations.
I would appreciate the authors' responses to these questions and will consider my final score in light of their clarifications.
5 comments
Q3: How is the 5% relative RMS tolerance enforced in the implementation? Please explain how representatives are selected, how the condition is checked, what fallback is used when the tolerance is exceeded, and whether these operations are included in the reported latency.
During inference, we compare the actual RMS value against all representatives and select the closest one. We then compute the relative error between the actual RMS value and the selected representative and set a Boolean flag if the error exceeds 5%, indicating that fallback is required.
Based on this Boolean flag, tl.where selects the pointer to either the precomputed Fused Matrix or the Transition Matrix, while the mask in tl.load ensures that the Correction Matrix is additionally loaded only when fallback occurs. All these operations are handled within a single Triton kernel.
All these operations are included in the reported end-to-end decoding latency. We have added this explanation and further implementation details to Appendix 8.2 in the revised manuscript.
Q4: Could the authors provide a self-contained description of how the activation- and weight-aware rotation is contructed and explain its statistical advantages over PCA? Please provide paired diagnostics under identical inputs and sparsification settings to show which statistics change the how those changes relate to reduced sparsification error, beyond the accuracy comparisons in Table 4.
To better explain the statistical mechanism of Activation- and Weight-Aware Rotation (AW-Rot), we conducted additional experiments comparing No Rotation, Activation-only Rotation (PCA), and AW-Rot under identical inputs and sparsification settings.
We evaluated a total of 128 projections (Q/K/V/Gate across all 32 layers) of Llama3.1-8B. We used the same 2,048 tokens sampled from the mixed calibration dataset and applied 50% sparsity with B128 Weight-aware Top-K to both methods. Lazy Shift was disabled to isolate the effect of rotation.
In the table below, Leading 50% refers to the first half of the coordinates, ordered by eigenvalues for PCA and AW-Rot, and by their original order for No Rotation. These are distinct from the coordinates actually selected by Top-K. Output NMSE (Normalized Mean Squared Error) is calculated as the squared output error normalized by the squared norm of the dense output for each projection, then averaged across all 128 projections.
| Diagnostic (50% Sparsity) | No Rotation | PCA | AW-Rot |
|---|---|---|---|
| Leading 50% Activation Variance | 49.51% | 83.69% | 80.49% |
| Leading 50% Weight Energy | 49.97% | 58.44% | 78.92% |
| Leading 50% Output Contribution Variance | 49.59% | 90.77% | 94.85% |
| Removed Contribution Proxy | 4.41% | 1.62% | 0.89% |
| Output NMSE (%) | 2.11% | 1.33% | 0.70% |
The results show that AW-Rot does not concentrate activation variance more effectively than PCA, but it achieves greater concentration of weight energy and output contribution variance. Furthermore, the removed output contribution proxy decreased from 1.62% to 0.89%, while the actual output NMSE decreased by 47.75% compared with PCA.
PCA constructs its rotation to maximize activation variance concentration without considering the corresponding weight energy. In contrast, AW-Rot constructs its rotation by jointly considering activation variance and weight energy. As a result, we observed that output contributions became more concentrated in the leading axes, accompanied by reductions in both the removed output contribution proxy and the actual sparsification error.
We have also clarified the construction of AW-Rot and its statistical differences from PCA in Section 3.3 of the revised manuscript. We would appreciate it if the reviewer could refer to the revised section.
Q5: How is effective computational sparsity defined after accounting for shift correction, rotation transitions, and other additional operations? At the reported nominal 40% sparsity, what effective spasity does each of TEAL, LaRoSA, and RoShift achieve under a common accounting convention? Please specify the fomula, denominator, and included operations.
We define effective computational sparsity based on the number of GEMV multiply-accumulate (MAC) operations performed in the seven Transformer linear projections, the LM-head, and the additional transition and correction matrix operations. Non-GEMV operations, such as attention score computation and RMS normalization, are excluded.
The denominator of effective computational sparsity is the total number of GEMV MACs performed by the Transformer projections and LM-head in the dense model. The formula is defined as follows:
In RoShift, correction fusion combines the transition and correction operations into a single GEMV. However, when the RMS tolerance condition is not satisfied, the fallback executes the two operations separately, introducing additional GEMV MACs proportional to the fallback rate.
RoShift's fallback rate is below 10% in most cases. Assuming a fallback rate of 10%, the effective computational sparsity of each method at 40% nominal sparsity is shown below for each model.
| Method | Llama-3.1-8B | Llama-2-7B | Mistral-7B | Qwen2.5-7B | Qwen3-8B |
|---|---|---|---|---|---|
| TEAL | 37.20% | 39.21% | 39.26% | 36.92% | 36.71% |
| WINA | 37.20% | 39.21% | 39.26% | 36.92% | 36.71% |
| LaRoSA | 30.05% | 31.08% | 31.71% | 31.83% | 28.73% |
| RoShift (Dense LM Head) | 29.33% | 30.27% | 30.96% | 31.32% | 27.93% |
| RoShift (Sparse LM Head) | 35.14% | 31.77% | 32.28% | 37.11% | 33.81% |
Please note, however, that effective computational sparsity does not directly reflect actual GPU execution efficiency. RoShift improves GPU execution efficiency by reusing Up output sparsity in Down, maintaining a fixed workload per block through Blockwise Top-K, and executing correction operations in parallel with the sparse FFN. Therefore, RoShift can achieve higher decoding speedups even at the same effective computational sparsity.
Thank you for your constructive feedback. We address your comments and questions below.
Thank you for pointing out the unclear definitions in our paper.
WA-Top-K stands for Weight-Aware Top-K, which selects components based on the importance score ∣Xi∣⋅∥Wi,:∥2, considering both activation magnitudes and the corresponding weight norms. However, for Up output sparsity, we use standard Top-K instead.
Lazy Shift + WA-Top-K is an ablation configuration of RoShift that excludes only the rotation component.
Transition Matrix Tl is indeed defined as Tl=Ql⊤Ql+1, as the reviewer pointed out.
We have revised several parts of the manuscript to clarify these definitions and ensure consistent mathematical notation throughout the paper.
Although LaRoSA reports end-to-end decoding speedups in its paper, its publicly released code contains only PyTorch code for accuracy evaluation and does not include implementations for decoding acceleration, such as Sparse GEMV kernels. We therefore omitted LaRoSA from the original speedup comparison, as noted in the original manuscript. We have clarified this rationale in the revised manuscript.
To address the reviewer’s concern, we implemented LaRoSA’s sparsification method (denoted by ‘LaRoSA’) using RoShift's GPU kernel backend and measured its decoding latency. However, while RoShift's kernel supports only B128 Blockwise Top-K, LaRoSA uses a kernel that fuses Top-K selection with sparse GEMV in its original paper. Furthermore, differences may exist in sparse mask generation and GEMV implementation. Therefore, the measured ‘LaRoSA’ performance below may differ from that of its original optimized implementation.
To separately quantify the overhead introduced by shift correction and the reduction achieved through correction fusion, we evaluated RoShift under the following configurations. For this ablation, all configurations use a dense LM-head.
The following table presents the end-to-end decoding latency measured under the same experimental conditions: RTX 5090 GPU, batch size 1, 50% uniform sparsity, and a dense LM-head. All values are reported in ms/token, where lower is better.
With the final Fused Parallel configuration, RoShift achieved lower decoding latency than LaRoSA on four out of five models.
Shift Correction Overhead and Fusion Effects
The table above quantifies the overhead of shift correction, the latency reduction from fusing correction and transition operations, and the additional benefit of multi-stream parallel execution.
Executing shift correction separately introduces an additional overhead of 0.49–0.76 ms/token. However, correction fusion reduces latency by 0.41–0.71 ms/token, mitigating most of this additional overhead. Furthermore, multi-stream parallel execution provides an additional latency reduction of 0.19–0.24 ms/token.
Overall, although RoShift adds computation for shift correction, correction fusion and parallel execution mitigate most of the overhead. As a result, RoShift achieves competitive decoding latency compared with LaRoSA implemented under the shared kernel backend.
We have also incorporated the above experimental results and analysis into Appendix 11 of the revised manuscript.