- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This paper proposed to use a low rank affine map to compensate the lost info of contextual sparsity
- Different from existing approaches which neglects the discarded neurons, this work tries to compensate the lost info by reconstructing the discarded output.
- Relevant literatures has applied similar recompensation in the quantization literature to compensate quantization loss, which motivates this paper.
- Experimental results show that the proposed method outperforms other comparison methods in tight budget settings.
Critical strengths
- The paper proposed an interesting problem to study instead of using dense MLP layers for prediction, one can use low rank predictions and compensators to reconstruct the output. The idea is novel and worth exploration.
- Experimental results show that the proposed method outperforms other comparison methods in tight budget settings.
Critical weaknesses
- Real world latency is not only determined by FLOPs, but more importantly could we fuse kernels to better use GPU. The introduced branches likely make the deployment more complicated even though the configuration is iso-flops.
- More refinement on writing to make it easier to follow would be helpful
- At higher budget B, it loses significantly to other comparison methods in Figure 2 for PPL and Figure 4 for MMLU. Any explanation why it loses
- Would be good to have a method comparison section to compare the methodology difference with baselines
Questions for authors
- What’s the difference between comparison methods of TEAL vs the proposed methods?
- While measuring PPL is an interesting metric, it would be good to show how does the method compare on various downstream tasks?
4 comments
Gentle Reminder
Dear Reviewer BnZy,
As the discussion period is nearing its end, we kindly ask if you could consider whether our newly added content and experiments have addressed your concerns. If so, we humbly ask that you re-evaluate our work. As for our writing, Section 4 has been rewritten and reorganized, and all revised portions of our manuscript are marked in blue. We are happy to provide any further clarifications that might be helpful to you. Thank you again for your time and consideration.
Sincerely,
Authors
Thanks for doing the extra experiments on latency, and the quality comparison on downstream tasks. It addressed some of my main concerns. I am happy to raise the score to 3.
Response to Reviewer BnZy
Thank you again for your time and constructive feedback. We very much appreciate the opportunity to improve our work based on your thoughts and review.
Response to Reviewer BnZy
We thank the reviewer for finding the idea novel and worth exploring. We have addressed each one of your concerns below.
Thank you for highlighting this important point. We have built and measured a fused kernel (Section 3.7). Each layer runs as a single kernel launch, where the predictor scores the neurons, a 4096-bin histogram picks the top-k without sorting, the selected rows are gathered, and then the compensator is applied. The compensator runs inside that launch as q linear neurons that are always on, so it adds no separate branch. Each decoding step is one CUDA graph, and ReComp replaces the MLP module without any other changes. On end-to-end decoding, this fused kernel is 1.34–1.45× faster than dense at B=0.25 on five 7–14B models, with perplexity within 0.1% of the reference (Appendix A.1).
Thanks for the great question. At higher budget B, ReComp loses to other comparison methods because of selection, not reconstruction (Sections 4.3–4.4). The following table, the relative WikiText-2 perplexity of Qwen2.5-7B at B=0.50 demonstrates this:
Since B getting larger means more neurons needing to be selected, which neurons are kept matters more than what is being reconstructed, and our cheap predictor selects worse than exact magnitudes. Our compensator is not the limit since on TEAL's selection, it is the best method in the row, and a method with our compensator is best in 11 of 12 model–budget rows on the 7–8B models (Table 1). This is the same for MMLU, since oracle top-k is within 1.7 points of dense on Qwen2.5-0.5B at B=0.35. Conversely, the advantage grows as B shrinks, since at B=0.10 on Qwen2.5-7B, ReComp reaches 1.31 against 9.7–10.5 for the baselines.
Thank you for the thoughtful suggestion and question. We have added a method comparison section in Section 4.1 and a method table.
Our co-design of a selector and reconstructor works since a low-rank selector misses neurons as a systematic function of x, so its omitted output is 1.4–2.3× more predictable than TEAL's (Section 4.4). Additionally, at equal omitted energy the compensator removes 40% of it against 17% of TEAL's.
Thanks for the important question. We have now made all four 7–8B models have six zero-shot tasks and MMLU at every budget. At B=0.25, ReComp has the best six-task average on three of four models and is best on HellaSwag, PIQA and WinoGrande on all four, though it loses for MMLU on three. Generation with dense verification is lossless (GSM8K, MATH-500, and AIME benchmarks).
We hope our responses address your concerns, and we would be grateful to answer any further questions.