- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a simple yet effective method, TACT, for accelerating the LLM inference during decoding. The main idea is to use the sparsity of the activation so that dropping or compressing columns of the weight matrix would not introduce much error. Specifically, the method considered tiered rules that keep the largest columns, compress the medium ones, and drop the smallest ones. The size for each tier is decided via searching on a calibration set, and the medium columns are collectively compressed via Trellis-coding. In experiments, their methods show superior budget-performance trade-off than other baselines.
Critical strengths
- The major strength lies in the novelty of the proposed method. Though simple, this method is plausible and effective, which makes sense to me.
- The performance of the proposed method is also good. When the budget is about half of the dense one, the perplexity increase is usually around 5%, which I believe is a satisfactory trade-off. The model size in the experiment is not small either.
Critical weaknesses
- I think the performance degradation is still obvious, particularly that of the downstream GSM8K task. I suggest the authors report settings where acceleration may be smaller but the performance is much closer to the dense model.
- It seems there is no experiment for MoE models, with dense models only. It raises concerns whether the proposed method could still perform well in current popular models that mostly apply the MoE structure.
Questions for authors
See Weakness.
4 comments
Thank you for the response.
Thank you for the provided results and response, which has moderately solved my concerns. In addition, I have another suggestion that, by comparing accuracy over multi-trials, like from Pass@1 to Pass@K, it may be clearer to show how TACT transfers the advantage in budget into real performance gains. For example, if your budget is 50% that of the baseline's, it is fair to compare Pass@2 acc of TACT with Pass@1 acc of the dense baseline. I believe that can make TACT's advantage much clearer rather than a trade-off result.
We thank the reviewer for the follow-up and for this suggestion. We ran the suggested comparison on LLAMA-ALL-BF16 at the 10% tolerance of Table 2.
With CPU memory offloading, TACT decodes about 2.9× faster than the dense model, so in about the time the dense model takes to answer a GSM8K problem once, TACT can answer it two or three times. We therefore sampled answers from TACT and compared its pass@1 to pass@3 with the dense model's pass@1:
| Answers per problem | Time per problem (vs. dense) | GSM8K | |
|---|---|---|---|
| BF16 (Dense) | 1 | 1.00 | 50.2 (pass@1) |
| TACT | 1 | 0.35 | 43.9 (pass@1) |
| TACT | 2 | 0.69 | 55.0 (pass@2) |
| TACT | 3 | 1.04 | 62.9 (pass@3) |
In this comparison, TACT's two answers solve more problems than the dense model in no more time than one dense answer, so TACT's smaller budget turns into higher accuracy rather than a trade-off.
We hope this makes TACT's advantage clearer, as the reviewer suggested, and we will explain this comparison more comprehensively in future work. We thank the reviewer again for the suggestion.
Thank you for the additional results. I have no further questions.
We thank the reviewer for their feedback and comments. We address their feedback below.
Measure at settings with very low PPL loss (Weakness 1). TACT's budget sets how many weight bytes a token reads, so TACT runs closer to dense by reading more of them. We therefore added a setting on LLAMA-ALL with a perplexity increase of 2% or less to Appendix I (Table 14):
In this setting, TACT stays within 2 points of dense on GSM8K, where it loses 6 to 9 points at the 10% tolerance. On the other tasks, it stays within 1 point. TEAL needs a 75% budget for similar accuracy.
Mixture-of-experts models (Weakness 2). TACT also requires a smaller budget than TEAL on a mixture-of-experts model. We added Gemma-4-26B-A4B in BF16, which routes each token to 8 of its 128 experts. Each expert is a gated FFN, so TACT splits its inputs and neurons into the three tiers as in the dense FFN. The budget is relative to the weights a token reads when no weights are skipped, which include the 8 routed experts but not the other 120. At the 10% tolerance, TACT requires a 19% budget, against 55% for TEAL. At the 20% tolerance, the budgets are 15% and 49%. We have discussed this in detail in Appendix J.
We hope these responses address your concerns, and we would be glad to answer any further questions.