acceptodds
Under review as a conference paper at ICLR 2027

TACT: Tiered Activation Sparsity

Abstract

Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repeatedly transfers offloaded weights from system RAM or flash storage into GPU at much lower bandwidth than local GPU-memory access. Activation sparsity reduces these transfers by skipping weights associated with zero or near-zero activations. However, as more activation contributions are omitted, model quality eventually degrades rapidly, indicating that weights associated with small-magnitude activations collectively influence model quality sharply. In this work, we improve the trade-off between model quality and decoding performance when exploiting activation sparsity. Our key idea is to replace the binary choice of whether or not to read a weight with three options: fully retain it, approximate it using a compressed weight representation, or omit it entirely. TACT skips weights associated with activations closest to zero, reads approximate weights for smaller-magnitude activations, and reads original weights for the largest-magnitude activations. Smaller-magnitude activations attenuate the errors introduced by approximate weights, while compressed weight representations require fewer bytes to be transferred. With weights offloaded to CPU memory, \method speeds up decoding by on average (up to ) with 16-bit weights and (up to ) with 4-bit weights, at a WikiText-2 perplexity increase of at most 10%. With weights offloaded to flash storage, the speedups are on average (up to ) and (up to ).

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Activation Sparsity, Deep Learning Performance, Model Compression, LLMs, On-device Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GuuLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a simple yet effective method, TACT, for accelerating the LLM inference during decoding. The main idea is to use the sparsity of the activation so that dropping or compressing columns of the weight matrix would not introduce much error. Specifically, the method considered tiered rules that keep the largest columns, compress the medium ones, and drop the smallest ones. The size for each tier is decided via searching on a calibration set, and the medium columns are collectively compressed via Trellis-coding. In experiments, their methods show superior budget-performance trade-off than other baselines.

Critical strengths
  1. The major strength lies in the novelty of the proposed method. Though simple, this method is plausible and effective, which makes sense to me.
  2. The performance of the proposed method is also good. When the budget is about half of the dense one, the perplexity increase is usually around 5%, which I believe is a satisfactory trade-off. The model size in the experiment is not small either.
Critical weaknesses
  1. I think the performance degradation is still obvious, particularly that of the downstream GSM8K task. I suggest the authors report settings where acceleration may be smaller but the performance is much closer to the dense model.
  2. It seems there is no experiment for MoE models, with dense models only. It raises concerns whether the proposed method could still perform well in current popular models that mostly apply the MoE structure.
Questions for authors

See Weakness.

4 comments

We thank the reviewer for their feedback and comments. We address their feedback below.

Measure at settings with very low PPL loss (Weakness 1). TACT's budget sets how many weight bytes a token reads, so TACT runs closer to dense by reading more of them. We therefore added a setting on LLAMA-ALL with a perplexity increase of 2% or less to Appendix I (Table 14):

BudgetPPLArc-CHSPIQAWGMMLUGSM8K
BF16 (Dense)100%–52.160.879.175.165.450.2
TEAL75%+1.5%52.160.479.173.065.048.7
TACT50%+2.0%51.660.479.074.264.648.4
Q4_K_M (Dense)100%–51.460.078.773.664.748.6
TEAL75%+1.5%51.659.778.772.564.245.5
TACT64%+1.6%51.059.678.772.964.346.7

In this setting, TACT stays within 2 points of dense on GSM8K, where it loses 6 to 9 points at the 10% tolerance. On the other tasks, it stays within 1 point. TEAL needs a 75% budget for similar accuracy.

Mixture-of-experts models (Weakness 2). TACT also requires a smaller budget than TEAL on a mixture-of-experts model. We added Gemma-4-26B-A4B in BF16, which routes each token to 8 of its 128 experts. Each expert is a gated FFN, so TACT splits its inputs and neurons into the three tiers as in the dense FFN. The budget is relative to the weights a token reads when no weights are skipped, which include the 8 routed experts but not the other 120. At the 10% tolerance, TACT requires a 19% budget, against 55% for TEAL. At the 20% tolerance, the budgets are 15% and 49%. We have discussed this in detail in Appendix J.

We hope these responses address your concerns, and we would be glad to answer any further questions.

Reviewer_GuuLOpenReview ↗

Thank you for the response.

Thank you for the provided results and response, which has moderately solved my concerns. In addition, I have another suggestion that, by comparing accuracy over multi-trials, like from Pass@1 to Pass@K, it may be clearer to show how TACT transfers the advantage in budget into real performance gains. For example, if your budget is 50% that of the baseline's, it is fair to compare Pass@2 acc of TACT with Pass@1 acc of the dense baseline. I believe that can make TACT's advantage much clearer rather than a trade-off result.

We thank the reviewer for the follow-up and for this suggestion. We ran the suggested comparison on LLAMA-ALL-BF16 at the 10% tolerance of Table 2.

With CPU memory offloading, TACT decodes about 2.9× faster than the dense model, so in about the time the dense model takes to answer a GSM8K problem once, TACT can answer it two or three times. We therefore sampled answers from TACT and compared its pass@1 to pass@3 with the dense model's pass@1:

Answers per problemTime per problem (vs. dense)GSM8K
BF16 (Dense)11.0050.2 (pass@1)
TACT10.3543.9 (pass@1)
TACT20.6955.0 (pass@2)
TACT31.0462.9 (pass@3)

In this comparison, TACT's two answers solve more problems than the dense model in no more time than one dense answer, so TACT's smaller budget turns into higher accuracy rather than a trade-off.

We hope this makes TACT's advantage clearer, as the reviewer suggested, and we will explain this comparison more comprehensively in future work. We thank the reviewer again for the suggestion.

Thank you for the additional results. I have no further questions.

Reviewer_nTmMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TACT, a training-free activation sparsity method for LLM inference with weights offloaded to CPU memory or flash storage. TACT introduces three tiers that retain original weights, use 2-bit compressed weights, or skip weight columns based on activation magnitudes. Experiments demonstrate improved accuracy–efficiency trade-offs over existing activation sparsity methods.

Critical strengths
  1. The three-tier approach effectively reduces weight transfers while preserving model quality.
  2. The paper provides comprehensive comparisons across multiple models, weight formats, and offloading configurations.
  3. Detailed latency breakdowns help explain the performance benefits and overheads.
Critical weaknesses
  1. Most experiments offload all targeted weights regardless of GPU memory capacity. Although the CA configuration keeps attention weights GPU-resident, evaluating partial FFN residency under different GPU memory budgets would better demonstrate the practical benefits of TACT.

  2. Figure 6 compares TACT with QTIP using a normalized read budget rather than the actual memory footprint. Since TACT stores both the original Q4_K_M weights and additional 2-bit representations, the methods have different storage requirements. A comparison of memory footprint and end-to-end latency under the same GPU memory constraints would provide a more complete evaluation of their practical efficiency.

  3. Inconsistent speedups with 4-bit weights. Table 3 shows that TACT is slower than TEAL in the LLAMA-ALL-Q4KM flash-offloading setting despite transferring fewer bytes. Although the paper discusses reconstruction and I/O overheads, a clearer analysis of when TACT provides actual speedups would strengthen the evaluation.

Questions for authors
  1. (with Weakness #1) How does the proportion of GPU-resident weights affect TACT's speedup over TEAL?

  2. Could the authors further clarify the advantages of TACT over conventional weight quantization? Since TACT requires storing both the original weights and additional quantized representations, it would be helpful to discuss the trade-off between memory overhead and reduced weight transfer costs.

  3. Although GRINQH [1] has not released its source code, it already proposes activation-dependent weight precision allocation and sparsification. The paper should more clearly distinguish TACT's methodological contributions from GRINQH in Related Work section. Since GRINQH provides some experiment configurations in its appendix, could the authors also consider a small-scale comparison under comparable accuracy or effective-bit budgets? This would help clarify the novelty and benefits of TACT.

2 comments

We thank the reviewer for their feedback and comments. We address their feedback below.

GPU residency (Weakness 1, Question 1). In our new evaluation, part of the FFN weights is held on the GPU at every budget, not only the attention weights as in CA. TACT stays faster than TEAL as more weights are held on the GPU, as long as decoding is bound by weight transfer. Holding weights in GPU memory removes part of the transfer for both methods, but the rest is still transferred, and TACT transfers fewer bytes of it. With BF16 weights, TACT-trellis is 1.31–2.06× faster than TEAL at every budget. With Q4_K_M weights, it is faster up to about half of the weights on the GPU. Beyond that, the remaining transfer is so small that TACT's reconstruction on the GPU outweighs the bytes it saves. Details are in common question CQ2.

Memory footprint and weight quantization (Weakness 2, Question 2). TACT is orthogonal to weight quantization. It runs on a quantized format such as Q4_K_M and reduces the bytes read from it. In exchange, TACT stores a 2-bit copy of the weights alongside the original ones, which takes about 40% more storage for Q4_K_M. This copy is kept in CPU memory or flash, which have far more capacity than GPU memory. Even when this copy shares the GPU-memory budget with the original weights, TACT on Q4_K_M decodes up to 1.7× faster than Q4_K_M alone (please find details in Appendix D). With offloaded weights, decoding speed is set by the bytes read per token, so Figure 6 compares the methods at the same bytes read. Quantizing to fewer bits also reduces the bytes read, but loses more quality. At the budget of 2-bit QTIP (Figure 6), QTIP increases perplexity by 44.8%, against 13.5% for TACT.

Speedups with 4-bit weights (Weakness 3). TACT is faster than TEAL when the transfer time it saves exceeds the time it spends reconstructing the middle tier on the GPU. We added this condition to Section 6.2. On LLAMA-ALL-Q4KM with flash offloading, TACT-trellis reads fewer bytes than TEAL but saves no transfer time. This is because it issues more small reads than TEAL, and flash serves small reads slowly. This limitation can be overcome by caching, as in prior flash-offloading systems. When TACT's 2-bit codes are cached in CPU memory, under the same CPU-memory budget as TEAL, TACT-trellis becomes 1.06× faster than TEAL on LLAMA-ALL-Q4KM. Details are in common question CQ3.

Comparison with GRINQH (Question 3). We reimplemented GRINQH on LLAMA-ALL-Q4KM, choosing its configuration by the procedure in its appendix. At 40% of the Q4_K_M weight bytes, TACT's perplexity increase is 13.5% against 28.9% for GRINQH. This advantage comes from how the two methods read a lower precision: GRINQH drops the low bits of a column's stored code, while TACT reads a separate 2-bit code made for that column. We revised Related Work to state this difference. Details are in common question CQ1.

We hope these responses address your concerns, and we would be glad to answer any further questions.

Thank you for the responses and additional experiments. However, with Q4_K_M weights, the practical setting for offloading, TACT is only about 1.1× faster than TEAL on average at matched perplexity (and slower with flash offloading on LLAMA-ALL), while its additional 2-bit copy requires ~40% more storage. I therefore maintain my score.

Reviewer_odV7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TACT, a training-free method for accelerating LLM decoding with weights offloaded to CPU memory or flash storage. Based on token-dependent activation magnitudes, weight columns are either read in their original deployment format, reconstructed from compressed 2-bit representations, or skipped. The method combines independently decodable column compression, offline tier-size calibration, and storage organization for efficient reads. Experiments on three models and two weight formats demonstrate improved byte–perplexity trade-offs relative to training-free activation-sparsity baselines, with decoding speedups that depend on the weight format and offloading configuration.

Critical strengths
  1. A useful improvement to the byte–quality trade-off. Approximating intermediate-magnitude contributions is a reasonable alternative to discarding them entirely. The reported gains are substantial in BF16: for Llama-3.1-8B with all projections offloaded, TACT requires a 27% read budget versus TEAL’s 57% at approximately matched perplexity degradation. This provides meaningful evidence that tiered reads can improve transfer efficiency.

  2. Implementation and measurement of the relevant system trade-offs. The evaluation includes both CPU-memory and flash offloading, measured decoding latency, and a breakdown of transfer and computation costs. These results demonstrate practical benefits in several configurations and explain why reconstruction overhead and fragmented reads can offset logical byte savings.

Critical weaknesses
  1. GRINQH already uses token-dependent activation magnitudes to select weight-column precision or skip columns. The paper does not clearly establish what TACT adds beyond this principle. The absence of public code explains the missing experiment but does not resolve the uncertainty about novelty.

  2. The experiments offload all targeted weights rather than compare methods under the same realistic GPU-memory budget. Moreover, TACT does not consistently outperform TEAL in Q4_K_M flash settings. For Llama-ALL-Q4KM, TEAL takes 702 ms/token, while TACT-trellis and TACT-scalar take 756 and 773 ms/token, respectively. Measuring only eight decoding steps also leaves the stability of these latency results unclear.

Questions for authors

What are the substantive differences between TACT and GRINQH, and which differences account for TACT’s advantage?

Does TACT outperform TEAL when both methods use the same GPU-memory budget and retain as many weights as possible on the GPU?

Are the reported latency differences stable across multiple prompts, longer generation sequences, and independent runs? Please report latency variability.

Why do the abstract and conclusion claim a perplexity increase of “at most 10%” when Tables 9 and 10 report TACT settings with increases of 11.5% and 11.4%?

3 comments

We thank the reviewer for their feedback and comments. We address their feedback below.

Difference from GRINQH (Weakness 1, Question 1). The difference in approach in GRINQH and TACT is answered in detail in common question CQ1. We have also reimplemented our version of GRINQH and compared it with TACT in Appendix G. At 40% of the Q4_K_M weight bytes, TACT's perplexity increase is 13.5% against 28.9% for GRINQH.

Same GPU-memory budget (Weakness 2, Question 2). Yes, as long as decoding is bound by weight transfer. Holding weights in GPU memory removes part of the transfer for both methods, but the rest is still transferred, and TACT transfers fewer bytes of it. With BF16 weights, TACT-trellis is therefore 1.31–2.06× faster than TEAL at every budget. With Q4_K_M weights, it is faster up to about half of the weights on the GPU. Beyond that, the remaining transfer is so small that TACT's reconstruction on the GPU outweighs the bytes it saves. Details are in common question CQ2.

Q4_K_M with flash offloading (Weakness 2). We agree that TACT-trellis can be slower than TEAL with Q4_K_M weights and flash offloading. This is because it issues more small reads than TEAL, and flash serves small reads slowly. This limitation can be overcome by caching, as in prior flash-offloading systems. When TACT's 2-bit codes are cached in CPU memory, under the same CPU-memory budget as TEAL, TACT-trellis becomes 1.06× faster than TEAL on LLAMA-ALL-Q4KM. Details are in common question CQ3.

Variability in inference latency (Weakness 2, Question 3). Yes, the reported inference latencies are stable over prompt lengths in the same range. To verify this, we re-measured LLAMA-ALL-BF16 and -Q4KM with CPU memory and flash offloading over five WikiText-2 prompts, 32 decoding steps each instead of eight. Each (method, prompt) pair runs as an independent process. Per-token latency varies by at most 1.4% (CV), and the speedup over TEAL varies by at most 0.05 across prompts. The prompts are 32 to 1,536 tokens long, so the measurements also cover decoding with longer contexts. Latency does not grow with context length, because each step reads the same number of weight bytes. Attention over the KV cache, the only cost that grows with context length, takes about 0.1% of a step even at 2K tokens. We added these results to Appendix F (Table 10).

Claim on perplexity increase termed “at most 10%” in introduction, abstract (Question 4). We thank the reviewer for the careful reading. "At most 10%" in the abstract, introduction and conclusion was imprecise wording for what we intended. Each method was measured over a sweep of a set of read budgets, and each budget gives some perplexity increase. Since perplexity is a measured outcome that we cannot set directly, no measured budget lands exactly on 10%. For every method, TEAL included, we report the smallest budget we measured whose perplexity increase is at most 11.5%. To fix this, we replaced "at most 10%" with "about 10% (at most 11.5%)" in the abstract, introduction and conclusion, and stated how each budget is chosen in Appendix A.

We hope these responses address your concerns, and we would be glad to answer any further questions.

Thank you for the additional experiments on GPU memory budgets and latency variability. Could you clarify which design differences account for TACT’s advantage over GRINQH and report the weight-read budget each method requires at comparable perplexity increases within the paper’s 11.5% limit?

I also suggest revising the title to “TACT: Tiered Activation Sparsity for LLMs” to make the scope explicit.

We thank the reviewer for the follow-up.

Design differences that lead to TACT's improvement. TACT quantizes each column directly to two bits, whereas GRINQH reads the top two bits of a four-bit code. TACT's two-bit code therefore approximates the weights more accurately, so both TACT-scalar and TACT-trellis require a smaller budget than GRINQH. We added these results to Appendix G.

Here we report the weight read budget of GRINQH and TACT at comparable perplexity increases. On LLAMA-ALL-Q4KM, the smallest budget we measured whose perplexity increase is at most 11.5% is:

GRINQHTACT-scalarTACT-trellis
Budget52.1%47.3%42.4%
Perplexity increase+11.3%+11.1%+10.6%

Title change. We revised the title to "TACT: Tiered ACTivation Sparsity for LLMs", as suggested.

We hope these answers address the remaining concerns, and we would be glad to answer any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reviews. We are encouraged that all reviewers found TACT's three tiers effective in reducing weight transfers while preserving model quality. Common questions CQ1, CQ2, CQ3 below answer the concerns raised by more than one reviewer. We have highlighted all changes in the revised PDF in red.

CQ1. Difference from GRINQH (Reviewer odV7, Weakness 1 and Question 1; Reviewer nTmM, Question 3).

GRINQH and TACT both use the magnitude of each input activation at each token to decide whether to read its weight column at full precision, at lower precision, or not at all. However, they differ in how the lower precision is obtained. GRINQH lowers a column's precision by dropping the low bits of its stored 4-bit code. TACT instead stores a separate 2-bit code for each column, chosen to minimize the column's error at 2 bits (Section 5.1). TACT's 2-bit code is therefore more accurate than GRINQH's.

Since the code for GRINQH is unavailable, we reimplemented GRINQH on LLAMA-ALL-Q4KM, choosing its configuration by the procedure in its appendix. At 40% of the Q4_K_M weight bytes, TACT's perplexity increase is 13.5% against 28.9% for GRINQH (Appendix G). We also revised the Related Works section to state this difference.

We would also like to point out that GRINQH is concurrent work that is not peer reviewed at the time of submission, and is available only on arXiv without public code.

CQ2. Speedup under a GPU-memory budget (Reviewer odV7, Weakness 2 and Question 2; Reviewer nTmM, Weakness 1 and Question 1).

Weight offloading is used when the weights do not fit in the available GPU memory. This happens either because the model is larger than the GPU's memory, or because part of that memory is already in use, for example by the KV cache or by other models served on the same GPU. Serving a multimodal model that couples two models through cross-attention is one such case (when the weights of both models together do not fit). How many weights can stay on the GPU therefore depends on the deployment. In the revision, we compare TACT and TEAL under the same GPU-memory budget, varied from 0% to about 75% of the weights, with TACT's 2-bit codes counted toward the budget (LLAMA-ALL, CPU-memory offloading):

TACT-trellis over TEAL0% (paper)~25%~50%~75%
BF161.72×2.06×1.79×1.31×
Q4_K_M1.03×1.06×1.06×0.81×

With BF16 weights, TACT-trellis is faster than TEAL at every budget. Its speedup first grows because a token reads more than half of TACT's small 2-bit codes but only about a fifth of its original weights, so a small budget that holds the codes cuts much of TACT's transfer. With Q4_K_M weights, TACT-trellis falls behind TEAL only when most weights are on the GPU and decoding is no longer dominated by weight transfer. Details are in Appendix D.

CQ3. Q4_K_M with flash offloading (Reviewer odV7, Weakness 2; Reviewer nTmM, Weakness 3).

TACT is faster than TEAL when the transfer time it saves exceeds the time it spends reconstructing the middle tier on the GPU. On LLAMA-ALL-Q4KM with flash offloading, this condition does not hold: TACT-trellis reads fewer bytes than TEAL but takes the same transfer time (Table 3), because it issues more small reads, which flash serves slowly (Section 6.2). This limitation can be overcome by caching, as in prior flash-offloading systems (Section 3). TACT's smallest reads are those of its 2-bit codes, so caching the codes in CPU memory takes these reads off flash. We gave both methods the same CPU-memory budget: TACT preloads its codes into it, and TEAL preloads part of its weights. TACT-trellis then becomes 1.06× faster than TEAL (from 0.93×) (Appendix E).

Updated flash-offloading results.

While preparing this response, the state of our test system changed. We therefore re-ran all flash-offloading experiments of the paper (our re-measured CPU-memory offloading results matched the paper), so that they and the new experiments in this response are measured under the same conditions. The conclusions of the paper are unchanged. For example, on LLAMA-ALL-Q4KM, TEAL and TACT-trellis now take 619 and 664 ms per token, against 702 and 756 ms in the submission, so their ratio stays at 0.93×.

Area_Chair_XgubOpenReview ↗

AC comment

Dear Reviewers

As we are now halfway through the discussion period, I encourage all reviewers to engage in the responses and discuss should there be more questions.

---AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.