acceptodds
Under review as a conference paper at ICLR 2027

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

Abstract

To tackle the high computational cost of large foundation models, post-training activation-aware quantization methods have been widely used. However, because these methods rely heavily on calibration data, calibration–deployment mismatch may arise on unseen downstream tasks. We propose test-time quantization (TTQ), which constructs prompt-specific low-bit weights online from the current prompt and reuses them during decoding. TTQ removes the need for a predetermined calibration dataset when constructing these weights and targets decode-time weight traffic and integer execution rather than resident model-memory reduction. Experiments across language, vision-language, and vision-language-action models show improved low-bit accuracy over offline baselines in several settings while retaining most of AWQ’s decode throughput at a measurable prefill cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, Activation-Aware Compression, Test-Time Compute, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NbnWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Test-Time Quantization (TTQ), which runs AWQ-style diagonal activation-aware quantization online during prefill, using activation statistics from the current prompt, and reuses the resulting low-bit weights for decoding. The authors also add an optional rank-16 SVD residual. The main argument is that this avoids picking a calibration set in advance and so reduces calibration mismatch. They back this with a short theoretical section and experiments on OPT/Qwen3/Gemma3 perplexity, GSM8K with Phi-4-mini, TextVQA with Qwen3-VL, and LIBERO with two VLA models, plus some kernel and full-model throughput numbers.

Critical strengths
  1. Simple, well-motivated idea. Calibration–deployment mismatch is a real issue for static PTQ, and the paper shows that AWQ results vary noticeably with the calibration domain. Estimating statistics from the prompt itself is a natural and easy-to-implement response.

  2. Extensive results shows the effectiveness of the proposed method.

Critical weaknesses
  1. The practical deployment case is unclear. TTQ keeps the full-precision weights resident and adds a quantized copy plus scales (and rank-16 factors), so memory is higher than FP16, not just unchanged. Relative to AWQ, TTQ is slower in prefill (11–24%, Table 3) and equal or slower in decode: TTQ (r = 0) trails AWQ-Marlin in most projection benchmarks (e.g., A40 Qwen3-32B 19.34 vs. 30.32 k tok/s), and r = 16 is slower still. So against AWQ, TTQ trades memory and speed for accuracy.

  2. The perplexity protocol may not match deployment. At deployment, TTQ builds weights from the prompt and applies them to tokens generated later. The paper does not state which tokens supply TTQ’s statistics in the perplexity experiments, but the code in App. K computes D from any multi-token input, which suggests that in standard full-window evaluation the statistics come from the same window being scored, including tokens after each predicted position. If so, Tables 4, 5, and 7 measure adaptation to the evaluated text rather than prompt-to-generation transfer, which Sec. 2.4 identifies as a failure mode, and the perplexity gains over AWQ may be optimistic. The generative results (GSM8K, VLA) are not affected by this concern. Could the authors clarify the protocol and report a prefix/continuation split, where statistics come from a prefix and only the continuation is scored?

Questions for authors
  1. Can you report full-model FP16, AWQ, and TTQ prefill latency, decode throughput, and peak memory on the same hardware, across several prompt and generation lengths?

  2. In the perplexity experiments, which tokens are used to compute TTQ’s statistics? Are they the same tokens being scored? How do results change with a prefix/continuation split?

  3. How does TTQ compare with HQQ and with AWQ + rank-16 residual across the main LLM tables?

  4. How does accuracy vary with prompt length, particularly for short prompts?

3 comments
AuthorsOpenReview ↗

Prefix/continuation evaluation and prompt-length sensitivity

We thank the reviewer for the constructive suggestion to evaluate TTQ using an explicit prefix/continuation split and to examine sensitivity to prompt length.

We agree that the usefulness of online quantization depends on how well the observed prefix represents subsequent token activations. A short prompt may provide a noisy estimate, whereas a longer prompt can improve the fidelity of the activation-dependent preconditioner.

New causal perplexity experiment. We evaluated SmolLM2-135M on WikiText-2 using 3-bit quantization and group size 32. For each sample, TTQ computes its preconditioner exclusively from the observed prefix. The quantized weights are then frozen and evaluated on the continuation. No future-token activations are used for quantization.

Causal perplexity (↓):

PrefixAWQ / TTQ, decode 32AWQ / TTQ, decode 128AWQ / TTQ, decode 512
3232.91 / 32.7434.19 / 34.8529.83 / 28.91
12824.43 / 23.7528.75 / 28.0427.82 / 27.15
51221.82 / 21.6524.65 / 23.4726.08 / 25.22

Observations. With a 32-token prefix, online quantization is not consistently better than offline AWQ, confirming the reviewer's concern that short prefixes may be insufficient for reliable adaptation. However, with 128- and 512-token prefixes, TTQ consistently achieves lower perplexity across all three continuation lengths.

We therefore agree that TTQ's advantage is conditional rather than universal. Its effectiveness depends on the quantity and representativeness of observed prompt statistics. We will add this qualification and the causal prefix/continuation protocol to the revised manuscript. We appreciate the suggestion, which helps establish a more realistic characterization of TTQ's benefits and limitations during autoregressive inference.

Thanks for your time and effort. I would like to keep my score of Weak acceptance.

AuthorsOpenReview ↗

Additional experiments: HQQ and matched rank-16 baselines

We thank the reviewer for the positive rating. Per your suggestion of additional comparisons with HQQ and AWQ equipped with the same low-rank residual, we further conducted a new experiment to examine the benefit of prompt-specific quantization under a limited offline-calibration budget.

Experimental setup. We evaluate SmolLM2-135M on WikiText-2 using 3-bit quantization, group size 32, and a causal prefix/continuation protocol with 256 prefix tokens and 384 continuation tokens. Offline AWQ uses 512 calibration tokens, fewer than the 131,072 tokens used in our earlier rebuttal experiment. HQQ uses its calibration-free quantization procedure.

For TTQ, the preconditioner is estimated using the observed prefix, and the quantized weights are fixed before continuation evaluation. AWQ(r16) and TTQ(r16) share identical static rank-16 residual factors.

Causal continuation perplexity (lower is better):

MethodPPL ↓
HQQ39.21
AWQ28.85
AWQ (r=16)26.24
TTQ27.56
TTQ (r=16)25.85

Key observations.

  • TTQ improves perplexity over AWQ by 1.30 points (28.85 → 27.56).
  • With identical rank-16 residual factors, TTQ further improves over AWQ by 0.39 points (26.24 → 25.85).
  • TTQ(r16) achieves substantially lower perplexity than the HQQ reference baseline (39.21 → 25.85).

The matched-rank comparison demonstrates that the benefit of prompt-specific activation statistics persists when both methods use the same low-rank reconstruction. The experiment also indicates the effectiveness of online quantization when only a small offline calibration set is available.

Reviewer_KqYZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TTQ's main method reconstructs low-bit weights using activation statistics from the current prompt during the prefill stage of each request, and then reuses them during decoding. A static low-rank residual can also be added. Its main novelty lies in the request-level quantization deployment strategy, while the quantization operator itself follows AWQ.

Critical strengths
  1. The problem is practically meaningful, and the design is natural. A mismatch between offline calibration data and deployment inputs can indeed affect quantization quality. Adapting to the activations of the current request requires no additional training or backpropagation. Concentrating the overhead in the prefill stage and reusing the weights during decoding is consistent with the characteristics of autoregressive inference.
  2. There are meaningful downstream accuracy improvements at 3-bit precision. On GSM8K with Phi-4-mini-instruct, TTQ achieves an accuracy of 59.36%, compared with 52.17% for GPTQ and 47.92% for AWQ, representing improvements of 7.19 and 11.44 percentage points, respectively. These gains are more convincing than improvements demonstrated solely through perplexity.
  3. The experiments have broad coverage. Experiments are conducted on LLMs, VLMs, and VLA models involving LLMs, and the evaluation on X-VLA is extended to 8,000 episodes. Ablations are also conducted with key components matched. In Appendix Table 12, after matching the L2 preconditioner and rank-16 residual, TTQ still improves perplexity over offline AWQ by 1.1–2.2 points. For example, with a group size of 32, perplexity decreases from 19.91 to 18.29. Therefore, the entire improvement cannot be attributed to changing the norm or adding a low-rank branch.
Critical weaknesses
  1. The full-model throughput results raise serious concerns regarding their reported orders of magnitude. Tables 3 and 11 explicitly label the results on an RTX 5090 as full-model throughput, with prefill measured in M tokens/s and decoding measured in k tokens/s. For Qwen3-8B, TTQ reports: • Prefill: 8.2 million tokens/s • Decode: 142.2 thousand tokens/s A rough estimate based on the computational cost of approximately (2N, N denotes the number of model parameters) operations per token for a dense model suggests that the former would correspond to approximately (131×10^15) operations per second, far exceeding the RTX 5090's officially listed peak performance of approximately (3.35×10^15) AI operations/s.
  2. The causal evaluation protocol for perplexity is not clearly specified, which could substantially affect the accuracy evidence. The authors should clarify which tokens are used to estimate the request-specific quantization parameters and which tokens are used to calculate the loss during perplexity evaluation. If the quantization parameters are estimated from an entire teacher-forced test sequence and then used for predictions at all positions in that same sequence, future tokens may influence the construction of the weights used to predict earlier tokens. Weakness 3: The best accuracy improvements are not matched by corresponding runtime measurements. For LLMs, Tables 1 and 2 show that AWQ achieves higher inference throughput than TTQ in most cases. Meanwhile, the limited results in Tables 4 and 5 indicate that AWQ with static quantization calibrated on a single dataset does not lag far behind in perplexity. For example, when (T = 2^17), AWQ achieves a perplexity of 25.02, compared with 24.17 for TTQ. Table 7 shows a more pronounced performance gap between TTQ and AWQ at extremely low precision (2-bit quantization), but no corresponding runtime throughput comparison is provided. Thus, an experiment essential to establishing the practical value of TTQ is missing i.e. under the same hardware and workload, how much TTQ improve the accuracy/latency.
Questions for authors

Answer questions in Weakness 2 and 3.

3 comments
AuthorsOpenReview ↗

Future-token contamination

We thank the reviewer for raising the important question of whether future tokens can influence test-time quantization during perplexity evaluation.

Clarification. Our TTQ inference procedure constructs the preconditioner during prefill and freezes the quantized weights for subsequent decoding. Hence, we do not use future-token statistics for quantization in our submission. We recognize that computing a preconditioner from an entire evaluation window would be different from this deployment setting, even with a causal attention mask.

We therefore performed an additional controlled causal-perplexity experiment. For each evaluation sequence, we separate the observed prefix from the future continuation. Only prefix activations are used to estimate the online preconditioner. The resulting quantized weights are fixed before continuation scoring, so future continuation tokens cannot influence the quantization parameters.

We compare offline AWQ-style quantization and TTQ on SmolLM2-135M, WikiText-2, using 3-bit quantization with group size 32.

Causal perplexity (↓):

PrefixAWQ / TTQ, decode 32AWQ / TTQ, decode 128AWQ / TTQ, decode 512
3232.91 / 32.7434.19 / 34.8529.83 / 28.91
12824.43 / 23.7528.75 / 28.0427.82 / 27.15
51221.82 / 21.6524.65 / 23.4726.08 / 25.22

TTQ achieves lower perplexity in eight of the nine tested configurations, including all six configurations with prefixes of at least 128 tokens. With very short prefixes, however, online estimation can be less reliable and occasionally underperform offline quantization.

These results directly evaluate prompt-to-continuation transfer without future-token contamination. We will clarify this causal protocol explicitly used in the revised paper, and distinguish from full-window perplexity evaluation.

AuthorsOpenReview ↗

Response to Weakness 1: Correction of throughput units

We thank the reviewer for carefully examining our throughput results and identifying the physically implausible values.

We confirmed a unit-conversion error between milliseconds and seconds in our throughput reporting. The reported token-per-second values were incorrectly inflated by a factor of 1,000. We apologize for this mistake and have corrected the values and units in Tables 3 and 11.

The corrected full-model phase throughput on RTX 5090 is:

Qwen3AWQ prefillTTQ prefillAWQ decodeTTQ decode
0.6B69.0k52.2k206.1206.1
1.7B38.3k32.6k206.8193.3
4B16.0k14.2k157.6155.6
8B10.1k8.2k142.7142.2

Importantly, this uniform conversion error does not change the relative throughput comparisons between AWQ and TTQ. TTQ incurs approximately 11–24% lower prefill throughput due to online calibration and quantization, while maintaining comparable decoding throughput (within approximately 6.5% of AWQ).

We agree that TTQ does not provide a speed advantage over optimized offline AWQ in these measurements. Its primary advantage is the ability to adapt quantization to the current input distribution, trading additional prefill computation and memory for potentially improved accuracy under calibration mismatch. In addition, TTQ can still enjoy fast integer kernels to accelerate over unquantized original model.

We appreciate the reviewer bringing this reporting error to our attention.

AuthorsOpenReview ↗

Latency measurement

We thank the reviewer for requesting a direct characterization of the runtime cost associated with TTQ.

We now provide request-level latency reports using the full-model prefill and decode throughput measurements on an RTX 5090. The TTQ prefill measurement includes online quantization, while decoding uses the resulting quantized weights.

Estimated request latency (ms):

ModelPrompt / OutputAWQ 4-bitTTQ 4-bitOverhead
Qwen3-1.7B512 / 128632.3677.9+7.2%
Qwen3-4B512 / 128844.2858.7+1.7%
Qwen3-8B128 / 128909.7915.8+0.7%
Qwen3-8B512 / 128947.7962.6+1.6%
Qwen3-8B2048 / 1281099.81149.9+4.6%
Qwen3-8B2048 / 5123790.73850.3+1.6%

The results quantify the runtime tradeoff between online and offline quantization. TTQ incurs approximately 0.7–7.2% additional request latency in the evaluated configurations, with most of the difference arising during prefill. Decode throughput remains comparable to AWQ, while it can be faster than BF16 thanks to the customized integer kernel.

These measurements complement our accuracy results by quantifying the computational cost of prompt-specific adaptation.

Reviewer_SwPQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Offline activation-aware quantization relies on calibration data that may differ from deployment inputs. TTQ instead collects activation statistics from the current prompt, constructs prompt-specific low-bit weights during prefill, and reuses them during decoding. An optional low-rank residual further reduces quantization error. The experiments cover language, vision-language, and vision-language-action models, measuring perplexity, task accuracy, and runtime. TTQ improves low-bit accuracy over offline baselines in several settings, including 3-bit GSM8K. It retains most of AWQ’s decoding throughput but adds prefill overhead and requires full-precision weights to remain available.

Critical strengths
  1. Adapting quantization to the current input is a reasonable way to address calibration mismatch. The experiments include actual task results across several model types, rather than relying only on perplexity.
  2. The matched ablations control the activation statistic and low-rank residual. Online estimation still helps after these controls, which supports the value of prompt-specific adaptation.
Critical weaknesses
  1. The perplexity evaluation needs a clearer description of which tokens are used to construct the quantized weights and which are scored. If statistics are collected from the entire scored segment, later ground-truth tokens could affect weights used at earlier positions. A causal attention mask alone would not prevent this. The paper should explicitly evaluate a continuation using weights constructed only from its preceding prompt.
  2. The evidence for practical acceleration is incomplete. Compared with AWQ, full-model prefill throughput drops by about 11–24%, while decoding throughput is similar or slightly lower. The larger speedups come from projection microbenchmarks. Total request latency across prompt lengths, output lengths, and batch sizes is needed, together with peak memory including the retained full-precision weights.
  3. The paper tests several types of models, but more tests are needed for each type. For language tasks, the downstream results mainly come from one model on GSM8K. The controlled comparison with AWQ also uses only one model and one dataset. It is therefore unclear whether the gains hold across other models and tasks.
Questions for authors

See weakness above.

3 comments
AuthorsOpenReview ↗

Causal evaluation for PPL evaluation

We thank the reviewer for identifying the need to distinguish the tokens used to construct the quantizer from those used to evaluate perplexity.

Causal evaluation protocol. TTQ is designed to estimate activation statistics during prefill and reuse the resulting quantized weights during decoding. We do not use future-token statistics for quantization in our paper. We agree that causal attention masking alone does not prevent future-token information from entering a preconditioner estimated over an entire teacher-forced sequence.

To address this concern explicitly, we conducted an additional prefix-to-continuation evaluation. The preconditioner is constructed using only the observed prefix, the quantized weights are frozen, and perplexity is measured exclusively on subsequent tokens. No continuation-token statistics are used for quantization as in our original submission.

We evaluate SmolLM2-135M on WikiText-2 with 3-bit quantization and group size 32.

Causal perplexity (↓):

PrefixAWQ / TTQ, decode 32AWQ / TTQ, decode 128AWQ / TTQ, decode 512
3232.91 / 32.7434.19 / 34.8529.83 / 28.91
12824.43 / 23.7528.75 / 28.0427.82 / 27.15
51221.82 / 21.6524.65 / 23.4726.08 / 25.22

The results reveal: short prefixes do not always provide sufficiently reliable activation statistics. In contrast, TTQ achieves lower perplexity than offline quantization for longer decoding lengths.

This additional experiment establishes the effectiveness of prefix-conditioned quantization under a strictly causal evaluation protocol. We will clarify the protocol and its limitations in the revised manuscript.

AuthorsOpenReview ↗

Latency and Peak Memory

We thank the reviewer for emphasizing the importance of request-level latency and peak memory for practical deployment.

Request latency

We provide additional request-level latency reports using the full-model throughput measurements on an RTX 5090. The measured TTQ prefill throughput includes online activation-statistic estimation and weight quantization, thereby accounting for their computational overhead. We evaluate multiple prompt and generation lengths.

Estimated request latency (ms):

ModelPrompt / OutputAWQ 4-bitTTQ 4-bitOverhead
Qwen3-1.7B512 / 128632.3677.9+7.2%
Qwen3-4B512 / 128844.2858.7+1.7%
Qwen3-8B128 / 128909.7915.8+0.7%
Qwen3-8B512 / 128947.7962.6+1.6%
Qwen3-8B2048 / 1281099.81149.9+4.6%
Qwen3-8B2048 / 5123790.73850.3+1.6%

These measurements show that TTQ maintains request latency close to AWQ across the evaluated workloads, with additional cost primarily arising during prefill. The relative overhead is particularly small for the larger model configurations and longer generation workloads.

Peak GPU memory

We also measured peak GPU memory under a 512-token workload:

ModelBF16AWQ 4-bitTTQ 4-bit
Qwen3-0.6B1.27 GiB0.67 GiB1.95 GiB
Qwen3-1.7B3.37 GiB1.42 GiB4.79 GiB
Qwen3-4B7.73 GiB2.67 GiB10.40 GiB

TTQ requires approximately 1.35–1.54 times the BF16 peak memory and 2.91–3.90 times the AWQ memory in these experiments. This overhead reflects retaining the original full-precision weights while constructing prompt-specific quantized weights.

TTQ provides test-time adaptation and integer kernel acceleration with a memory and prefill-computation tradeoff.

Reviewer_SwPQOpenReview ↗

response

Thanks for the response. The peak-memory results clarify the memory cost. However, the additional causal evaluation does not verify the main perplexity results under the same protocol. The request latencies are estimated from throughput rather than measured directly, and the broader downstream evaluation remains limited. Therefore, I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.