- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Test-Time Quantization (TTQ), which runs AWQ-style diagonal activation-aware quantization online during prefill, using activation statistics from the current prompt, and reuses the resulting low-bit weights for decoding. The authors also add an optional rank-16 SVD residual. The main argument is that this avoids picking a calibration set in advance and so reduces calibration mismatch. They back this with a short theoretical section and experiments on OPT/Qwen3/Gemma3 perplexity, GSM8K with Phi-4-mini, TextVQA with Qwen3-VL, and LIBERO with two VLA models, plus some kernel and full-model throughput numbers.
Critical strengths
-
Simple, well-motivated idea. Calibration–deployment mismatch is a real issue for static PTQ, and the paper shows that AWQ results vary noticeably with the calibration domain. Estimating statistics from the prompt itself is a natural and easy-to-implement response.
-
Extensive results shows the effectiveness of the proposed method.
Critical weaknesses
-
The practical deployment case is unclear. TTQ keeps the full-precision weights resident and adds a quantized copy plus scales (and rank-16 factors), so memory is higher than FP16, not just unchanged. Relative to AWQ, TTQ is slower in prefill (11–24%, Table 3) and equal or slower in decode: TTQ (r = 0) trails AWQ-Marlin in most projection benchmarks (e.g., A40 Qwen3-32B 19.34 vs. 30.32 k tok/s), and r = 16 is slower still. So against AWQ, TTQ trades memory and speed for accuracy.
-
The perplexity protocol may not match deployment. At deployment, TTQ builds weights from the prompt and applies them to tokens generated later. The paper does not state which tokens supply TTQ’s statistics in the perplexity experiments, but the code in App. K computes D from any multi-token input, which suggests that in standard full-window evaluation the statistics come from the same window being scored, including tokens after each predicted position. If so, Tables 4, 5, and 7 measure adaptation to the evaluated text rather than prompt-to-generation transfer, which Sec. 2.4 identifies as a failure mode, and the perplexity gains over AWQ may be optimistic. The generative results (GSM8K, VLA) are not affected by this concern. Could the authors clarify the protocol and report a prefix/continuation split, where statistics come from a prefix and only the continuation is scored?
Questions for authors
-
Can you report full-model FP16, AWQ, and TTQ prefill latency, decode throughput, and peak memory on the same hardware, across several prompt and generation lengths?
-
In the perplexity experiments, which tokens are used to compute TTQ’s statistics? Are they the same tokens being scored? How do results change with a prefix/continuation split?
-
How does TTQ compare with HQQ and with AWQ + rank-16 residual across the main LLM tables?
-
How does accuracy vary with prompt length, particularly for short prompts?
3 comments
Thanks for your time and effort. I would like to keep my score of Weak acceptance.
Additional experiments: HQQ and matched rank-16 baselines
We thank the reviewer for the positive rating. Per your suggestion of additional comparisons with HQQ and AWQ equipped with the same low-rank residual, we further conducted a new experiment to examine the benefit of prompt-specific quantization under a limited offline-calibration budget.
Experimental setup. We evaluate SmolLM2-135M on WikiText-2 using 3-bit quantization, group size 32, and a causal prefix/continuation protocol with 256 prefix tokens and 384 continuation tokens. Offline AWQ uses 512 calibration tokens, fewer than the 131,072 tokens used in our earlier rebuttal experiment. HQQ uses its calibration-free quantization procedure.
For TTQ, the preconditioner is estimated using the observed prefix, and the quantized weights are fixed before continuation evaluation. AWQ(r16) and TTQ(r16) share identical static rank-16 residual factors.
Causal continuation perplexity (lower is better):
| Method | PPL ↓ |
|---|---|
| HQQ | 39.21 |
| AWQ | 28.85 |
| AWQ (r=16) | 26.24 |
| TTQ | 27.56 |
| TTQ (r=16) | 25.85 |
Key observations.
- TTQ improves perplexity over AWQ by 1.30 points (28.85 → 27.56).
- With identical rank-16 residual factors, TTQ further improves over AWQ by 0.39 points (26.24 → 25.85).
- TTQ(r16) achieves substantially lower perplexity than the HQQ reference baseline (39.21 → 25.85).
The matched-rank comparison demonstrates that the benefit of prompt-specific activation statistics persists when both methods use the same low-rank reconstruction. The experiment also indicates the effectiveness of online quantization when only a small offline calibration set is available.
Prefix/continuation evaluation and prompt-length sensitivity
We thank the reviewer for the constructive suggestion to evaluate TTQ using an explicit prefix/continuation split and to examine sensitivity to prompt length.
We agree that the usefulness of online quantization depends on how well the observed prefix represents subsequent token activations. A short prompt may provide a noisy estimate, whereas a longer prompt can improve the fidelity of the activation-dependent preconditioner.
New causal perplexity experiment. We evaluated SmolLM2-135M on WikiText-2 using 3-bit quantization and group size 32. For each sample, TTQ computes its preconditioner exclusively from the observed prefix. The quantized weights are then frozen and evaluated on the continuation. No future-token activations are used for quantization.
Causal perplexity (↓):
Observations. With a 32-token prefix, online quantization is not consistently better than offline AWQ, confirming the reviewer's concern that short prefixes may be insufficient for reliable adaptation. However, with 128- and 512-token prefixes, TTQ consistently achieves lower perplexity across all three continuation lengths.
We therefore agree that TTQ's advantage is conditional rather than universal. Its effectiveness depends on the quantity and representativeness of observed prompt statistics. We will add this qualification and the causal prefix/continuation protocol to the revised manuscript. We appreciate the suggestion, which helps establish a more realistic characterization of TTQ's benefits and limitations during autoregressive inference.