- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PrismQ introduces noise-aware (sensitivity-aware) resource allocation for elastic PTQ. PrismRank allocates reconstruction ranks based on SVD of clipping errors, PrismLoRA combines shared and precision-specific low-rank components, and Sparse Token selectively propagates high-error states during calibration. The method is evaluated across LLMs, ViTs, and VLMs.
Critical strengths
-
Noise-aware allocation of reconstruction capacity and calibration states provides an interesting perspective on elastic PTQ.
-
Extensive evaluations across model architectures and precision configurations support the generality of the approach.
Critical weaknesses
-
Limited scalability analysis of SVD-based rank allocation. PrismRank requires SVD across layers and precision configurations, potentially incurring substantial computational and memory overhead. However, evaluations are limited to models up to 13B, with no analysis of SVD overhead or comparison of total quantization time against existing methods.
-
Insufficient evidence of practical serving efficiency. Since LLM decoding is often memory-bandwidth-bound, the practical advantages of elastic activation quantization over weight-only quantization remain unclear. Comparisons with recent OFA methods, such as AnyBCQ and MoBiQuant, are missing. Furthermore, the inference analysis lacks prefill latency measurements and runtime precision-switching overhead.
Questions for authors
- Latest LLMs. Could the authors evaluate PrismQ on a recent large-scale LLM, such as Qwen3.8-2.4T-A95B or Qwen3-235B-A22B if architectural compatibility is an issue?
- SVD scalability. How does SVD overhead scale with model size? Please report SVD latency, peak memory usage, and total quantization time relative to existing PTQ methods.
- Serving efficiency. Could the authors compare PrismQ with AnyBCQ and MoBiQuant under weight-only quantization, reporting accuracy, prefill latency, decode throughput, and memory usage? A controlled comparison with AnyBCQ on a single model would suffice. Please also clarify the overhead of runtime precision switching, including compilation, weight repacking, and CUDA Graph management.
I would consider raising my recommendation to acceptance if the authors demonstrate scalability to recent large-scale LLMs, quantify SVD overhead, and establish competitive serving efficiency against recent OFA methods.
7 comments
Response to Weakness 2 & Question 3 (Part II)
Weakness 2 & Question 3:Serving efficiency and precision switching
Weight-only. Completed LLaMA-3-8B measurements use packed W4A16/W8A16 exports from one calibrated superset, with the low-rank correction merged before packing and BF16 activations/KV cache. Both run on one RTX 6000 Ada with vLLM 0.15.1 and verified Marlin kernels.
Table R12. Calibrated-superset PPL and packed weight-only serving.
| Format | WikiText-2 PPL | C4 PPL | Prefill (ms) | Decode (tok/s) | Weight files (GiB) | Sampled GPU peak (GiB) |
|---|---|---|---|---|---|---|
| W4A16 | 6.55 | 9.54 | 48.67 | 150.31 | 5.31 | 41.63 |
| W8A16 | 6.15 | 8.91 | 64.46 | 93.81 | 8.56 | 41.65 |
PPL is from fake-quantized calibrated-superset evaluation; latency and memory are from packed serving. Prefill is mean engine time for 512 input/1 output token; decode is median client-observed throughput for 128 input/256 output tokens, concurrency 1. GPU peak includes the reserved KV-cache pool (memory utilization 0.85). The supplementary deploy/README_deploy.md describes the protocol; deploy/vllm_w4_*/ and deploy/vllm_w8_*/ contain the packed-serving run.json, performance.json, memory.json and kernels.json records. PPL is logged in deploy/ptq_20261110_135709_2720530/log_2026_11_10_14_29_Llama-3-8b.txt.
Table R13. Additional LLaMA-3-8B W4 comparison on one RTX 6000 Ada.
| Method | WikiText-2 PPL | MMLU Acc. (%) | Decode (tok/s) |
|---|---|---|---|
| FP16 | 6.12 | 65.02 | — |
| AnyPrecisionLLM | 6.70 | 64.04 | 87 |
| AnyBCQ | 6.84 | 63.15 | 90 |
| MoBiQuant | 7.31 | 63.29 | 129 |
| PrismQ | 6.55 | 64.10 | 150 |
MoBiQuant throughput is estimated from its reported A100 40 GB result by memory-bandwidth scaling to RTX 6000 Ada; the other quantized throughput entries are measured. PrismQ's 150 tok/s rounds the 150.31 tok/s above. A dash denotes an unreported measurement.
Weight + activation. Manuscript Table 16 uses LLaMA-2-7B with ABQ-LLM kernels. At batch 1, 512 input and 128 generated tokens, W8A8 reduces sampled peak GPU memory from 13,820 to 7,654 MiB (44.6%) while decode falls from 58.491 to 53.418 tok/s (8.7%), providing a different speed/memory operating point. These are total serving-memory measurements relative to FP16 on this model/backend; the weight-only results above use LLaMA-3-8B and vLLM.
Each weight-only format uses a separately initialized engine. Dynamic precision switching and its isolated overhead, including repacking, compilation and CUDA Graph handling, remain future work.
Response to Question 1 (Part III)
Question 1:Evaluation on recent Qwen3 models
Quality evaluation covers Qwen3-0.6B–14B. Qwen3-32B and Qwen3-235B-A22B have calibration-cost records but no reported PPL or task accuracy under our single 48 GB GPU budget.
Table R14. Qwen3 quality and calibration cost. PPL: WikiText-2/C4; Avg.: five tasks (%).
| Model | Type | Total trainable params (M) | Stage1 (h) | All (h) | W4A4-PPL | W4A4-Avg. | W6A6-PPL | W6A6-Avg. | W8A8-PPL | W8A8-Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-0.6B | Dense | 34.86 | 0.100 | 0.604 | 28.12/33.83 | 46.64 | 21.73/26.22 | 51.67 | 21.15/25.60 | 52.23 |
| Qwen3-1.7B | Dense | 70.22 | 0.139 | 1.001 | 19.90/23.45 | 54.58 | 17.41/19.79 | 60.58 | 16.79/19.43 | 61.21 |
| Qwen3-4B | Dense | 148.22 | 0.228 | 2.403 | 15.46/18.96 | 62.48 | 13.96/16.90 | 67.22 | 13.70/16.67 | 68.60 |
| Qwen3-8B | Dense | 228.88 | 0.304 | 3.976 | 10.99/14.85 | 68.85 | 9.94/13.43 | 71.35 | 9.72/13.31 | 71.65 |
| Qwen3-14B | Dense | 383.70 | 0.469 | 7.074 | 9.67/13.34 | 72.41 | 8.74/12.08 | 74.54 | 8.67/12.01 | 74.93 |
| Qwen3-32B | Dense | 859.94 | 1.036 | 16.557 | — | — | — | — | — | — |
| Qwen3-30B-A3B | MoE | 650.25 | 0.728 | 15.470 | — | — | — | — | — | — |
| Qwen3-235B-A22B | MoE | 2547.09 | 1.536 | 34.242 | — | — | — | — | — | — |
Quality rows use seed 42 and 12 epochs. Tables R14/R11 use normalized accuracy for PIQA, ARC-E/C and HellaSwag, and accuracy for WinoGrande.
Table R15. Comparable calibration settings for Dense and MoE models.
| Model type | Calibration targets | Epochs | Reference rank | Minimum rank per precision | Precision-specific rank |
|---|---|---|---|---|---|
| Dense | 5 | 12 | 40 | 2 | 2 |
| MoE | 3 | 10 | 10 | 1 | 1 |
These comparable settings follow the supplied Dense and MoE launch scripts.
Thank you for highlighting recent-model coverage, analysis cost, and practical serving performance.
Response to Weakness 1 & Question 2 (Part I)
Dear Reviewer j1jF,
Thank you for your careful review. Manuscript references refer to the revised paper; supplementary paths refer to files in the uploaded supplementary materials or the anonymous archive. Bold marks the best compared quantized results, including displayed ties; reference rows are unranked.
Table R10. Qwen3 calibration cost and monitor-reported memory peaks.
Stage1 includes clipping pretraining, clipping-induced error analysis, singular-value computation and allocation. All spans the first log record through PTQ completion, including pre-rotation, initialization and logging; evaluation is excluded.
RSS is resident host memory; allocated and reserved denote GPU allocator usage and reservation. The memory columns report monitoring peaks.
Current Dense/MoE script settings are compared in Question 1 (Table R15). Timing logs for the completed Dense runs are in the supplementary
PrismQ_exp/qwen3_*_seed42_epochs12/directories.Table R11. Matched calibration comparison, seed 42, 12 epochs, W4A4.
All uses the boundary above. The runs match seed, corpus, sample counts, grouping and learning rates, while retaining native capacity and calibration-state designs. At W4A4, PrismQ takes less time and gives lower PPL and higher five-task means on all four models. Configurations and per-block timing records are in the first entries and PTQ sections of the supplementary
PrismQ_exp/andQuEPT_exp/WA_Epochs/logs.