- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper focuses on the problem of "inference not terminating" that occurs in the MoE inference model after about 2-bit extremely low bit quantization. The author's core viewpoint is: The performance collapse after extremely low bit quantization is not entirely due to the loss of knowledge or reasoning ability. Many errors are caused by the model's inability to generate</hint>in a timely manner, resulting in inference loops, excessively long outputs, or reaching the maximum generation length. 2. Before generating the</hint>, the router will steadily select a small group of experts. The author refers to it as termination aware experts, accounting for approximately 1.6% -2.5% of all routing experts. If these experts are fixed to higher accuracy and the remaining experts are allocated bits based on sensitivity, it can significantly improve accuracy and generation length. Based on this, the paper proposes a three-stage RTAQ: -Build activation aware codebook banks with different quantization levels (K=3, ldots, 8); -Fix termination aware experts as \ (K=8 ), and other experts optimize allocation accuracy based on Fisher sensitivity and knapsack; -Use fixed codebook for GPTQ/OBS error compensation, affine parameter re fitting, and output gain correction. On three 30B-80B MoE models and five types of inference/code benchmarks, RTAQ achieved the highest average accuracy among all quantization methods with approximately 2.016-2.025 "logical bit/routing expert weights", while the output length was closer to BF16.
Critical strengths
The problem is novel and has practical significance. The paper specifically locates the low bit quantization failure to the generation and inference termination of</hint>, rather than broadly attributing it to "capability decline". This observation is very enlightening for the deployment of long inference models.
The method has formed a complete engineering pipeline. Codebook construction, mixed precision allocation, GPTQ style correction, and output scale correction can be combined into a relatively complete PTQ method without the need for retraining the model.
The experimental models and tasks cover a wide range. The paper tested three different 30B-80B MoE models and covered mathematical, scientific reasoning, and coding tasks. The average accuracies of RTAQ were 85.81%, 77.68%, and 81.03%, respectively, all of which were better than the comparison methods.
Critical weaknesses
- The evidence for the core claim that 'termination failure leads to performance collapse' is still insufficient The most important proposition of the paper is that as long as</hint>is successfully generated after quantification, the accuracy will approach BF16; Most errors come from the inability to close the thought process. However, the main text and appendix did not provide complete data to support this conclusion, such as: -The proportion of</hint>generated by each model and method; -EOS termination rate and the proportion of maximum length truncation; -Conditional accuracy rate \ (P (\ text {correct} \ mid</hint>) ); -Logit, ranking, or generation probability of</hint>; -What are the proportions of cyclic reasoning, excessive reasoning, and normal long reasoning. Figure 2b only shows the average generated length. The average length cannot distinguish true non termination, reasonable longer reasoning, and a small number of extreme outliers. After removing the termination aware expert pinning, the paper only reports changes in length without synchronously reporting accuracy and termination rate. In addition, the expert set \ (P ) is identified from successful trajectories that already contain</hint>, which may result in conditional selection bias. It is recommended to perform restorative interventions directly in the quantitative model: only restore the accuracy of P and measure the changes in close token probability, termination rate, and accuracy.
Questions for authors
What are the generation rate, EOS rate, truncation rate, and conditional accuracy of each model and method?
Is RTAQ still better than baseline under strictly the same final physical storage budget?
Can similar improvements be achieved by replacing P with random experts or experts who match the overall routing frequency?
3 comments
[Q3] Replacing with random or frequency-matched experts
No. Under the same bit budget, pinning random or frequency-based experts does not reproduce the gains from pinning .
On Qwen3.6-35B-A3B (LiveCodeBench v6), we pinned 223 experts at 3 bits under the same budget (2.022 logical bits). We chose these experts in four ways: , a random set, the top-223 by routing frequency, and the top-223 by Fisher sensitivity.
- : the only choice that clearly improves over the unpinned model. The
</think>rate rises from 85.2% to 90.1%, and cyclic loops drop from 14.0% to 9.4%. - Other choices: they stay near or below the unpinned model, with
</think>rates of 81.4–86.3% and loop rates of 13.0–17.8%.
| Pinned experts (223) | </think> (%) | Cyclic loops (%) | Accuracy (%) | Avg. length |
|---|---|---|---|---|
| None (unpinned) | 85.2 | 14.0 | 69.7 | 42.1K |
| (ours) | 90.1 | 9.4 | 70.2 | 37.0K |
| Random | 81.4 | 17.8 | 68.4 | 43.9K |
| Top by routing frequency | 86.3 | 13.0 | 68.7 | 41.3K |
| Top by Fisher sensitivity | 82.2 | 16.0 | 66.4 | 44.1K |
Cyclic loops: responses that never emit </think> and end in verbatim repetition.
Blocking experts in the BF16 model shows the same pattern. Blocking routing to lowers the termination rate on GPQA from 98.0% to 60%. Blocking two random sets of 223 experts matched to 's routing frequency leaves it unchanged, at 98.0–98.5%.
[Weaknesses]
(1) Termination Failure Leads to Performance Collapse Most of the accuracy loss at near 2 bits comes from responses that never terminate, and restoring only the experts in brings termination back. Below, we report all the requested statistics (Qwen3.6-35B-A3B, 3 samples per problem; LCB budget 100K tokens, GPQA 70K).
| LCB | Acc. | </think> | EOS | Trunc. | Acc. given </think> | Cyclic loops | Excessive | Normal long |
|---|---|---|---|---|---|---|---|---|
| BF16 | 77.6 | 97.2 | 96.9 | 3.1 | 79.8 | 1.5 | 1.3 | 9.7 |
| GPTQ (2.25) | 50.1 | 75.3 | 72.5 | 27.5 | 64.2 | 23.2 | 1.5 | 9.2 |
| MC-MoE (2.36) | 72.3 | 92.6 | 91.9 | 8.1 | 76.1 | 6.9 | 0.5 | 9.2 |
| GSQ (2.125) | 66.4 | 84.0 | 84.0 | 16.0 | 77.0 | 14.0 | 2.0 | 9.9 |
| RTAQ (2.02) | 70.2 | 90.1 | 90.1 | 9.9 | 77.1 | 9.4 | 0.5 | 7.6 |
| GPQA | Acc. | </think> | EOS | Trunc. | Acc. given </think> | Cyclic loops | Excessive | Normal long |
|---|---|---|---|---|---|---|---|---|
| BF16 | 83.0 | 98.7 | 98.7 | 1.3 | 83.4 | 0.7 | 0.7 | 9.8 |
| GPTQ (2.25) | 72.9 | 87.5 | 87.5 | 12.5 | 80.2 | 12.5 | 0.0 | 3.4 |
| MC-MoE (2.36) | 72.6 | 94.3 | 94.3 | 5.7 | 76.1 | 5.6 | 0.2 | 2.5 |
| GSQ (2.125) | 80.6 | 96.8 | 96.8 | 3.2 | 82.3 | 3.0 | 0.2 | 8.2 |
| RTAQ (2.02) | 79.3 | 95.1 | 95.1 | 4.9 | 81.9 | 4.7 | 0.2 | 9.8 |
Cyclic loops: never emit </think> and end in verbatim repetition.
Excessive: never emit </think> and are truncated without repetition.
Normal long: closed but longer than BF16's 90th-percentile closed length.
The main failure is getting trapped in repetitive loops before reaching the stopping point, not an inability to predict </think>. When RTAQ successfully terminates, its accuracy remains close to BF16 (77.1% vs. 79.8% on LCB), even under more aggressive quantization. Forcing termination recovers 41% of GPTQ's and 55% of MC-MoE's non-terminating failures, demonstrating that failure to stop is a major source of accuracy degradation.
(2) Causal Evidence for Termination-Aware Expert Protection As suggested, we restore only (to 3 bits) inside the quantized model, keeping the total budget fixed, and compare it with same-size control sets (LCB):
| Experts restored to 3 bits | </think> | Cyclic loops | Accuracy |
|---|---|---|---|
| None (2.02 bits) | 85.2 | 14.0 | 69.7 |
| (2.02 bits) | 90.1 | 9.4 | 70.2 |
| None (1.63 bits) | 28.2 | 71.5 | 22.7 |
| (1.63 bits) | 37.4 | 61.8 | 34.6 |
Restoring raises termination and cuts loops, and the gain grows when the budget is tighter (+11.9 accuracy points at 1.63 bits, paired ). Sets of the same size chosen by frequency, Fisher sensitivity, or at random do not reproduce this, so the effect is specific to rather than a selection artifact. (Also refer our answer of Q3)
Reference
[1] E. Frantar et al. GPTQ: Accurate post-training quantization for generative pre-trained transformers. ICLR, 2023.
[2] W. Huang et al. Mixture compressor for mixture-of-experts LLMs gains more. ICLR, 2025.
[3] A. Dadgarnia et al. GSQ: Highly-accurate low-precision scalar quantization for LLMs via Gumbel-softmax sampling. arXiv:2604.18556, 2026.
Gentle reminder: author-reviewer discussion period concludes soon
Dear Reviewer aPXz,
Just a brief reminder that the author–reviewer discussion period concludes soon. We have posted our responses to your questions and would be grateful if you could take a look.
If you have any follow-up questions or remaining concerns, we would be happy to address them within the remaining discussion period. If our responses resolve your concerns, we would really appreciate it if you could consider reassessing your rating.
Thank you again for the time and effort you put into the review.
Best regards,
Authors
We appreciate the reviewer’s helpful comments and valuable feedback. We address each concern and question below.
[Q1] Termination statistics per model and method
We report the
</think>emission rate, EOS rate, truncation rate, and accuracy given</think>for every method on Qwen3.6-35B-A3B and Nemotron-3-Nano-30B-A3B below. On Qwen3.6, most of the accuracy loss comes from responses that never terminate. On Nemotron, truncation is low for all methods, and RTAQ gives the highest accuracy.We evaluate each method on LiveCodeBench v6 (LCB) and GPQA-Diamond (flexible-extract), with generation budgets of 100K (LCB) and 70K (GPQA) tokens. Truncation is the fraction of responses that reach this budget without EOS. Almost every response that emits
</think>also reaches EOS: the gap is at most 2.8%.Qwen3.6-35B-A3B. GPTQ fails to terminate on 24.7% of LCB responses, and GSQ on 16.0%. For most methods, accuracy given
</think>stays within about 3% of BF16. For example, on LCB, GSQ reaches 77.0% and RTAQ 77.1%, compared with 79.8% for BF16. This means that when these models do close their reasoning, they answer almost as accurately as BF16.</think></think></think></think>Nemotron-3-Nano-30B-A3B. Truncation is low for every method (1–10%), so the accuracy gap here comes mostly from terminated responses. RTAQ gives the highest accuracy on both benchmarks. On GPQA, it also gives the lowest truncation rate (1.3%) and the highest accuracy given
</think>(67.4%) among the quantized models.</think></think></think></think>All values are in %. Cyclic loop rates are given in Q3.
[Q2] Matched physical storage
Yes. At the same physical bit budget, RTAQ outperforms the baselines.
To answer this, we re-ran RTAQ at the physical budgets of GSQ (2.125 bits) and GPTQ (2.25 bits) on Qwen3.6-35B-A3B (LiveCodeBench v6). At equal storage, RTAQ achieves higher accuracy than GSQ (71.3% vs. 66.4%, +4.9%) and GPTQ (71.8% vs. 50.1%, +21.7%). For all methods, physical bits include every quantization overhead, such as codewords and affine parameters. Note that our default RTAQ uses 2.10 physical bits, which is already below every baseline. We will add the comparison with MC-MoE [2] at its matched budget (2.36 bits) in the final draft.
Termination: the fraction of responses that emit
</think>within the generation budget.