acceptodds
Under review as a conference paper at ICLR 2027

RTAQ: Reasoning Termination-Aware Quantization for Mixture-of-Experts Architecture

Abstract

Extreme low-bit post-training quantization is key to serving large-scale mixture-of-experts (MoE) language models, yet reasoning models quantized to around two bits often collapse on long-horizon reasoning tasks. We show that this collapse stems not from a loss of knowledge but from a failure to stop reasoning properly. When a 2-bit or near-1-bit reasoning model emits the end-of-thinking token (</think>), it terminates in every case we observe and answers nearly as accurately as its full-precision counterpart; most errors arise from reasoning traces that never close. We trace this termination decision to a small subset of routed experts (approximately 2% of all routed experts) that are consistently selected at the end-of-thinking position but are largely overlooked by standard sensitivity metrics. Building on this finding, we propose RTAQ (Reasoning-Termination-Aware Quantization), a weight-only quantization pipeline with three stages: (1) constructing a bank of activation-weighted, DP-optimal scalar codebooks for every expert and bit-width; (2) performing termination-aware bit allocation, which fixes termination experts at higher precision within the bit budget and assigns the remaining bits through knapsack optimization; and (3) applying grid-preserving recalibration, which reassigns codes using GPTQ-style error compensation and rescales expert outputs in under one second per expert. At an average bit-width of 2.0 for routed experts, RTAQ preserves over 95% of the full-precision performance on long-reasoning benchmarks such as AIME.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Extreme Quantization, Reasoning, Mixture-of-Experts, Post Training Quantization(PTQ)

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aPXzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the problem of "inference not terminating" that occurs in the MoE inference model after about 2-bit extremely low bit quantization. The author's core viewpoint is: The performance collapse after extremely low bit quantization is not entirely due to the loss of knowledge or reasoning ability. Many errors are caused by the model's inability to generate</hint>in a timely manner, resulting in inference loops, excessively long outputs, or reaching the maximum generation length. 2. Before generating the</hint>, the router will steadily select a small group of experts. The author refers to it as termination aware experts, accounting for approximately 1.6% -2.5% of all routing experts. If these experts are fixed to higher accuracy and the remaining experts are allocated bits based on sensitivity, it can significantly improve accuracy and generation length. Based on this, the paper proposes a three-stage RTAQ: -Build activation aware codebook banks with different quantization levels (K=3, ldots, 8); -Fix termination aware experts as \ (K=8 ), and other experts optimize allocation accuracy based on Fisher sensitivity and knapsack; -Use fixed codebook for GPTQ/OBS error compensation, affine parameter re fitting, and output gain correction. On three 30B-80B MoE models and five types of inference/code benchmarks, RTAQ achieved the highest average accuracy among all quantization methods with approximately 2.016-2.025 "logical bit/routing expert weights", while the output length was closer to BF16.

Critical strengths

The problem is novel and has practical significance. The paper specifically locates the low bit quantization failure to the generation and inference termination of</hint>, rather than broadly attributing it to "capability decline". This observation is very enlightening for the deployment of long inference models.

The method has formed a complete engineering pipeline. Codebook construction, mixed precision allocation, GPTQ style correction, and output scale correction can be combined into a relatively complete PTQ method without the need for retraining the model.

The experimental models and tasks cover a wide range. The paper tested three different 30B-80B MoE models and covered mathematical, scientific reasoning, and coding tasks. The average accuracies of RTAQ were 85.81%, 77.68%, and 81.03%, respectively, all of which were better than the comparison methods.

Critical weaknesses
  1. The evidence for the core claim that 'termination failure leads to performance collapse' is still insufficient The most important proposition of the paper is that as long as</hint>is successfully generated after quantification, the accuracy will approach BF16; Most errors come from the inability to close the thought process. However, the main text and appendix did not provide complete data to support this conclusion, such as: -The proportion of</hint>generated by each model and method; -EOS termination rate and the proportion of maximum length truncation; -Conditional accuracy rate \ (P (\ text {correct} \ mid</hint>) ); -Logit, ranking, or generation probability of</hint>; -What are the proportions of cyclic reasoning, excessive reasoning, and normal long reasoning. Figure 2b only shows the average generated length. The average length cannot distinguish true non termination, reasonable longer reasoning, and a small number of extreme outliers. After removing the termination aware expert pinning, the paper only reports changes in length without synchronously reporting accuracy and termination rate. In addition, the expert set \ (P ) is identified from successful trajectories that already contain</hint>, which may result in conditional selection bias. It is recommended to perform restorative interventions directly in the quantitative model: only restore the accuracy of P and measure the changes in close token probability, termination rate, and accuracy.
Questions for authors

What are the generation rate, EOS rate, truncation rate, and conditional accuracy of each model and method?

Is RTAQ still better than baseline under strictly the same final physical storage budget?

Can similar improvements be achieved by replacing P with random experts or experts who match the overall routing frequency?

3 comments

We appreciate the reviewer’s helpful comments and valuable feedback. We address each concern and question below.

[Q1] Termination statistics per model and method

We report the </think> emission rate, EOS rate, truncation rate, and accuracy given </think> for every method on Qwen3.6-35B-A3B and Nemotron-3-Nano-30B-A3B below. On Qwen3.6, most of the accuracy loss comes from responses that never terminate. On Nemotron, truncation is low for all methods, and RTAQ gives the highest accuracy.

We evaluate each method on LiveCodeBench v6 (LCB) and GPQA-Diamond (flexible-extract), with generation budgets of 100K (LCB) and 70K (GPQA) tokens. Truncation is the fraction of responses that reach this budget without EOS. Almost every response that emits </think> also reaches EOS: the gap is at most 2.8%.

Qwen3.6-35B-A3B. GPTQ fails to terminate on 24.7% of LCB responses, and GSQ on 16.0%. For most methods, accuracy given </think> stays within about 3% of BF16. For example, on LCB, GSQ reaches 77.0% and RTAQ 77.1%, compared with 79.8% for BF16. This means that when these models do close their reasoning, they answer almost as accurately as BF16.

Qwen3.6, LCBBitsAcc.</think>EOSTruncationAcc. given </think>
BF161677.697.296.93.179.8
GPTQ [1]2.2550.175.372.527.564.2
MC-MoE [2]2.3672.392.691.98.176.1
GSQ [3]2.12566.484.084.016.077.0
RTAQ2.0270.290.190.19.977.1
Qwen3.6, GPQABitsAcc.</think>EOSTruncationAcc. given </think>
BF161683.098.798.71.383.4
GPTQ [1]2.2572.987.587.512.580.2
MC-MoE [2]2.3672.694.394.35.776.1
GSQ [3]2.12580.696.896.83.282.3
RTAQ2.0279.395.195.14.981.9

Nemotron-3-Nano-30B-A3B. Truncation is low for every method (1–10%), so the accuracy gap here comes mostly from terminated responses. RTAQ gives the highest accuracy on both benchmarks. On GPQA, it also gives the lowest truncation rate (1.3%) and the highest accuracy given </think> (67.4%) among the quantized models.

Nemotron, LCBBitsAcc.</think>EOSTruncationAcc. given </think>
BF161666.296.796.93.168.4
GPTQ [1]2.2551.994.794.75.354.8
MC-MoE [2]2.3658.094.194.15.961.6
GSQ [3]2.12560.190.390.39.766.5
RTAQ2.02560.393.493.46.664.6
Nemotron, GPQABitsAcc.</think>EOSTruncationAcc. given </think>
BF161673.198.598.51.573.7
GPTQ [1]2.2559.498.198.11.959.3
MC-MoE [2]2.3663.898.398.31.764.4
GSQ [3]2.12537.794.994.95.138.7
RTAQ2.02567.098.798.71.367.4

All values are in %. Cyclic loop rates are given in Q3.

[Q2] Matched physical storage

Yes. At the same physical bit budget, RTAQ outperforms the baselines.

To answer this, we re-ran RTAQ at the physical budgets of GSQ (2.125 bits) and GPTQ (2.25 bits) on Qwen3.6-35B-A3B (LiveCodeBench v6). At equal storage, RTAQ achieves higher accuracy than GSQ (71.3% vs. 66.4%, +4.9%) and GPTQ (71.8% vs. 50.1%, +21.7%). For all methods, physical bits include every quantization overhead, such as codewords and affine parameters. Note that our default RTAQ uses 2.10 physical bits, which is already below every baseline. We will add the comparison with MC-MoE [2] at its matched budget (2.36 bits) in the final draft.

Physical bitsMethodAccuracy (%)Termination (%)
2.10RTAQ (default)70.290.1
2.125GSQ [3]66.484.0
2.125RTAQ71.389.3
2.25GPTQ [1]50.175.3
2.25RTAQ71.892.1

Termination: the fraction of responses that emit </think> within the generation budget.

[Q3] Replacing P\mathcal{P} with random or frequency-matched experts

No. Under the same bit budget, pinning random or frequency-based experts does not reproduce the gains from pinning P\mathcal{P}.

On Qwen3.6-35B-A3B (LiveCodeBench v6), we pinned 223 experts at 3 bits under the same budget (2.022 logical bits). We chose these experts in four ways: P\mathcal{P}, a random set, the top-223 by routing frequency, and the top-223 by Fisher sensitivity.

  • P\mathcal{P}: the only choice that clearly improves over the unpinned model. The </think> rate rises from 85.2% to 90.1%, and cyclic loops drop from 14.0% to 9.4%.
  • Other choices: they stay near or below the unpinned model, with </think> rates of 81.4–86.3% and loop rates of 13.0–17.8%.
Pinned experts (223)</think> (%)Cyclic loops (%)Accuracy (%)Avg. length
None (unpinned)85.214.069.742.1K
P\mathcal{P} (ours)90.19.470.237.0K
Random81.417.868.443.9K
Top by routing frequency86.313.068.741.3K
Top by Fisher sensitivity82.216.066.444.1K

Cyclic loops: responses that never emit </think> and end in verbatim repetition.

Blocking experts in the BF16 model shows the same pattern. Blocking routing to P\mathcal{P} lowers the termination rate on GPQA from 98.0% to 60%. Blocking two random sets of 223 experts matched to P\mathcal{P}'s routing frequency leaves it unchanged, at 98.0–98.5%.

[Weaknesses]

(1) Termination Failure Leads to Performance Collapse Most of the accuracy loss at near 2 bits comes from responses that never terminate, and restoring only the experts in P\mathcal{P} brings termination back. Below, we report all the requested statistics (Qwen3.6-35B-A3B, 3 samples per problem; LCB budget 100K tokens, GPQA 70K).

LCBAcc.</think>EOSTrunc.Acc. given </think>Cyclic loopsExcessiveNormal long
BF1677.697.296.93.179.81.51.39.7
GPTQ (2.25)50.175.372.527.564.223.21.59.2
MC-MoE (2.36)72.392.691.98.176.16.90.59.2
GSQ (2.125)66.484.084.016.077.014.02.09.9
RTAQ (2.02)70.290.190.19.977.19.40.57.6
GPQAAcc.</think>EOSTrunc.Acc. given </think>Cyclic loopsExcessiveNormal long
BF1683.098.798.71.383.40.70.79.8
GPTQ (2.25)72.987.587.512.580.212.50.03.4
MC-MoE (2.36)72.694.394.35.776.15.60.22.5
GSQ (2.125)80.696.896.83.282.33.00.28.2
RTAQ (2.02)79.395.195.14.981.94.70.29.8

Cyclic loops: never emit </think> and end in verbatim repetition. Excessive: never emit </think> and are truncated without repetition. Normal long: closed but longer than BF16's 90th-percentile closed length.

The main failure is getting trapped in repetitive loops before reaching the stopping point, not an inability to predict </think>. When RTAQ successfully terminates, its accuracy remains close to BF16 (77.1% vs. 79.8% on LCB), even under more aggressive quantization. Forcing termination recovers 41% of GPTQ's and 55% of MC-MoE's non-terminating failures, demonstrating that failure to stop is a major source of accuracy degradation.

(2) Causal Evidence for Termination-Aware Expert Protection As suggested, we restore only P\mathcal{P} (to 3 bits) inside the quantized model, keeping the total budget fixed, and compare it with same-size control sets (LCB):

Experts restored to 3 bits</think>Cyclic loopsAccuracy
None (2.02 bits)85.214.069.7
P\mathcal{P} (2.02 bits)90.19.470.2
None (1.63 bits)28.271.522.7
P\mathcal{P} (1.63 bits)37.461.834.6

Restoring P\mathcal{P} raises termination and cuts loops, and the gain grows when the budget is tighter (+11.9 accuracy points at 1.63 bits, paired p<10−4p<10^{-4}). Sets of the same size chosen by frequency, Fisher sensitivity, or at random do not reproduce this, so the effect is specific to P\mathcal{P} rather than a selection artifact. (Also refer our answer of Q3)

Reference

[1] E. Frantar et al. GPTQ: Accurate post-training quantization for generative pre-trained transformers. ICLR, 2023.

[2] W. Huang et al. Mixture compressor for mixture-of-experts LLMs gains more. ICLR, 2025.

[3] A. Dadgarnia et al. GSQ: Highly-accurate low-precision scalar quantization for LLMs via Gumbel-softmax sampling. arXiv:2604.18556, 2026.

AuthorsOpenReview ↗

Gentle reminder: author-reviewer discussion period concludes soon

Dear Reviewer aPXz,

Just a brief reminder that the author–reviewer discussion period concludes soon. We have posted our responses to your questions and would be grateful if you could take a look.

If you have any follow-up questions or remaining concerns, we would be happy to address them within the remaining discussion period. If our responses resolve your concerns, we would really appreciate it if you could consider reassessing your rating.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

Reviewer_12fpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the impact of extremely low-bit quantization on mixture-of-experts (MoE) reasoning models, with particular attention to cases where quantized models fail to emit </think> at the appropriate time, resulting in prolonged or repetitive reasoning. The authors observe that a small subset of routed experts is frequently activated when the model predicts the end-of-reasoning marker. Motivated by this observation, they propose RTAQ. The method first constructs an activation-weighted quantization codebook library, assigns 3-bit precision to experts closely associated with reasoning termination, and allocates precision to the remaining experts according to loss sensitivity. It then further reduces quantization error through GPTQ-style error compensation with fixed codebooks, refitting of affine parameters, and output scaling. The method is evaluated on three MoE models ranging from 30B to 80B parameters across five benchmarks. At an average logical bitwidth of approximately 2.02 bits for routed-expert weights, RTAQ achieves the highest average accuracy among the compared quantization methods on all three models. It also mitigates the increase in generation length caused by quantization in some experimental settings.

Critical strengths
  1. The problem is practically relevant, and the perspective is insightful. The paper connects quantization sensitivity to a specific model behavior: reasoning termination. It highlights that average reconstruction error or loss sensitivity measured on ordinary tokens may not adequately capture infrequent but important functions. This provides a worthwhile direction for further research on quantizing reasoning models.
  2. The overall accuracy results are competitive. In Table 1, RTAQ achieves average accuracies of 85.81%, 77.68%, and 81.03% on the three models, exceeding the strongest quantization baseline for each model by 2.74, 1.38, and 2.24 percentage points, respectively. The evaluation covers mathematics, scientific question answering, and code generation, suggesting that the benefits are not confined to a single task category.
  3. The behavioral analysis is well targeted. Figure 5 examines expert activations at reasoning termination positions and uses ordinary newline positions as a control. This partially rules out the possibility that the observed differences arise solely from the input token itself. Comparisons with conventional sensitivity metrics also provide additional motivation for explicitly protecting these experts.
  4. The appendix provides substantial implementation detail. The authors describe the construction of calibration data, the choice of expert coverage, the held-out set, the procedure for solving affine parameters, and the distinction between logical bitwidth and actual storage overhead. These details help clarify the method and provide a useful foundation for reproduction.
Critical weaknesses
  1. The central attribution is insufficiently supported. The paper does not systematically report termination/truncation rates or accuracy conditional on normal completion. Successfully completed responses may also favor easier questions. Paired analyses and controlled interventions are needed to support termination failure as the primary source of degradation.
  2. The ablations do not isolate the termination-aware contribution. Generation length alone cannot explain accuracy gains. Comparisons against protecting random, frequently activated, and high-Fisher-sensitivity experts under the same total bit budget, together with a separate Stage III ablation, are needed.
  3. The mathematical formulations are inconsistent. Assumption A5 in Appendix D.1 specifies a diagonal Fisher approximation, but Equation (32) equates its quadratic form with a squared inner product, which is generally incorrect. Equation (11) constrains total logical bitwidth, whereas Equation (35) penalizes codebook size. The authors should clarify the implemented formulations and whether these discrepancies affect the results.
  4. Expert-selection data and evaluation settings require clarification. Despite the calibration details in Appendix A.1, it remains unclear whether the AIME/GPQA responses in Figure 5 were also used to identify termination experts. Data separation, generation settings, sampling counts, random seeds, the LiveCodeBench version, and uncertainty estimates should be specified.
  5. Some results contradict the accompanying text. In Figure 2(b), GPQA generation length decreases from 18.3K to 17.1K when protection is removed, contradicting the claimed consistent increase. For Qwen3-Next, the improvement over the strongest baseline, TileQ, is 2.24 percentage points; the reported 3.31-point gain is relative to MC-MoE.
Questions for authors
  • Do paired analyses and controlled interventions support termination failure as the main source of degradation?
  • Under a strictly matched total storage budget, how much do termination-based expert selection and Stage III each contribute to accuracy and termination rates?
  • Does the implementation retain only the coordinate-wise diagonal of the Fisher approximation, or the full outer product within each token block? Does the allocation penalty depend on codebook size or logical bitwidth? Would correcting the manuscript discrepancies affect the results?
  • Are the data used to identify termination experts strictly separate from evaluation data? How stable is the selection across calibration domains, sample sizes, and activation thresholds?
6 comments

We thank the reviewer for the careful reading and constructive feedback. We address each concern and question below.

[Q1 & W1] Termination Failure as a Primary Source of Performance Degradation: Paired Analyses and Controlled Interventions

Our results demonstrate that reasoning termination failure is a major contributor to performance degradation under extreme low-bit quantization. To validate this, we conducted two additional experiments on Qwen3.6-35B-A3B.

(1) Paired Analysis

We analyzed cases where BF16 succeeds but quantized models fail, revealing the following findings:

BenchmarkMethod (bits)FailedNon-terminated
LCBGPTQ [1] (2.25)11353 (47%)
LCBMC-MoE [2] (2.36)3611 (31%)
LCBGSQ [3] (2.125)5324 (45%)
LCBRTAQ (2.02)4413 (30%)
GPQAGPTQ [1] (2.25)9038 (42%)
GPQAMC-MoE [2] (2.36)10121 (21%)
GPQAGSQ [3] (2.125)5210 (19%)
GPQARTAQ (2.02)5813 (22%)

Non-termination accounts for 47% and 45% of GPTQ and GSQ failures on LiveCodeBench, respectively. Across 16 method–benchmark combinations (4 methods × 4 benchmarks: LCB, GPQA, AIME25, and AIME26), non-termination strongly correlates with accuracy degradation (Pearson r=0.93r=0.93; Spearman ρ=0.87\rho=0.87).

(2) Output-Side Intervention: Forced Termination

We truncated the quantized model's reasoning at the corresponding BF16 reasoning length, appended </think>, and generated the final answer using the same quantized model. This intervention recovered 41% and 55% of eligible non-terminating failures for GPTQ and MC-MoE, respectively. These results indicate that many failures arise from impaired reasoning termination.

GPTQ [1] (50.1% accuracy; BF16 77.6%; 113 failed samples)

Cut point ttNon-terminated → correctRecovered / failed
BF16's length21/51 (41%)31/113 (27%)
Injected30/53 (57%)51/113 (45%)

MC-MoE [2] (72.3% accuracy; 36 failed samples)

Cut point ttNon-terminated → correctRecovered / failed
BF16's length6/11 (55%)6/36 (17%)
Injected7/11 (64%)13/36 (36%)

We will incorporate these additional experimental results into the revised manuscript.

[Q2 & W2-1] Ablation Study of Termination-Aware Expert Selection and Stage III Refinement Under a Matched Storage Budget

(1) Ablation Study of Termination-Aware Expert Selection

The benefit of protecting P\mathcal{P} cannot be explained solely by allocating additional bits to arbitrary or conventionally important experts.

We conducted controlled experiments under the same effective bitwidth budget of 2.0218 bits, comparing four expert selection strategies:

  1. Termination-aware selection (Ours): Assigning 3-bit precision to the 223 </think>-related experts identified in our paper.
  2. Random selection: Assigning 3-bit precision to 223 randomly selected experts.
  3. Routing-frequency-based selection: Assigning 3-bit precision to the 223 most frequently routed experts.
  4. Fisher-based selection: Assigning 3-bit precision to the 223 experts with the highest Fisher-based sensitivity scores.

The following table presents the experimental results. Experiments were conducted on LiveCodeBench v6, which exhibited the lowest reasoning termination rate among the evaluated benchmarks.

ConfigurationAccuracy (%)Closure Rate (%)Accuracy Given Closure (%)Avg. LengthPaired Closure ComparisonPaired Accuracy Comparison
Original Model77.6197.279.834.0K——
P\mathcal{P} (ours)70.2390.177.137.0KReferenceReference
Random 22368.4581.481.243.9K55:21 (p<0.001p<0.001)42:35 (p=0.49p=0.49)
Top-223 by Frequency68.7086.376.441.3K36:21 (p=0.06p=0.06)39:33 (p=0.56p=0.56)
Top-223 by Fisher66.4182.278.044.1K51:20 (p<0.001p<0.001)43:28 (p=0.10p=0.10)

(Closure rate: the fraction of generated samples in which the model emits the </think> token within the generation budget)

As shown in the table, our termination-aware expert selection consistently outperforms alternative selection strategies in reasoning termination.

Importantly, the paired comparisons show statistically significant improvements in closure rate over random selection and Fisher-based selection (both p<0.001p<0.001).

(2) Ablation Study of Stage III

The following table presents the ablation study of Stages II and III on LiveCodeBench v6. Both components contribute to improved reasoning termination and overall accuracy, with their combination achieving the highest closure rate and accuracy.

Termination-Aware PinningStage III RefinementAccuracy (%)Closure Rate (%)Avg. Length
✓✓70.290.137K
✓✗65.484.0—
✗✓69.7285.242K
✗✗63.3672.847K

We will incorporate these additional experimental results into the revised manuscript.

[Q3 & W3] Clarification and Correction of the Fisher Approximation and Bitwidth Allocation Formulations

We thank the reviewer for spotting this. Our method consistently uses the diagonal of the Fisher approximation. We will correct our description as follows:

(1) Diagonal Empirical Fisher Approximation

As can be verified in our supplementary code, the implementation retains only the coordinate-wise diagonal of the empirical Fisher at each token, consistent with Assumption A5:

\frac{1}{N}\sum_i\frac{T_i}{2}\sum_t\sum_d g_{t,d}^2\delta_{t,d}(e,K)^2= \frac{1}{N}\sum_i\frac{T_i}{2}\sum_t \left\|g_t\odot\delta_t(e,K)\right\|_2^2.$$ However, Eqs. (10), (32), and (33) incorrectly use $\langle g_t,\delta_t\rangle^2=\delta_t^\top g_tg_t^\top\delta_t$, which retains the full outer product within each token block. We will correct these equations to use the diagonal formulation. #### **(2) Logical Bitwidth Allocation** The allocation penalty depends on **logical bitwidth ($\log_2 K$)**, not codebook size ($K$). Specifically, each expert selects $$K_e(\lambda) \in \arg\min_{K\in\{3,\ldots,6\}} \left[ \Delta(e,K)+\lambda\log_2 K \right],$$ where $\lambda$ is determined through bisection to satisfy the average logical bitwidth constraint in Eq. (11). We will correct Eq. (35) and its subsequent budget expression to use $\log_2 K$ and $\sum_e\log_2 K_e$, respectively, ensuring consistency with Eq. (11). All reported numbers were produced by the implementation described above, so the corrections only align the text with it. **Re-running the released allocation reproduces the paper's per-expert levels exactly.**

[Q4 & W4] Data Separation, Expert Selection Stability, and Evaluation Protocol Clarification

The identified termination-related expert sets exhibit consistent overlap across diverse domains and answer formats.

To evaluate the stability of termination-related experts across domains, we used seven datasets spanning coding (CodeContests, APPS), mathematics (MATH, NuminaMath), and science (OpenBookQA, ARC, Nemotron Science). All evaluation data were strictly excluded, and samples were balanced across difficulty levels and subject categories.

We generated reasoning trajectories using the original Qwen3.6-35B-A3B BF16 model and applied our expert selection criteria to the routed experts activated at </think> positions in successfully terminated responses.

The table below reports the number of termination positions, selected experts, and their overlap with the reference set (P\mathcal{P}) for each dataset.

Dataset# Termination Positions# Selected ExpertsOverlap with P\mathcal{P}Precision w.r.t. P\mathcal{P}
APPS711811600.88
ARC951771640.93
CodeContests511771570.89
MATH891631570.96
Numina871641620.99
OBQA962141740.81
Nemotron Science3641491460.98

The identified expert sets exhibit substantial overlap with the reference set P\mathcal{P}, indicating that </think>-specific expert routing patterns remain largely consistent across datasets.

In summary, the </think>-specific expert group (P\mathcal{P}) defined in our paper remains stable across diverse datasets, suggesting that these termination-related routing patterns reflect an intrinsic characteristic of the model rather than dataset-specific behavior.

[W5] Clarification of Generation-Length Trends and Baseline Performance Comparisons

We clarify both points below.

(a) Generation-Length Trends on GPQA-Diamond On LiveCodeBench, where termination is a bottleneck, removing pins increases generation length (37.0K → 42.1K) and reduces termination rate (90.1% → 85.2%, p=0.02p=0.02). However, on GPQA-Diamond, since termination is not a main bottleneck, both models terminate at 95%, while the unpinned model generates fewer tokens (17.1K vs. 18.9K) with slightly lower conditional accuracy (78.9% vs. 81.9%). We will revise the manuscript.

(b) Performance Comparison with the Strongest Baseline We agree that TileQ [4] is the strongest baseline on Qwen3-Next, where RTAQ achieves a 2.24-point improvement (81.03% vs. 78.79%). The reported 3.31-point improvement is relative to MC-MoE [2] (77.72%).

We selected MC-MoE [2] as the strongest baseline based on its highest average accuracy across all three models (79.03%), compared with TileQ [4] (76.16%). But we will revise the manuscript to distinguish the strongest baseline on each model from the strongest baseline averaged across models.

Once again, we sincerely thank the reviewer for the thoughtful and constructive feedback. We greatly appreciate your time and consideration, and we hope our responses adequately address your concerns.


References

[1] E. Frantar et al. GPTQ: Accurate post-training quantization for generative pre-trained transformers. ICLR, 2023.

[2] W. Huang et al. Mixture compressor for mixture-of-experts LLMs gains more. ICLR, 2025.

[3] A. Dadgarnia et al. GSQ: Highly-accurate low-precision scalar quantization for LLMs via Gumbel-softmax sampling. arXiv:2604.18556, 2026.

[4] H. Gu et al. TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling. arXiv preprint arXiv:2605.09281, 2026.

Reviewer_vNzPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Idea: at ~2 bits, quantized MoE reasoning models mostly do not lose knowledge, they lose ability to stop reasoning (no closing token, loops until budget). Authors show termination decision sits in small subset of routed experts (~2%), activated right before closing token, and standard sensitivity criteria (Fisher, Hessian trace, output error, NLL) rank these experts low.

RTAQ has three stages: bank of activation-weighted scalar codebooks per expert (exact weighted 1D k-means, dynamic programming), pin termination experts at 3 bits and allocate rest by Fisher sensitivity via Lagrangian knapsack with trust region, then GPTQ-style correction inside fixed codebook plus per-expert scalar gain. On three MoE models and five benchmarks at ~2.02 bits, RTAQ beats GPTQ, MC-MoE, GSQ, TileQ on every model with fewer bits, keeps 97-98% of BF16 on Qwen models, generation length near BF16

Overall, a good paper; however, I would recommend further improvement on text/presentation

Critical strengths
  • Finding is solid, evidence layered properly: activation skew at closing positions (Fig 2a), newline control 83% vs 41% (Fig 5a) kills token confound, Fig 5b shows usual criteria miss these experts (39% outside Fisher top 10%).
  • Pipeline is principled: exact DP, Lagrangian relaxation with proper lemmas, trust region, on-grid GPTQ (Prop 3), scalar gain for SiLU/down shrinkage. Table 2 separating exact vs approximate is honest.
  • Calibration appendix careful: coverage-greedy selection, on-policy answers, held-out split, split-half reliability 0.47 to 0.69.
  • Strong results with fewer bits. Bit accounting honest, I rederived Eq. 36 and affine overhead, correct
Critical weaknesses
  • No seeds, no intervals in Table 1. AIME is n=180, some per-benchmark gaps text leans on are inside noise. Aggregate conclusions still hold.
  • Paper is about termination, but non-termination rates are never reported, only average generation length. This table is missing and cheap to make.
  • tau=0.8 and 3-bit pin never ablated. How sensitive are |P|, accuracy, length?
  • Only routed experts quantized (defensible, more than 97% of params, but headline is routed-expert figure). Packing discussed only for ternary. No QAT comparison.
  • Abstract says >95% AIME retention, but Nemotron AIME25 is 81.1 vs 87.8, that is 92%

All fixable I believe. Please address during the rebuttal period

Questions for authors
  1. Non-termination rates per method and model please, direct test of your story.
  2. Ablations on tau and pin width, stability of P across calibration data.
  3. Multi-seed Table 1, are gaps stable?
  4. Stage 1 cost, kernel throughput for K in {4,5,6}?
  5. At 1.8 or 1.6 bits, does pinning still work? Why?
4 comments

We thank the reviewer for the careful review and the concrete requests. We answer each point below.


[Q1, W2] Non-termination rates

We report the non-termination rates below. Non-termination is a major cause of accuracy loss at ~2 bits, and RTAQ keeps it low while using the fewest bits.

We define the non-termination rate as the fraction of responses that never emit </think>. We measured this rate for each method on two models.

Qwen3.6-35B-A3BLCBGPQAAIME26
BF162.8%1.3%2.2%
GPTQ [1]24.7%12.5%6.7%
MC-MoE [2]7.4%5.7%4.4%
GSQ [3]16.0%3.2%7.8%
RTAQ9.9%4.9%0.0%
Nemotron-3-Nano-30B-A3BLCBGPQAAIME26
BF163.3%1.5%2.2%
GPTQ [1]5.3%1.9%2.2%
MC-MoE [2]5.9%1.7%3.3%
GSQ [3]9.7%5.1%8.9%
RTAQ6.6%1.3%1.1%

To check how much of the accuracy loss comes from non-termination, we took the questions that the BF16 model answers correctly but the quantized model gets wrong (Qwen3.6), and counted how many of these responses never to terminate. For the baselines, 19–71% of these errors are non-terminated responses.

MethodLCBGPQAAIME26
GPTQ [1]53 / 113 (47%)38 / 90 (42%)5 / 11 (45%)
MC-MoE [2]11 / 36 (31%)21 / 101 (21%)3 / 8 (38%)
GSQ [3]24 / 53 (45%)10 / 52 (19%)5 / 7 (71%)
RTAQ13 / 44 (30%)13 / 58 (22%)0 / 3 (0%)
(# non-terminated) / (# BF16-correct but quantized-incorrect)

[Q2, W3] Sensitivity to τ\tau and pin width, and stability of P\mathcal{P}

P\mathcal{P} is stable across calibration data, and accuracy changes little with τ\tau or with the size and width of the pinned set.

Stability of P\mathcal{P}. We re-identified termination experts on seven datasets that are disjoint from all evaluation benchmarks. These cover code (APPS, CodeContests), math (MATH, NuminaMath), and science (ARC, OpenBookQA, Nemotron-Science). We used Qwen3.6 BF16 trajectories and τ=0.8\tau=0.8. On every dataset, 81–99% of the selected experts belong to the paper's P\mathcal{P}.

Dataset# </think> positions# selected# in P\mathcal{P}Precision
APPS711811600.88
CodeContests511771570.89
MATH891631570.96
NuminaMath871641620.99
ARC951771640.93
OpenBookQA962141740.81
Nemotron-Science3641491460.98

Sensitivity to τ\tau in selection. We varied τ\tau on the same seven datasets. As τ\tau increases, precision approaches 1.0. This means τ\tau controls how many experts are selected, not which ones.

Datasetτ\tau=0.60.70.80.9
APPS0.750.790.880.99
CodeContests0.740.810.890.93
MATH0.800.870.961.00
NuminaMath0.830.920.991.00
ARC0.810.870.930.98
OpenBookQA0.720.760.810.87
Nemotron-Science0.850.930.981.00

Effect on ∣P∣|\mathcal{P}|, accuracy, and length. We ran Qwen3.6 on LCB with each setting:

  • τ=0.6\tau=0.6,
  • no pinning.

∣P∣|\mathcal{P}| grows from 223 to 336 as τ\tau decreases, while accuracy stays within about 1 point. Pinning mainly improves termination: the termination rate rises from 85.2% to 90.1%, loops drop from 14.0% to 9.4%, and length falls from 42.1K to 37.0K tokens.

Setting∣P∣\lvert\mathcal{P}\rvertAcc.TerminationLoopsLength
τ=0.8\tau=0.8 (paper)22370.290.1%9.4%37.0K
τ=0.6\tau=0.633669.088.0%9.9%38.5K
No pinning069.785.2%14.0%42.1K

Pin width. We pinned a larger set of 471 experts at 3 bits (K=8K=8) and at 4 bits (K=16K=16). LCB accuracy was 71.3 and 70.5, respectively, compared with 70.2 for the paper's setting. Neither a larger set nor a wider pin brings further gains, so 3-bit pinning of P\mathcal{P} is sufficient.


[Q3, W1] Multi-seed Table 1

Across multiple seeds, our method achieves competitively high performance despite using the low bitwidth

We re-ran 4 long-reasoning benchmarks of Table 1 on Qwen3.6 with 3 seeds and report mean ± std across seeds. RTAQ's largest drop from BF16 is 7.4 points (LCB). The largest drops for the baselines are 27.5 points (GPTQ, LCB), 10.4 points (MC-MoE, GPQA), and 11.2 points (GSQ, LCB). We agree that some per-benchmark gaps are within noise, such as RTAQ vs. GSQ on GPQA. We will revise the text to rely on the aggregate and worst-case comparisons, and we will add these intervals to Table 1.

MethodBitsAIME25AIME26GPQALCB
BF161691.11 ± 3.8591.11 ± 5.0983.00 ± 1.5477.61 ± 1.92
GPTQ [1]2.2580.00 ± 0.0081.11 ± 1.9272.90 ± 2.7850.13 ± 2.33
MC-MoE [2]2.3688.89 ± 1.9285.56 ± 6.9472.56 ± 2.3972.26 ± 0.44
GSQ [3]2.1384.44 ± 3.8586.67 ± 3.3380.64 ± 1.6266.41 ± 2.29
RTAQ2.0290.00 ± 0.0093.33 ± 0.0079.29 ± 2.8170.23 ± 1.53

[Q4] Stage I cost and kernel latency for K∈{4,5,6}K \in \{4,5,6\}

Stage I takes 0.94 hour on 4 GPUs and runs once per model. The MoE Kernel for every KK is faster than BF16, and the full 2.02-bit model decodes about 2× faster than BF16.

Cost (Qwen3.6, all 10,240 routed experts, 4× RTX PRO 6000):

  • Stage I: 0.94 hour on four GPUs. This covers the codebook bank for K=3,…,8K=3,\dots,8, which is reused for any bit budget.
  • Stage II: the knapsack takes a few seconds on a CPU.
  • Stage III: about half an hour on 112 CPU cores.

Kernel latency. We measured the latency of one MoE layer (µs) for each KK on a single RTX PRO 6000 GPU with Qwen3.6, where every expert in the layer uses the same KK. Speedup over BF16 is shown in parentheses.

Concurrent tokensBF16K=4K=4K=5K=5K=6K=6
141.322.3 (1.85×)27.1 (1.52×)27.3 (1.51×)
16420.874.8 (5.63×)106.4 (3.95×)104.5 (4.03×)
48788.9140.9 (5.60×)207.6 (3.80×)207.8 (3.80×)
1281013.7183.5 (5.52×)264.7 (3.83×)263.4 (3.85×)
2561053.2242.9 (4.34×)331.4 (3.18×)325.3 (3.24×)

End-to-end (mixed-KK RTAQ at 2.02 bits, single RTX PRO 6000):

  • MoE layer: 4.0–4.3× faster than BF16 for batch size ≥ 16.
  • Decoding: 3,182 tok/s at batch 48 (2.09×) and 4,880 tok/s at batch 128 (1.92×).
  • Memory: expert memory drops from 64.4 GB to 10.3 GB, which allows 3.8× more KV cache.

[Q5] Pinning at lower bit budgets

Yes. Pinning still works, and its benefit grows as the budget shrinks: +0.5 points at 2.02 bits versus +11.9 points at 1.63 bits.

We lowered the knapsack budget to 1.63 bits and kept P\mathcal{P} at 3 bits (Qwen3.6, LCB). Even at 1.63 bits, pinned models stay accurate once they terminate (78.2%, vs. 77.1% at 2.02 bits). Without pinning, this conditional accuracy falls to 61.3%. The reason is that at extreme budgets, the experts that decide when to stop are themselves heavily quantized. The model then loops more often (71.5% vs. 61.8% with pinning) and answers less accurately even when it does terminate.

BitsSettingAcc.TerminationAcc. given terminationLoopsLength
2.02Pinning70.290.1%77.19.4%37.0K
2.02No pinning69.785.2%77.614.0%42.1K
1.63Pinning34.637.4%78.261.8%71.3K
1.63No pinning22.728.2%61.371.5%77.1K

Termination-aware pinning becomes increasingly beneficial under more aggressive quantization, improving accuracy by 11.9 points at 1.63 bits compared with only 0.5 points at 2.02 bits on Qwen3.6 LiveCodeBench.


[W4] Quantization scope and comparison with QAT

We quantize only routed experts because they hold about 90% (Qwen3.6) to 95% (Qwen3-Next) of the parameters and dominate storage. In our QAT experiment, RTAQ outperforms a GPTQ + STE baseline without any training.

We compare RTAQ with QAT by fine-tuning the GPTQ [1]-quantized Qwen3.6-35B-A3B model using STE [4] (3 epochs, learning rate 5×10−55\times10^{-5}, on a math, science, and code fine-tuning set). QAT recovers much of GPTQ's loss (73.15 → 81.29). RTAQ, a post-training method, achieves higher accuracy on every benchmark with fewer bits (85.74 average). QAT can be applied on top of RTAQ, which we leave for future work.

ModelBitsGSM8KAIME25AIME26GPQA (S/F)LCBAvg. (S/F)
BF161696.891.191.182.3 / 83.077.3587.73 / 87.90
GPTQ [1]2.2593.2575.5683.3369.36 / 69.3644.2773.15 / 73.15
GPTQ [1] + QAT (STE [4])2.2595.7085.5691.1170.20 / 71.0463.8781.29 / 81.46
RTAQ (no training)2.0296.690.093.378.6 / 79.370.285.74 / 85.88

[W5] AIME retention claim in the abstract

The reviewer is correct, and we will fix the abstract. RTAQ's AIME retention ranges from 92.4% (Nemotron, AIME25) to 102.4% (Qwen3.6, AIME26). Its average retention across the five benchmarks is 94.0–97.7%. We will revise the abstract to: "RTAQ retains 94–98% of the full-precision average accuracy across three MoE models."

ModelAIME25AIME26Average
Qwen3.6-35B-A3B98.8%102.4%97.7%
Nemotron-3-Nano-30B-A3B92.4%95.1%94.0%
Qwen3-Next-80B-A3B98.6%100.0%97.7%

Each value is RTAQ accuracy divided by BF16 accuracy (Table 1).


References

[1] E. Frantar et al. GPTQ: Accurate post-training quantization for generative pre-trained transformers. ICLR, 2023.

[2] W. Huang et al. Mixture compressor for mixture-of-experts LLMs gains more. ICLR, 2025.

[3] A. Dadgarnia et al. GSQ: Highly-accurate low-precision scalar quantization for LLMs via Gumbel-softmax sampling. arXiv:2604.18556, 2026.

[4] Y. Bengio et al. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv:1308.3432, 2013.

AuthorsOpenReview ↗

Gentle reminder: author-reviewer discussion period concludes soon

Dear Reviewer vNzP,

Just a brief reminder that the author–reviewer discussion period concludes soon. We have posted our responses to your questions and would be grateful if you could take a look.

If you have any follow-up questions or remaining concerns, we would be happy to address them within the remaining discussion period. If our responses resolve your concerns, we would really appreciate it if you could consider reassessing your rating.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.