acceptodds
Under review as a conference paper at ICLR 2027

ORQA: Fast and Memory-Efficient Hybrid Reasoning with a Quantized Reasoner

Abstract

Reasoning improves the capability of large language models, but long reasoning traces increase latency and decoding cost. Hybrid reasoning reduces this cost by offloading intermediate reasoning to a lightweight model, which we call an efficient reasoner. Existing methods, however, require extra model parameters and KV cache, increasing memory consumption. Moreover, the capability gap between the reasoner and the target LLM often limits speedup or reduces accuracy. To this end, we propose ORQA, a hybrid reasoning framework that improves speed and accuracy over prior methods without adding memory overhead. Instead of loading an extra reasoner, ORQA uses a quantized counterpart of the target LLM and a shared KV cache pool. We propose weight infusion, which integrates parameters between the target LLM and reasoner, eliminating separate model storage while maintaining lossless inference for both. We further devise Quant-ifier, a routing signal designed for a quantized reasoner that predicts token disagreement using the quantization error distribution. Experiments show that ORQA improves speed and accuracy over prior methods with no additional memory.

Primary area:
generative models
Keywords:
LLM, Hybrid Reasoning, Reasoning, Acceleration, Quantization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_HtciOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the trade-off between reasoning accuracy and inference efficiency in large language models and proposes ORQA, a hybrid reasoning framework that combines a full-precision target model with its low-bit quantized counterpart, rather than relying on an independent small reasoner. The framework addresses three key challenges. First, Weight Infusion introduces a unified weight encoding scheme that enables lossless reconstruction of the original model weights while supporting quantized inference, reducing the memory overhead of maintaining both models. Second, Quant-ifier estimates the probability of quantization-induced reversals in the ranking of the top-two candidate tokens based on calibrated hidden-state error statistics, and accumulates this risk to determine when the full-precision model should intervene. Third, KV Transplant enables the two models to share a KV cache pool, allowing the quantized reasoner to reuse KV states computed by the full-precision model while the full-precision model recomputes and overwrites the reasoner-generated KV entries upon intervention, thereby eliminating the need for a separate reasoner cache and improving inference quality. The quantized reasoner is further improved through reasoning-oriented calibration and scale-only distillation. Experiments on AIME, GPQA-Diamond, and LiveCodeBench show that ORQA achieves up to 2.9× higher token throughput than full-precision inference, with relatively small accuracy degradation on the 32B model and no additional peak GPU memory usage in the evaluated settings, while offering a favorable accuracy–throughput trade-off compared with the evaluated hybrid reasoning baselines.

Critical strengths

S1. Practical system design. ORQA addresses a real deployment challenge in hybrid reasoning: an additional reasoner normally requires extra model weights and KV cache. Using a quantized version of the target model, together with shared weight encoding and KV storage, provides a concrete way to reduce this overhead. The paper also describes the runtime weight reconstruction and KV refresh procedures.

S2. Promising accuracy–efficiency results. The Qwen3-32B results are encouraging. On AIME, ORQA increases throughput from 18.8 to 55.1 TPS while accuracy changes from 78.3% to 75.6%. On GPQA, throughput rises from 19.2 to 49.1 TPS with almost no accuracy loss. Peak GPU memory is also lower than target-only inference in the reported setting. These results show practical potential beyond a purely conceptual proposal.

Critical weaknesses

W1. Missing comparisons with standalone quantization. The paper mainly compares ORQA with full-precision inference and hybrid systems using separate small reasoners, but not with simply running a 4-bit or 8-bit version of the same target model. Without these baselines, it is difficult to judge whether ORQA offers enough benefit in accuracy, speed, and memory to justify its additional complexity.

W2. Unclear connection between routing decisions and actual benefit. Quant-ifier decides when to invoke the full-precision model based on the predicted risk of reversing the top-two token rankings. Yet a reversal may have little effect on the generated answer. For example, if two tokens have probabilities of 0.5001 and 0.4999, a tiny change can reverse their order while leaving the sampling distribution almost unchanged. The paper does not directly show that positions with higher routing scores are the ones where full-precision intervention improves final-answer accuracy.

W3. Incomplete evaluation protocol and uncertainty reporting. The paper specifies the datasets, decoding settings, and scoring rules, but does not clearly state how many responses are sampled per question or how they are aggregated. The main results also lack standard errors or confidence intervals. These details are important for interpreting the smaller accuracy differences between methods.

W4. Limited router baselines. The paper compares Quant-ifier with entropy-based routing, but does not evaluate simpler alternatives such as top-two logit-margin routing or fixed-interval interventions. Comparing these methods under similar inference costs would help show whether Quant-ifier's more elaborate error model is needed.

W5. Missing end-to-end time-to-answer evaluation. The paper reports token throughput but not the time needed to complete each problem or the lengths of the generated reasoning traces. ORQA and the full-precision model may produce different numbers of tokens, so higher TPS does not necessarily mean faster answers. Per-question wall-clock times and output lengths would better establish the practical speedup.

Questions for authors

Q1. Inconsistent Qwen3-8B AIME operating points. Table 1 reports 63.3% AIME accuracy with an 18.4% target-LLM token ratio for Qwen3-8B ORQA, whereas the final ORQA row in Table 2 reports 66.7% accuracy with an 8.2% ratio. Could the authors clarify the experimental differences underlying these results (e.g., routing thresholds, decoding samples or settings) and how the operating points were selected?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Htci (1/5)

We thank the reviewer for the constructive feedback, which has helped us strengthen our manuscript. We are preparing a revised version incorporating the reviewer’s valuable suggestions, and will make it available as soon as possible. In the meantime, we would like to provide our response below.

(W1) Compare accuracy, speed, and memory with standalone, 4bit & 8bit quantized models.

→\to We thank the reviewer for the valuable comment.

We additionally compare ORQA against standalone 4-bit and 8-bit quantized Qwen3-32B models using GPTQ and AWQ. The results in Table R27 show that ORQA achieves substantially higher throughput and lower end-to-end latency at comparable accuracy.

Table R27. Accuracy, TPS, latency, and memory comparison of ORQA with standalone quantized models on AIME.

MethodAcc. ↑\uparrowTPS ↑\uparrowRelative TPS ↑\uparrowE2E latency (s) ↓\downarrowLatency speedup ↑\uparrowPeak memory (GiB) ↓\downarrow
Qwen3-32B (BF16)78.3%18.81.0$\times$748.31.0$\times$74.0
GPTQ 4-bit75.0%35.41.9$\times$429.91.7$\times$33.7
AWQ 4-bit66.7%35.91.9$\times$360.12.1$\times$33.7
GPTQ 8-bit71.7%25.61.4$\times$619.31.2$\times$46.4
AWQ 8-bit80.0%25.61.4$\times$603.41.2$\times$46.4
ORQA (Ours)75.6%55.12.9$\times$273.42.7$\times$71.5

Compared to 4-bit GPTQ, ORQA achieves comparable accuracy (75.6% vs. 75.0%) while providing 1.56$\times$ higher throughput and 1.57$\times$ faster end-to-end inference. Compared to 8-bit AWQ, ORQA achieves 2.15$\times$ higher throughput and 2.21$\times$ faster end-to-end inference, with a 4.4 percentage-point accuracy difference.

Standalone quantization achieves lower memory consumption by reducing the precision of the entire model. In contrast, ORQA retains the full-precision target LLM and uses a 2-bit quantized counterpart to accelerate intermediate reasoning. Thus, ORQA targets a different operating point: substantially faster inference while retaining access to full-precision target-model computation, without exceeding its original memory footprint.

AuthorsOpenReview ↗

Response to Reviewer Htci (2/5)

(W2) Unclear connection between Quant-ifier's routing decisions and actual benefit: a top-2 reversal (e.g., 0.5001 vs. 0.4999) may barely change the sampling distribution or final-answer accuracy.

→\to We thank the reviewer for this point. We agree that Quant-ifier estimates the risk of a reversal, not its downstream effect. We answer in two parts: the end-to-end benefit, and the reviewer's example.

1. End-to-end benefit (Table 2). In the ablation, the row "+Calibration recipes" uses the entropy-based router, and the next row "+Quant-ifier" changes only the router. Accuracy rises from 58.9% to 66.7% on AIME and from 50.3% to 52.5% on GPQA-Diamond, while the LLM ratio (the fraction of tokens generated by the target LLM) falls from 11.4% to 8.2% and from 17.3% to 13.8%. Quant-ifier thus reaches higher final-answer accuracy with 20–28% fewer target interventions.

2. The 0.5001 / 0.4999 example. The example assumes that when the quantized reasoner is nearly tied between two tokens, the target LLM is nearly tied as well, so invoking it changes little. We tested this assumption directly.

Setup. For each of Qwen3-8B and Qwen3-32B, we used 192 reasoning traces (985K and 920K tokens) with the target LLM and its 2-bit quantized reasoner. At every token, both models receive the same prefix, and we compare their sampling distributions (T = 0.6, top-p 0.95, as in our experiments) by L1 distance (0 = identical, 2 = no overlap). For scale, 0.50 / 0.50 versus 0.55 / 0.45 has an L1 distance of 0.1.

(a) When the reasoner is nearly tied, the target LLM usually is not. We take every token where the reasoner's top-2 probabilities are within 0.02 of each other (a near-tie) and look at the target LLM at the same token. The analysis results are shown in Table R28.

Table R28. Target-model sampling behavior at reasoner near-ties.

At reasoner near-tiesQwen3-8BQwen3-32B
Share of all tokens0.24%0.22%
Target LLM's probability gap between the same two tokens (median)0.480.52
L1 distance ≥\geq 0.191%93%
Target LLM prefers the reasoner's second token48.8%48.6%

Near-ties are rare, and at them the target LLM typically separates the two tokens clearly (e.g., 0.74 vs 0.26). This follows from Eq. 10: when the reasoner's margin mQ≈0m_Q \approx 0, the target LLM's margin is determined by the quantization error herr⋅dh_{err} \cdot d, which is not small for a 2-bit reasoner. Target intervention therefore replaces a near-uniform draw between two tokens with the target LLM's distribution. The last row also shows that Quant-ifier's estimate is accurate here: it assigns p^≈0.5\hat{p} \approx 0.5 (Eq. 13) at near-ties, and the target LLM indeed reverses the pair in about half of them.

(b) The reviewer's scenario is a negligible share of target interventions. We count target interventions that occur at a reasoner near-tie and change the sampling distribution by an L1 distance below 0.1. Table R29 shows that the near-ties take 0.1% to 0.22% of the total target interventions, empirically validating that the scenario is rare.

Table R29. Proportion of target interventions at reasoner near-ties with minimal sampling-distribution changes.

Tokens routed to the target LLM5%10%15%20%
Qwen3-8B0.23%0.17%0.14%0.10%
Qwen3-32B0.22%0.13%0.11%0.08%

In summary, Table 2 shows that Quant-ifier's routing decisions yield higher benchmark accuracy with fewer target interventions. The near-tie case raised by the reviewer is rare. When it does occur, the target LLM's sampling distribution clearly differs from the reasoner's in over 90% of cases.

AuthorsOpenReview ↗

Response to Reviewer Htci (3/5)

(W3) Report standard errors, inference settings for Table 1.

→\to  We report additional experimental results with standard errors across three runs in Table R4. As for the inference setting, ORQA and other hybrid reasoning baselines generate one response per question using standard autoregressive decoding. Each response is evaluated directly, without aggregation across multiple responses. 

Table R4. Accuracy with standard errors.

Method32B·AIME32B·GPQA32B·LiveCodeBench8B·AIME8B·GPQA8B·LiveCodeBench
Target LLM only78.3% ±\pm 1.0%59.6% ±\pm 0.6%64.3% ±\pm 0.8%66.7% ±\pm 3.3%57.4% ±\pm 0.6%54.5% ±\pm 1.5%
BERT43.3% ±\pm 5.0%41.4% ±\pm 1.1%44.8% ±\pm 1.5%26.7% ±\pm 1.7%36.2% ±\pm 1.6%28.6% ±\pm 2.0%
Causal LLM70.0% ±\pm 1.0%37.4% ±\pm 1.3%38.6% ±\pm 1.9%58.3% ±\pm 3.3%32.7% ±\pm 1.1%20.7% ±\pm 2.4%
MF41.7% ±\pm 4.4%38.4% ±\pm 1.8%38.1% ±\pm 1.8%21.1% ±\pm 2.0%35.4% ±\pm 1.3%22.9% ±\pm 0.7%
SW_RANKING48.3% ±\pm 2.9%42.4% ±\pm 1.8%42.4% ±\pm 2.4%33.3% ±\pm 0.0%34.8% ±\pm 2.8%27.6% ±\pm 1.3%
R-Stitch66.7% ±\pm 4.4%54.6% ±\pm 0.8%56.2% ±\pm 2.1%46.7% ±\pm 1.0%39.9% ±\pm 2.0%37.1% ±\pm 1.7%
R2R70.0% ±\pm 2.9%60.1% ±\pm 2.8%60.0% ±\pm 1.4%58.3% ±\pm 2.5%47.0% ±\pm 3.7%43.8% ±\pm 3.6%
ORQA (Ours)75.6% ±\pm 3.9%59.4% ±\pm 2.6%62.4% ±\pm 2.7%63.3% ±\pm 3.5%53.5% ±\pm 2.0%46.7% ±\pm 1.2%

The results show that ORQA's performance improvements are substantial relative to the observed standard errors, supporting the robustness of its gains.

(W4) Limited router baselines: compare Quant-ifier with routing alternatives such as top-two logit-margin routing or fixed-interval interventions.

→\to We conduct additional experiments comparing Quant-ifier against entropy-based, top-2 logit margin, and fixed-interval routing at comparable LLM ratios. The results show that Quant-ifier consistently achieves higher accuracy than all three baselines at comparable or lower target-LLM usage

Table R15. Accuracy and LLM usage comparison of Quant-ifier and routing baselines at comparable target-token budgets.

AIMEGPQA
Acc. ↑\uparrowLLM Ratio ↓\downarrowAcc. ↑\uparrowLLM Ratio ↓\downarrow
entropy61.7%9.85%50.5%27.8%
top-two margin56.67%9.63%50.0%28.7%
fixed intervention56.7%10.0%49.0%25.0%
Quant-ifier66.7%8.2%53.5%26.5%

We keep the calibrated 2-bit reasoner and KV transplant fixed across all routing methods. We sweep routing thresholds to obtain comparable LLM ratios.

Quant-ifier consistently improves accuracy over all three routing baselines on both AIME and GPQA, despite using comparable or fewer target-LLM tokens. These results demonstrate that Quant-ifier provides more effective target intervention than conventional routing methods at comparable target-token budgets.

Reviewer_TQyoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ORQA, a hybrid reasoning framework that uses a quantized version of the target LLM as a "reasoner" to accelerate inference without additional memory overhead. The key insight is that a quantized counterpart of the target model can serve as an effective draft/reasoner model while sharing parameters and KV cache infrastructure. The authors propose three main technical contributions: (1) Weight Infusion, a lossless compression technique that encodes target model weights using 4-bit delta offsets relative to quantization scales, enabling both models to share a single parameter representation; (2) Quantifier, a routing mechanism that estimates the probability of token disagreement between the quantized reasoner and target model by modeling quantization error in logit margins; and (3) KV Transplant, a cache-sharing scheme where the reasoner reuses the target model's KV cache entries while the target model can overwrite reasoner-generated KV entries. Experiments on AIME, GPQA, and LiveCodeBench with Qwen3-32B and Qwen3-8B models demonstrate that ORQA achieves near-target-only accuracy with significantly higher throughput (up to 2.9×) and no additional GPU memory beyond the target model's footprint.

Critical strengths

. Novel and practical problem formulation. The paper identifies an important inefficiency in existing hybrid reasoning approaches: they require loading separate reasoner models and maintaining separate KV caches, which consumes significant additional memory. The proposed solution—using a quantized counterpart of the target model—is elegant and well-motivated, as it naturally inherits target model knowledge while requiring no independent reasoner storage.

  1. Weight Infusion is technically innovative. The observation that BF16 weight exponents and quantization scale exponents occupy overlapping narrow ranges (Figure 3) is empirically validated across multiple models and quantization methods (Figures 7-11, Table 8). The delta encoding scheme is principled, achieves ~99.98% coverage with 4-bit deltas, and reduces target model weight storage by ~20% while enabling lossless reconstruction. The runtime decompression strategies (fused small-M path, Triton kernel for large-M) are practical.

  2. Quantifier is well-designed and empirically validated. The derivation connecting quantization error to token disagreement probability through logit margin projection is theoretically sound. The low-rank covariance approximation (reducing O(D²) to O(Dr)) makes it computationally tractable. Figure 5c demonstrates that Quantifier captures 10% more token disagreements than entropy-based routing at 10% selection rate, and Figure 6 shows reasonable calibration properties.

  3. KV Transplant addresses both memory and quality. Beyond eliminating the separate reasoner KV cache (saving 10 GB for Qwen3-32B), Table 4 shows that using target model KV cache improves reasoner accuracy from 53.3% to 61.7% on AIME for Qwen3-32B, indicating that KV quality substantially impacts quantized reasoning performance.

  4. Comprehensive experimental evaluation. The paper compares against multiple baselines (RouteLLM variants, R-Stitch, R2R) across three challenging benchmarks. Table 3 provides detailed memory breakdowns, and ablation studies (Table 2) isolate each component's contribution. The memory efficiency gains are clearly demonstrated (71.60 GB for ORQA vs 74.02 GB for target-only with 32B model).

  5. Strong results. ORQA consistently achieves better accuracy-efficiency tradeoffs than baselines. Notably, on AIME with Qwen3-32B, ORQA reaches 75.6% accuracy (vs 78.3% target-only) at 55.1 TPS (vs 18.8 TPS target-only), with LLM ratio of only 22.9%. The accuracy gap to target-only is much smaller than baselines at comparable LLM ratios.

Critical weaknesses
  1. Limited model coverage. All main experiments use Qwen3 family models (Qwen3-32B, Qwen3-8B as targets; Qwen3-1.7B, Qwen3-0.6B as reasonsers). While Section F.3 includes DeepSeek-R1-0528-Qwen3-8B for weight infusion analysis, the core hybrid reasoning evaluation is limited to one model family. Given that weight infusion exploits BF16 exponent distributions that may vary across models, broader validation would strengthen generality claims.

  2. Calibration data may overlap with reasoning domain. The quantization recipe uses OpenThoughts-114k and Pile-10k for calibration, and the evaluation includes AIME, GPQA, and LiveCodeBench—all reasoning-heavy benchmarks. While the authors state they do not fine-tune on benchmark answers, the calibration data distribution (reasoning traces) closely matches the evaluation distribution. Performance on out-of-distribution tasks (e.g., general knowledge, creative writing) is not evaluated, leaving the broader applicability of ORQA unclear.

  3. Quantifier calibration requires reference model outputs. Section B describes estimating κ and Σ using paired hidden states from reference (target) and quantized models on calibration data. This requires running the full target model during calibration—acceptable offline, but the paper does not discuss how sensitive Quantifier is to calibration set size or distribution. If deployment involves a different task distribution, recalibration may be needed.

  4. KV Transplant's asymmetric access requires careful implementation. The description in Section 4.3 involves tracking target-verified boundaries and partial prefill during target model invocation. The paper does not analyze worst-case scenarios where the reasoner generates long sequences before target intervention, potentially causing significant recomputation overhead. Table 1 shows LLM ratios, but the latency impact of KV cache repair during interventions is not isolated.

  5. Missing wall-clock time and latency analysis. While TPS is reported, actual latency (time-to-first-token, inter-token latency) and end-to-end generation time are not provided. The decompression overhead of weight infusion (Section A) and partial prefill costs during target interventions could impact real-world deployment, especially for latency-sensitive applications.

  6. Comparison with speculative decoding is absent. Recent speculative decoding methods (e.g., EAGLE, Medusa, self-speculative approaches) also accelerate inference without separate models. While the paper cites QuantSpec and SPEQ in related work, it does not empirically compare against them. Since ORQA shares conceptual similarities (using a smaller/quantized model to draft tokens), a direct comparison would clarify its relative advantages.

  7. Limited discussion of failure modes. The paper reports average accuracy and LLM ratios but does not analyze when the hybrid framework fails—e.g., whether there are systematic patterns in tokens where Quantifier's routing decisions are incorrect. Figure 6b shows a local bias around 24-25% predicted probability, but the practical impact of this miscalibration is not quantified in terms of end-to-end accuracy.

Questions for authors
  1. Scalability to larger models: Weight infusion memory savings (Table 8) are roughly consistent percentage-wise across model sizes (18.5-20.8% reduction). However, for models with more layers or wider hidden dimensions, does the delta coverage (99.98%) remain stable? Would extremely large models (e.g., 70B+) require different delta bitwidths?

  2. Reasoner quantization sensitivity: The paper uses 2-bit quantized reasoners. How sensitive is ORQA to the reasoner's bitwidth? Would a 4-bit reasoner (closer to target quality) provide better accuracy at acceptable memory cost, given weight infusion's savings?

  3. Quantifier robustness: How does Quantifier's routing performance degrade if the calibration set is drawn from a different distribution than the test set (e.g., calibrated on reasoning data but deployed for code generation)? Are κ and Σ task-specific?

8 comments
AuthorsOpenReview ↗

Response to Reviewer TQyo (1/7)

We thank the reviewer for the valuable suggestions that greatly helped us improve the manuscript. We are preparing the revised manuscript and will make it available as soon as possible. In the meantime, we provide our response below.

(W1.1) Limited model coverage: the core hybrid reasoning evaluation is limited to one model family.

→\to We also evaluate ORQA on the DeepSeek-R1 model family in Appendix E.3 (Table 6), using DeepSeek-R1-0528-Qwen3-8B as the target model. For convenience, we present the results in Table R19.

Table R19. Experimental results on AIME and GPQA, evaluated using the DeepSeek-R1-Distill-Qwen model family.

Target LLMReasonerMethodAIME Acc. ↑\uparrowAIME TPS ↑\uparrowAIME LLM Ratio ↓\downarrowGPQA Acc. ↑\uparrowGPQA TPS ↑\uparrowGPQA LLM Ratio ↓\downarrow
DeepSeek-R1-0528-Qwen3-8B--Target Only68.33%37.7100.0%60.86%39.6100.0%
DeepSeek-R1-Distill-Qwen-1.5BR-Stitch45.00%48.434.88%54.04%47.444.13%
R2R56.67%62.522.22%52.02%61.226.76%
2-Bit Q.ORQA (Ours)61.67%73.716.22%56.06%74.920.41%

The results show that ORQA outperforms both R2R and R-Stitch in accuracy, throughput (TPS), and LLM ratio on AIME and GPQA. These results demonstrate that ORQA's advantages extend beyond the Qwen3 models evaluated in the main experiments, supporting its applicability across different model families.

(W1.2 & Q1) Broader validation for weight infusion: Does the delta coverage (99.98%) remain stable for extremely large models(e.g., 70B+) with given delta bitwidths?

→\to We extend our weight infusion analysis to Qwen2.5-72B-Instruct and Llama-3.1-70B-Instruct, using publicly available 2-bit quantized checkpoints produced with different quantization methods [1, 2]. The results in Table R20 show that 4-bit delta encoding maintains over 99.9% weight coverage and approximately 23% memory reduction even for models exceeding 70B parameters.

Table R20. Delta coverage and memory savings of weight infusion on 70B+ models.

ModelDelta bitsCoverageSparse outliersCompressed target (GiB)Original target (GiB)Memory ratio ↓\downarrow
Qwen2.5-72B-Instruct [1]397.09%2.91%107.00135.430.79$\times$
499.98%0.02%103.84135.430.77$\times$
Llama-3.1-70B-Instruct [2]396.86%3.14%104.59131.420.80$\times$
499.92%0.08%100.86131.420.77$\times$

First, delta coverage remains stable across model sizes and architectures. With 4-bit delta encoding, Qwen2.5-72B and Llama-3.1-70B achieve 99.98% and 99.92% coverage, respectively, consistent with the 99.98% coverage reported in the original paper. This indicates that the same 4-bit configuration remains effective for substantially larger models without increasing the delta bitwidth.

Second, weight infusion maintains consistent memory savings at larger scales. The compressed target weights require approximately 77% of the original memory for both models, corresponding to a 23% reduction. Moreover, increasing the delta bitwidth from 3 to 4 improves coverage while further reducing total memory consumption by decreasing the storage required for sparse outliers.

These results demonstrate that weight infusion remains effective for models exceeding 70B parameters, without requiring a larger delta bitwidth. 

[1] Cheng, Wenhua, et al. "Optimize weight rounding via signed gradient descent for the quantization of LLMs," EMNLP Findings, 2024. [2] Dadgarnia, Alireza, et al. "GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling," arXiv:2604.18556, 2026.

AuthorsOpenReview ↗

Response to Reviewer TQyo (2/7)

(W2, W3 & Q3) Conduct an evaluation on out-of-distribution tasks in ORQA: Are κ and Σ task-specific?

→\to No, κ\kappa and Σ\Sigma are not task-specific. They are chosen based on quantization error measured on a separate reasoning trace outside the benchmark dataset. We also empirically validate that κ\kappa and Σ\Sigma are not task-specific. 

We evaluate five subjects from the MMLU benchmark: World Religions, Public Relations, Econometrics, Philosophy, and Business Ethics, each covering a domain distinct from our dataset. These subjects represent diverse domains beyond the router’s calibration distribution. We compare ORQA against the full-precision Qwen3-8B target model and R2R (using a 0.6B reasoner) as baselines. We report accuracy, TPS, and LLM ratio measured on two NVIDIA A6000 GPUs.  These results in Table R21 demonstrate that Quant-ifier generalizes beyond its calibration data, delivering substantially high accuracy across unseen domains.

Table R21. Accuracy, TPS, and LLM usage comparison of ORQA and baselines on MMLU.

MethodWorld ReligionsPublic RelationsEconometricsPhilosophyBusiness Ethics
Acc. ↑\uparrowTPS ↑\uparrowLLM Ratio ↓\downarrowAcc. ↑\uparrowTPS ↑\uparrowLLM Ratio ↓\downarrowAcc. ↑\uparrowTPS ↑\uparrowLLM Ratio ↓\downarrowAcc. ↑\uparrowTPS ↑\uparrowLLM Ratio ↓\downarrowAcc. ↑\uparrowTPS ↑\uparrowLLM Ratio ↓\downarrow
Qwen3-8B86.0%41.0100%73.64%41.13100%78.95%40.88100%76.53%41.11100%80.00%40.97100%
R2R81.9%78.228.8%71.82%67.0030.97%74.56%74.0334.74%73.31%83.1530.88%76.00%73.6034.48%
ORQA (Ours)84.8%78.524.5%75.45%73.2929.94%79.82%78.7335.04%74.28%83.6530.00%79.00%74.4834.64%

Regarding distribution shifts, ORQA uses the same calibrated Quant-ifier across different reasoning tasks, including AIME, GPQA, and LiveCodeBench, without task-specific recalibration. The results in Table 1 demonstrate its effectiveness across these benchmarks, suggesting that Quant-ifier generalizes across different task distributions. While substantial distribution shifts may benefit from recalibration, this process is performed offline without additional inference overhead.

(W3) Quant-ifier calibration requires reference model outputs. How sensitive is Quant-ifier to calibration set size?

→\to Quant-ifier requires paired hidden states from the target LLM and quantized reasoner for offline calibration. To evaluate its sensitivity to calibration set size, we conduct an additional experiment on AIME by varying the number of calibration samples from 8 to 192. We used Qwen3-32B and its quantized reasoner for ORQA.

Table R22. Sensitivity of ORQA accuracy and LLM usage to calibration set size on AIME.

Calibration size8163264128192 (full)
Accuracy (%)71.775.075.078.376.775.6
LLM ratio (%)22.622.822.818.621.922.9

The results are shown in Table R22. The results show that Quant-ifier maintains competitive performance even with relatively few calibration samples. With only 16 samples, ORQA achieves 75.0% accuracy and a 22.8% LLM ratio, comparable to 75.6% and 22.9% with the full 192 samples. These results demonstrate limited sensitivity to calibration set size in the evaluated setting.

AuthorsOpenReview ↗

Response to Reviewer TQyo (3/7)

(W4 & W5.2) Discuss worst-case scenarios of long-sequence partial-prefill costs, and the following latency impact of KV cache repair. Analyze the decompression overhead of weight infusion.

→\to All TPS results in Table 1 are measured with a batch size of 1, following the evaluation setting commonly used in token-level hybrid reasoning methods. 

To quantify the runtime overhead of weight infusion, we additionally compare the target model's inference latency using infused weights and the original BF16 weights under identical settings.

Table R6. BF16 and weight-infused target-call latency across call lengths.

Number of Tokens per Target Call123456789256 (Worst Case)
Baseline BF16 Target Call Latency (sec)0.0270.0280.0280.0280.0290.0290.0290.0290.0290.060
Target Call Latency with Weight Infusion (sec)0.0220.0220.0230.0230.0240.0250.0270.0270.0610.096
Relative Speedup ↑\uparrow1.18$\times$1.26$\times$1.22$\times$1.25$\times$1.22$\times$1.15$\times$1.10$\times$1.09$\times$0.48$\times$0.63$\times$
Target Calls, AIME34.6%19.1%12.3%7.6%5.3%3.9%2.9%2.3%1.8%0.001%
Target Calls, GPQA39.8%21.8%13.1%8.1%5.2%3.3%2.1%1.5%1.0%0.020%
Target Calls, LiveCodeBench21.5%18.8%14.9%11.2%8.2%6.0%4.4%3.3%2.4%0.022%

Table R6 provides the comparison results. For the TP=2 target model with an 8,192-token prefix, weight infusion reduces latency for target calls when processing 1–8 tokens as partial-prefill. Across these sizes, it achieves  1.09 to 1.26 speedup over the original BF16 model, despite the additional decompression operations. The median partial-prefill length is 2~3 tokens across the datasets, where weight loading remains a dominant component of execution time. Weight infusion reduces the model-weight memory footprint and associated memory traffic, offsetting the decompression overhead and improving inference latency.

Target calls processing more than 8 tokens are slower with weight infusion: the measured latency is 0.096 seconds for the 256-token worst case (If the reasoner reaches 255 consecutive generation, we route to target LLM for KV cache correction.). However, the benchmark distributions show that calls requiring prefill for more than 9 tokens are relatively very uncommon in ORQA, and 256-token calls are exceptionally rare (0.001%~0.022%). Thus, the slowdown affects a small fraction of target calls in these workloads.

Table R7. Target-call length distribution and workload-weighted latency change with weight infusion.

MetricAIMEGPQALiveCodeBench
Target Calls with Tokens ≤\leq 887.9%94.9%88.2%
Target Calls with Tokens > 812.1%5.1%11.8%
Weighted Sum of Latency Difference δ\delta (sec) ↓\downarrow-0.0003-0.0028-0.0004

Table R7 shows that target calls processing more than 8 tokens are relatively uncommon, accounting for 5.1%~12.1% of calls across the three datasets.  Although weight infusion slows down these longer calls, the latency savings on the more frequent calls of 8 tokens or fewer outweigh that overhead.  This is empirically shown in the final row: we multiply the latency difference between weight infusion and BF16 by the observed frequency of each call length and sum across call lengths. The resulting negative values (of the final row) show an overall latency reduction for all three datasets.

Reviewer_NN6pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ORQA, a hybrid reasoning framework that pairs a target LLM with its 2-bit quantized counterpart instead of loading a separate small model. To avoid extra memory overhead and keep accuracy high, the authors combine four main ideas: weight infusion for compact parameter storage, Quant-ifier for error-aware token routing, KV transplant for cache sharing, and a two-stage calibration recipe (SignRoundV2 plus scale-only distillation). Experiments on AIME, GPQA-Diamond, and LiveCodeBench show good speedups over target-only decoding while keeping accuracy much closer to the full model than standard small-model routers.

Critical strengths
  1. The idea of using a quantized version of the target model rather than a separate small model (like Qwen3-0.6B/1.7B) is interesting and makes sense for hybrid reasoning. Sharing the parameter storage and the KV cache pool directly attacks the memory bottleneck of prior methods at long context lengths.
  2. The proposed Quant-ifier goes beyond output entropy by leveraging hidden-state quantization errors to predict token disagreements. Its effectiveness under tight token-selection budgets, along with the calibration analysis and discussion of its theoretical limitations, strengthens the method's empirical validity.
  3. The experiments cover three challenging reasoning benchmarks across math, science QA, and coding, tested on both 8B and 32B scales, demonstrating the effectiveness of the proposed method.
Critical weaknesses
  1. The weight memory numbers do not match between Table 3 and Table 8. Table 3 lists the compressed target weights for Qwen3-32B and Qwen3-8B as 46.65 GB and 9.91 GB, whereas Table 8 reports 48.34 GiB and 12.43 GiB for the exact same 4-bit \delta setting. Beyond mixing GB and GiB, it is unclear where this gap comes from—perhaps shared BF16 layers or quantization scales are counted under the reasoner column in Table 3. It would be better for the authors to provide more explanations and clarify if Table 3 shows per-GPU or total memory across both GPUs?
  2. In Section 4.2, Eqs. (9)–(13) only model whether the reasoner's top-2 tokens flip their order. In practice, the target model might prefer a third token outside the reasoner's top-2, especially when the 2-bit model drifts. Moreover, Eq. (14) accumulates crossing probabilities assuming step-wise independence, while Appendix F.1 only checks single-step reliability on fixed prefixes. Given that decoding uses temperature T=0.6T=0.6 and top-p=0.95 (Appendix C), how often do disagreements fall outside the top-2, and how does qkq_k behave on long unverified suffixes?
  3. Under KV transplant (Section 4.3), every target intervention runs a partial prefill over the entire unverified reasoner suffix C>ukC>u_k, but "LLM Ratio" only counts newly generated target tokens. If the reasoner writes 15 tokens and the target generates 1, the LLM ratio is 1/16 even though the target model just processed 16 tokens. Thus, a low LLM ratio hides the prefill refresh cost. Could the authors report end-to-end latency per problem, average output length, and total tokens processed by the target model? A latency breakdown (reasoner decoding, target prefill, decompression, routing) across batch sizes and context lengths—especially around the M=8M=8 kernel switch in Appendix A—would also help.
  4. In Table 2, adding Quant-ifier after KV transplant and calibration improves accuracy, but the LLM ratio also changes (from 11.4% to 8.2% on AIME, and 17.3% to 13.8% on GPQA). To fairly compare routers, it would be better for the authors to keep the calibrated 2-bit reasoner and KV transplant fixed, and compare Quant-ifier against entropy and a top-2 logit margin baseline at matched target token budgets. Also, how do standalone 2-bit (with your calibration) and standard 4-bit (e.g., AWQ/GPTQ) models perform on these benchmarks without hybrid routing? Finally, please either compare against a quantized speculative baseline (like QSpec or QuantSpec [1, 2]) or explain why it is not applicable here.
  5. The authors provide component‑level ablations (Table 2) but lacks sensitivity analysis for key newly‑introduced hyperparameters (e.g., covariance rank rr, threshold q⋆q_\star). It remains unclear whether the strong performance originates from core algorithmic gains or heavy manual hyper‑parameter tuning. It would be better for the authors to provide more sensitivity analysis.
  6. ORQA adopts weight‑only 2‑bit quantization (W2A16), leaving activations and KV cache in BF16 precision. While this design avoids extra activation‑quantization noise for the Quant‑ifier module, it limits runtime memory savings to only model weights. It wouled be better for the authors to add a discussion about the challenges, potential trade‑offs, and open issues when extending ORQA towards activation‑quantized settings in the paper.

Minor issues

  1. In Appendix E.3, the authors claim that the target model is DeepSeek-R1-0528-Qwen3-8B, but Table 6 shows that the target model is DeepSeek-R1-Distill-Qwen3 32B.
  2. In Table 1, ORQA drops from 54.5% (target-only) to 46.7%, so claiming that it preserves "near-target-only quality" across the board is an overstatement.

[1] QSpec: Speculative Decoding with Complementary Quantization Schemes. EMNLP, 2025.

[2] QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache. ICML, 2025.

Questions for authors

see weakness

8 comments
AuthorsOpenReview ↗

Response to Reviewer NN6p (1/7)

We thank the reviewer for the helpful comments that helped us enrich our findings. A revised manuscript incorporating these valuable suggestions will follow shortly. For now, we share our responses below. 

(W1) The weight memory numbers do not match between Table 3 and Table 8. Clarify if Table 3 shows per-GPU or total memory across both GPUs.

→\to We deeply thank the reviewer for helping us revise the manuscript. Table 3 shows the total memory across both GPUs. We have revised Table 3 to correct the inconsistent categorization of shared components that ORQA enables both the target and reasoner to use (e.g., quantization scales, embedding tables, and head layers). We added a separate column to show the size of those shared components. Below is the revised Table 3. 

Table R8. Memory breakdown of ORQA and baselines for 32B and 8B targets (GiB).

Target LLMMethodTarget WeightsTarget KVReasoner WeightsReasoner KVShared ComponentsOthersTotal ↓\downarrow
32BStandalone61.0210.000.000.000.003.0074.02
R-Stitch61.0210.003.204.380.006.3584.95
R2R61.0210.003.204.380.007.6386.24
Naive61.0210.0011.9810.000.006.1499.15
ORQA (Ours)43.6810.007.270.004.725.8771.53
8BStandalone15.265.630.000.000.001.7222.60
R-Stitch15.265.631.114.380.004.6631.03
R2R15.265.631.114.380.005.4031.77
Naive15.265.634.345.630.003.4434.29
ORQA (Ours)9.735.631.620.002.721.6221.32

The large inconsistency between Tables 3 and 8 is resolved. We additionally note that the minimal numerical difference arises from the runtime metadata (e.g., runtime pointers for weights) added at inference. We thank the reviewer again for bringing this matter to our attention.

(W2.1) The router only models whether the reasoner’s top-2 tokens flip their order. How often do disagreements fall outside the top-2?

→\to We analyze token disagreements between the quantized reasoner and target LLM, including cases where the target's top-1 token falls outside the reasoner's top-2 candidates. The results in Table R9 show that 24.3% and 26.4% of disagreements occur outside the top-2 candidates for Qwen3-8B and Qwen3-32B, respectively, while Quant-ifier captures approximately 80% of these disagreements.

Table R9. Token disagreement distribution and Quant-ifier detection rates within and beyond the reasoner’s top-2 candidates.

Target LLMTop-2 disagreements (%)Captured (%)Outside-top-2 disagreements (%)Captured (%)
Qwen3-8B75.786.824.381.6
Qwen3-32B73.684.326.479.7

The disagreement proportions are measured over all token disagreements, while the capture rates indicate the percentage successfully identified by Quant-ifier within each category.

Although Eqs. (9)–(13) approximate the disagreement probability using the reasoner's top-2 logit margin, Quant-ifier captures 81.6% and 79.7% of outside-top-2 disagreements for Qwen3-8B and Qwen3-32B, respectively. This demonstrates that the top-2 approximation remains useful for identifying disagreements beyond the two highest-ranked candidates.

Furthermore, the top-2 approximation only affects disagreement estimation, not token selection. When the target LLM is invoked, it uses its full output distribution without restricting token selection to the reasoner's top-2 candidates.

AuthorsOpenReview ↗

Response to Reviewer NN6p (2/7)

(W2.2) How does qkq_k behave on long unverified suffixes?

→\to We conducted additional experiments to examine the reliability of qkq_k over long unverified suffixes and during practical hybrid reasoning. The results show that qkq_k follows the trend of empirical disagreement probability and remains closely aligned with empirical measurements during hybrid reasoning.

1. Accumulated probability over long suffixes. We evaluate qkq_k on 192 reasoning traces for Qwen3-8B and Qwen3-32B, sampled with T=0.6 and top-p=0.95. Since long unverified suffixes rarely occur in practical inference, we artificially increase the routing threshold to q∗=0.99q^*=0.99 to examine their behavior. We measure the empirical probability of at least one token disagreement within each suffix in Table R10.

Table R10. Estimated and empirical disagreement probabilities over long unverified suffixes.

Suffix depth8B qkq_k / Empirical (%)32B qkq_k / Empirical (%)
111.7 / 11.313.1 / 11.9
11–2072.6 / 69.275.8 / 70.4
51–10091.6 / 86.992.1 / 83.7
101–25594.1 / 86.893.5 / 81.7

qkq_k captures the increasing disagreement risk as suffix length grows. Although the independence approximation introduces estimation errors, qkq_k follows a similar trend to the empirical probability, with larger deviations at extreme suffix lengths.

2. Unverified suffix lengths in practical inference. We measure the average number of unverified reasoner tokens per target intervention under different routing budgets. The results are shown in Table R11.

Table R11. LLM usage and average unverified suffix length across routing budgets.

Routing budgetAIME: LLM ratio / Avg. lengthGPQA: LLM ratio / Avg. length
~5%3.9% / 23.54.8% / 17.8
~10%8.5% / 10.510.7% / 8.2
~20%16.2% / 5.021.5% / 3.6

Under practical routing configurations, unverified suffixes are substantially shorter. Across AIME, GPQA, and LiveCodeBench, 87.9%, 94.9%, and 88.2% of target interventions process at most 8 partial-prefill slots, respectively. Interventions reaching 256 slots account for at most 0.02% of calls.

3. Reliability during hybrid reasoning. We evaluate qkq_k during actual Qwen3-8B hybrid decoding, where reasoner-generated tokens affect subsequent predictions.

Table R12. Estimated and empirical disagreement probabilities during hybrid reasoning.

Suffix depthqkq_k (%)Empirical (%)Difference (pp)
110.49.6−0.8
3–523.222.1−1.1
11–2030.129.6−0.5
21–5033.031.9−1.1
51–10034.736.8+2.2
101–25531.530.0−1.5

In Table R12, qkq_k closely matches the empirical disagreement probability during hybrid reasoning, with deviations within 2.2 percentage points across the reported depth ranges.

Overall, these results demonstrate that qkq_k captures accumulated disagreement risk over long suffixes and provides reliable probability estimates under practical hybrid reasoning.

AuthorsOpenReview ↗

Response to Reviewer NN6p (3/7)

(W3.1) Clarification of LLM Ratio and Partial-Prefill Overhead Through End-to-End Latency Analysis

→\to LLM ratio measures the fraction of tokens generated by the target LLM, following the convention of prior hybrid reasoning methods such as R-Stitch and R2R. As the reviewer noted, this metric does not include tokens processed during KV cache repair. In ORQA, the total number of target-processed tokens is approximately equal to the output length, including both target-generated tokens and tokens processed during partial prefill.

However, the number of processed tokens does not directly represent inference latency. Unlike autoregressive decoding, partial prefill processes multiple tokens in parallel and achieves higher token-processing throughput. Therefore, repairing the reasoner's KV cache is substantially more efficient than generating the same number of tokens autoregressively.

To quantify the actual runtime overhead, we provide additional measurements of end-to-end generation latency, average output length, total target-processed tokens, and partial-prefill latency. Table R13 summarizes statistics obtained from the Qwen3-32B ORQA results in Table 1.

Table R13. End-to-end latency and token counts of ORQA and baselines.

MetricMethodAIMEGPQALiveCodeBench
Average End-to-End Latency (sec) ↓\downarrowR-Stitch376.3165.4427.6
R2R382.6153.8368.3
ORQA (Ours)273.4114.1272.8
Average Output Length (# Tokens)R-Stitch17126.35983.915052.8
R2R16585.35553.213923.0
ORQA (Ours)14814.35775.714736.2
Average Target-generated Tokens ↓\downarrowR-Stitch3960.82128.55247.1
R2R3731.21769.04046.8
ORQA (Ours)3401.21843.03199.5

ORQA achieves the lowest average end-to-end latency, leading to the best TPS values across the datasets. Output length and target-generated tokens generally follow a similar trend, except for a single exception on GPQA: R2R generates fewwer tokens with SLM than ORQA but exhibits worse latency and TPS. We analyze the partial-prefill cost of target calls below to help explain this result. We also examine the kernel-switching boundary at M=8M = 8 described in Appendix A.  

Table R6. BF16 vs weight-infused target calls: comparison.  

Number of Tokens per Target Call123456789256 (Worst Case)
Baseline BF16 Target Call Latency (sec)0.0270.0280.0280.0280.0290.0290.0290.0290.0290.060
Target Call Latency with Weight Infusion (sec)0.0220.0220.0230.0230.0240.0250.0270.0270.0610.096
Relative Speedup ↑\uparrow1.18$\times$1.26$\times$1.22$\times$1.25$\times$1.22$\times$1.15$\times$1.10$\times$1.09$\times$0.48$\times$0.63$\times$
Target Calls, AIME34.6%19.1%12.3%7.6%5.3%3.9%2.9%2.3%1.8%0.001%
Target Calls, GPQA39.8%21.8%13.1%8.1%5.2%3.3%2.1%1.5%1.0%0.020%
Target Calls, LiveCodeBench21.5%18.8%14.9%11.2%8.2%6.0%4.4%3.3%2.4%0.022%

Table R6 show the comparison results. For the TP=2 target model with an 8,192-token prefix, we observe that weight infusion reduces target call latency when processing 1–8 tokens as partial-prefill. Across these sizes, it achieves 1.09 to 1.26 speedup over the original BF16 model, despite the additional decompression operations. Weight infusion reduces the model-weight memory footprint and associated memory traffic, offsetting the decompression overhead and improving inference latency.

Target calls processing more than 8 tokens are slower with weight infusion: the measured latency is 0.096 seconds for the 256-token worst case. (If the reasoner reaches 255 consecutive generation, we route to target LLM for KV cache correction.) However, the benchmark distributions show that calls requiring prefill for more than 9 tokens are relatively very uncommon in ORQA, and 256-token calls are exceptionally rare (0.001%~0.022%). Thus, the slowdown affects a small fraction of target calls in these workloads. (The response for this comment is continued right below)

Reviewer_XcmPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ORQA, a token-level hybrid reasoning framework. qrora uses a 2-bit quantized copy of the target LLM as the reasoner, so no separate small model is needed. Weight infusion stores the BF16 target weights as low-bit exponent offsets relative to the quantization scales, which allows bit-exact recovery with less memory. KV transplant lets the two models share one KV cache. Quant-ifier routes tokens to the target model based on the estimated probability that quantization error flips the top-1 token. A reasoning-oriented calibration recipe improves the reasoner. On Qwen3-8B/32B with AIME, GPQA-Diamond and LiveCodeBench, ORQA stays close to target-only accuracy at up to 2.9× its throughput (about 1.7-1.9× on Qwen3-8B), compares favorably with R2R and R-Stitch, especially on AIME, and uses slightly less peak memory than target-only inference.

Critical strengths
  1. QRORA uses the target's own quantized copy as the reasoner, with a shared KV pool, which is a simple and effective idea, and the evidence supports the claim. Table 4 is the most informative result in the paper. A 2-bit Qwen3-8B that reads the BF16 target's KV cache improves from 31.7% to 51.7% on AIME (32B: 53.3% → 61.7%). So much of the gap between the quantized and full-precision models comes from degraded cached states, not only from degraded next-token computation. This finding motivates KV transplant directly. It also explains why a quantized counterpart works much better as a reasoner than a 0.6B/1.7B SLM. Under a low target budget (Table 7), ORQA keeps 60.0% on AIME at a 3.89% LLM ratio, while R2R gets 30.0% at 7.30%. This is the regime that matters most for offloading.

  2. Quant-ifier is a principled routing signal designed for this setting, and it is well calibrated. The derivation in Eq. (9)–(13) ties the routing decision to the event that the target reverses the reasoner's top-2 order. The low-rank structure of the error covariance (8 of 4,096 directions explain 44.6% of the variance, Fig. 5b) justifies the O(Dr) approximation. The reliability curves in Fig. 6 (ECE of about 1 pp on held-out reasoning and AIME traces, without refitting) show that the score behaves as a probability. Fig. 5c shows it captures about 13 pp more disagreements than entropy at a 10% routing budget.

Critical weaknesses
  1. The memory comparison could be fairer. Much of the saving appears to come from losslessly compressing the BF16 target (0.79× in Table 8), which could also be applied to the baselines. For reference, DFloat11 (cited) reaches about 0.70×. By my rough estimate, a DFloat11 target plus a separate 2-bit reasoner would use slightly less weight memory than ORQA. A comparison with compressed-target baselines, and a short discussion of how weight infusion differs from DFloat11/ZipServ and SPEQ (Zhao et al., 2025b), would make the memory claim much more convincing.

  2. The advantage over prior hybrid methods is not consistent across settings. ORQA is clearly better on AIME and in the low-budget regime. Elsewhere the picture is mixed: R2R is slightly more accurate on 32B/GPQA, and R-Stitch is faster on 8B/GPQA and 8B/LiveCodeBench. Since AIME has only 60 problems, reporting variance across runs would help readers judge these gaps. A quantized-draft speculative decoding baseline (e.g., QSpec or QuantSpec) would also be a useful reference point, because it is lossless.

Questions for authors
  • Could you compare memory against baselines whose target model is also losslessly compressed (e.g., with DFloat11)?
  • How many runs or samples per problem are behind the numbers in Table 1? Could you add standard deviations?
  • What batch size is used for TPS? How much slower is the target model with infused weights than with plain BF16 weights?
  • In Table 2, the naive baseline gets 13.3% on AIME, lower than the 2-bit model alone (31.7%, Table 4). Could you explain this?
  • the model names in Table 6 and §E.3 differ; the memory numbers in Tables 2, 3 and 8 do not quite match (GB vs GiB?); the AI Use Statement still contains template brackets.
6 comments
AuthorsOpenReview ↗

Response to Reviewer XcmP (1/5)

We thank the reviewer for the valuable suggestions and comments that helped enhance our manuscript. We have conducted experiments to address the comments, and will soon update the revised manuscript that includes the constructive feedback. We first share our response below. 

(W1&Q1) Further memory comparison of weight infusion.

→\to We conducted two additional experiments to highlight memory saving benefits of weight infusion, and provide discussion to clarify its differences against prior works.

1. Applying weight infusion to baselines to observe memory benefits: We applied weight infusion to token-level routing baselines, R2R and R-Stitch, to further evaluate its memory benefits. Table R1 reports results measured on PRO6000 GPUs, using Qwen3-32B as the target and Qwen3-1.7B as the reasoner on AIME.

Table R1. Memory and latency benefits of weight infusion.

MethodPeak Memory (GiB) ↓\downarrowRelative Memory Cost ↓\downarrowEnd-to-End Latency (sec) ↓\downarrowLLM Ratio ↓\downarrow
R2R86.241.00$\times$382.620.9%
R2R + Weight Infusion76.270.88$\times$330.720.3%
R-Stitch84.951.00$\times$376.323.7%
R-Stitch + Weight Infusion74.770.88$\times$372.822.9%

Weight infusion reduces memory consumption by approximately 10 GiB without increasing latency. This is enabled by tailored kernels for minimal decompression cost, which we explain in detail in Appendix A. We confirm that weight infusion can be a lossless compression technique broadly applicable to other LLM inference frameworks when LLM quantization scales are available.

2. Comparison against other lossless target-compression methods: We compared weight infusion with other lossless compression methods for LLMs on GPUs, including DFloat11 [1] and ZipServ [2], in terms of memory consumption and latency. Table R2 reports results measured on PRO6000 GPUs, using Qwen3-32B target and the quantized reasoner on AIME.

Table R2. Memory and latency comparison of lossless target-compression methods.

MethodPeak Memory (GiB) ↓\downarrowRelative Memory Cost ↓\downarrowEnd-to-End Latency (sec) ↓\downarrowLatency Speedup ↑\uparrowLLM Ratio ↓\downarrow
Qwen3-32B74.01.00$\times$748.31.00$\times$100.0%
ORQA using DFloat1169.40.94$\times$441.01.70$\times$22.5%
ORQA using ZipServ70.40.95$\times$362.82.06$\times$23.1%
ORQA (original, with weight infusion)71.50.97$\times$273.42.74$\times$22.9%

Although DFloat11 and ZipServ reduce peak memory as much as or more than weight infusion, their decompression overhead adds about 90 seconds of latency, limiting their suitability for hybrid reasoning. DFloat11’s Huffman coding makes decompression difficult to optimize, and ZipServ focuses on accelerating decoding in standalone LLM inference, performing less efficiently on repeated short partial-prefill calls by the target. Weight infusion, by contrast, uses a GPU-friendly decompression path designed for these frequent partial-prefill workloads (Appendix A).

For SPEQ [3], we do not directly compare with weight infusion, as it targets a vastly different deployment setting of a dedicated accelerator. SPEQ co-designs its weight transformation with specialized hardware support to address its memory and latency overhead. We therefore leave comparison with SPEQ outside the scope of our work, as we focus on commodity GPU deployment. We thank the reviewer for the helpful suggestion to further evaluate weight infusion.

  • [1] Zhang, Tianyi, et al. "70% size, 100% accuracy: Lossless LLM compression for efficient GPU inference via dynamic-length float (DFloat11).", NeurIPS, 2025.
  • [2] Fan, Ruibo, et al. "Zipserv: Fast and memory-efficient llm inference with hardware-aware lossless compression.", ASPLOS, 2026.
  • [3] Zhao, Yushu, et al. "From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing.", arXiv:2510.18525, 2025.
AuthorsOpenReview ↗

Response to Reviewer XcmP (2/5)

(W2.1) The advantage over prior hybrid methods is not consistent across settings, and seems rather mixed.

→\to We would like to clarify that ORQA demonstrates consistent improvements across most settings, with substantial average performance gains over prior hybrid methods. We provide the following table comparing accuracy, throughput, and LLM ratio against R-Stitch and R2R.

Table R3. Accuracy, TPS, and LLM usage differences between ORQA and prior hybrid methods.

TargetBenchmarkBaselineΔ\Delta Acc. (pp) ↑\uparrowΔ\Delta TPS (%) ↑\uparrowΔ\Delta LLM Ratio (pp) ↓\downarrow
8BAIMEORQA vs R-Stitch+16.6+2.1−5.8
ORQA vs R2R+5.0+7.4−7.6
8BGPQAORQA vs R-Stitch+13.6−8.6−1.7
ORQA vs R2R+6.5+0.4−4.6
8BLiveCodeBenchORQA vs R-Stitch+9.6−11.3+2.9
ORQA vs R2R+2.9−3.30.0
8B Avg.AllORQA vs R-Stitch+13.3−5.9−1.5
8B Avg.AllORQA vs R2R+4.8+1.5−4.1
32BAIMEORQA vs R-Stitch+8.9+19.8−0.8
ORQA vs R2R+5.6+2.8+2.0
32BGPQAORQA vs R-Stitch+4.8+35.6+3.0
ORQA vs R2R−0.7+41.5+2.3
32BLiveCodeBenchORQA vs R-Stitch+6.2+62.7−13.6
ORQA vs R2R+2.4+36.1−6.9
32B Avg.AllORQA vs R-Stitch+6.6+39.4−3.8
32B Avg.AllORQA vs R2R+2.4+26.8−0.9

First, ORQA consistently outperforms prior hybrid methods across most settings. It achieves higher accuracy in 11 of 12 comparisons and higher throughput in 9 of 12 comparisons.

Second, ORQA achieves substantial average gains despite occasional regressions. Across both model sizes and all benchmarks, ORQA improves accuracy by 10.0 and 3.6 percentage points over R-Stitch and R2R, respectively. It also increases throughput by 16.7% and 14.2%, while reducing the LLM ratio by 2.7 and 2.5 percentage points, respectively. These average improvements demonstrate that ORQA's gains outweigh its occasional performance regressions.

Furthermore, Figure 4 illustrates the accuracy–throughput and accuracy–LLM ratio trade-offs across different routing thresholds. ORQA generally achieves higher accuracy at comparable throughput or lower target-LLM usage at comparable accuracy, further supporting its favorable accuracy-efficiency trade-off across different operating points.

(W2.2 & Q2) Report variance across runs, inference settings for Table 1.

→\to We report additional experimental results with standard errors across three runs with different seeds.

Table R4. Accuracy with standard errors.

Method32B·AIME32B·GPQA32B·LiveCodeBench8B·AIME8B·GPQA8B·LiveCodeBench
Target LLM only78.3% ±\pm 1.0%59.6% ±\pm 0.6%64.3% ±\pm 0.8%66.7% ±\pm 3.3%57.4% ±\pm 0.6%54.5% ±\pm 1.5%
BERT43.3% ±\pm 5.0%41.4% ±\pm 1.1%44.8% ±\pm 1.5%26.7% ±\pm 1.7%36.2% ±\pm 1.6%28.6% ±\pm 2.0%
Causal LLM70.0% ±\pm 1.0%37.4% ±\pm 1.3%38.6% ±\pm 1.9%58.3% ±\pm 3.3%32.7% ±\pm 1.1%20.7% ±\pm 2.4%
MF41.7% ±\pm 4.4%38.4% ±\pm 1.8%38.1% ±\pm 1.8%21.1% ±\pm 2.0%35.4% ±\pm 1.3%22.9% ±\pm 0.7%
SW_RANKING48.3% ±\pm 2.9%42.4% ±\pm 1.8%42.4% ±\pm 2.4%33.3% ±\pm 0.0%34.8% ±\pm 2.8%27.6% ±\pm 1.3%
R-Stitch66.7% ±\pm 4.4%54.6% ±\pm 0.8%56.2% ±\pm 2.1%46.7% ±\pm 1.0%39.9% ±\pm 2.0%37.1% ±\pm 1.7%
R2R70.0% ±\pm 2.9%60.1% ±\pm 2.8%60.0% ±\pm 1.4%58.3% ±\pm 2.5%47.0% ±\pm 3.7%43.8% ±\pm 3.6%
ORQA (Ours)75.6% ±\pm 3.9%59.4% ±\pm 2.6%62.4% ±\pm 2.7%63.3% ±\pm 3.5%53.5% ±\pm 2.0%46.7% ±\pm 1.2%

The results show that ORQA's performance improvements are substantial relative to the observed standard errors, supporting the robustness of its gains.

AuthorsOpenReview ↗

Response to Reviewer XcmP (3/5)

(W2.3) Compare ORQA with quantized-draft speculative decoding baselines. (e.g., QSpec or QuantSpec)

→\to We thank the reviewer for the constructive suggestion.

We compare ORQA with QSpec and QuantSpec, which similarly leverage quantized models for efficient draft generation. However, their designs differ from ORQA in several important aspects.

QSpec uses a 4-bit weight-only quantized target model for verification and a weight-activation quantized model for drafting, whereas ORQA retains the original BF16 target model. QSpec’s overall accuracy is therefore constrained by the quantized target model. QuantSpec employs a 4-bit weight-quantized drafter with a full-precision target model. It also introduces hierarchical 4-bit and 8-bit KV cache representations for efficient speculative decoding. In contrast, ORQA employs a 2-bit quantized reasoner and eliminates separate reasoner-weight storage through weight infusion. KV transplant further enables both models to use a shared KV cache pool without additional reasoner KV cache memory.

Beyond these architectural differences, we provide an experimental comparison on AIME and GPQA.

Table R5. Accuracy, TPS, and memory comparison of ORQA with QSpec and QuantSpec.

MethodAIME Acc. ↑\uparrowAIME TPS ↑\uparrowAIME Relative TPS ↑\uparrowGPQA Acc. ↑\uparrowGPQA TPS ↑\uparrowGPQA Relative TPS ↑\uparrowMemory (GiB) ↓\downarrow
Qwen3-32B78.3%18.81.0$\times$59.6%19.21.0$\times$74.02
QSpec66.7%29.91.6$\times$59.6%30.31.6$\times$36.47
QuantSpec80.0%26.61.4$\times$62.1%27.91.5$\times$86.76
ORQA (Ours)75.6%55.12.9$\times$59.4%49.12.6$\times$71.53

The results in Table R5 show that ORQA achieves higher throughput than QSpec and QuantSpec on both long-reasoning benchmarks. Compared to target-only inference, ORQA achieves 2.9$\times$ and 2.6$\times$ speedups on AIME and GPQA, respectively, outperforming QSpec (1.6$\times$ on both) and QuantSpec (1.4$\times$ and 1.5$\times$). Moreover, ORQA requires the lowest memory among methods preserving the full-precision target model, at 71.53 GiB, compared to 86.76 GiB for QuantSpec. Although QuantSpec achieves higher accuracy, ORQA provides greater throughput with lower memory consumption. These results highlight the distinct advantages of ORQA's hybrid reasoning approach over speculative decoding methods for long-reasoning tasks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to the Reviewers

We sincerely appreciate all reviewers and AC for their time and efforts throughout the review process. We are particularly pleased that the reviewers recognize our well-motivated, interesting, innovative, and well-designed approach (XcmP, NN6p, TQyo) to addressing practical deployment challenges (NN6p, TQyo, Htci). The reviewers also highlight the supporting analysis and theoretical discussion (NN6p), empirical evidence (XcmP), and comprehensive experimental evaluation (TQyo). The reviewers also recognize the promising empirical results and strong experimental validation of ORQA.

We have conducted additional experiments and analyses to address the reviewers' concerns and questions. The key points of our responses are summarized below.

  • Further experimental validation: We demonstrate ORQA's favorable accuracy-efficiency trade-off against quantization-based speculative decoding methods (QSpec and QuantSpec) and standalone 2–8-bit quantized LLMs. We also present the DeepSeek-R1 evaluation to demonstrate ORQA's effectiveness across different model families. Additional out-of-distribution evaluations support ORQA's generalization to different task distributions.
  • End-to-end latency comparison: We demonstrate ORQA's end-to-end generation latency improvements over the baselines, complementing the throughput (TPS) results.
  • Component-level latency and memory breakdown: We provide a component-level latency and memory breakdown and show that weight infusion reduces memory consumption while improving target-call latency for the most common short partial-prefill calls.
  • Further validation for weight infusion: We compare ORQA's memory consumption against lossless weight-compression methods (DFloat11 and ZipServ) and baseline methods augmented with weight infusion. We further show that weight infusion remains effective for models exceeding 70B parameters, with similar delta coverage and target-weight storage savings across the tested models.
  • Further analysis of Quant-ifier router: We demonstrate that Quant-ifier outperforms entropy, top-2 logit margin, and fixed-interval routing at comparable target-LLM usage. We further analyze token disagreements outside the reasoner's top-2 candidates and the behavior of the accumulated routing score qkq_k.
  • Additional sensitivity study: We evaluate ORQA's sensitivity to covariance rank, routing thresholds, calibration set size, and reasoner bitwidth, demonstrating robust performance across these configurations.
  • Further discussion: We discuss extensions to activation-quantized inference, analyze ORQA's failure modes and routing behavior, and discuss the transferability of the same calibrated κ\kappa and Σ\Sigma across the evaluated tasks.
  • Clarification of experimental results: We clarify ORQA's consistency across experimental settings and report standard errors across multiple runs. We also correct ambiguities and provide additional details on experimental configurations and reported results.

We sincerely thank the reviewers for their constructive comments, which have helped strengthen the manuscript and provide richer context for our findings. Importantly, these revisions strengthen the experimental validation and analysis without changing the core components of ORQA. 

We will soon update the revised manuscript accordingly, taking into account all the suggestions and feedback provided by the reviewers. We first share the responses below for the reviewers.  We welcome further discussion with the reviewers and look forward to addressing any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.