acceptodds
Under review as a conference paper at ICLR 2027

Distill Globally, Train Locally: Error-Localized Recovery for 4-bit LLMs

Abstract

Low-bit quantization enables efficient large language model (LLM) inference, but maintaining accuracy with 4-bit weights and activations (W4A4) remains challenging. Local calibration in post-training quantization (PTQ) lacks direct supervision of final predictions, limiting further accuracy recovery. Full-model quantization-aware distillation (QAD) can recover near-BF16 accuracy through global supervision, but requires costly model-wide optimization. To preserve the accuracy of full-model QAD while minimizing training cost, we propose LoDiQ, a lightweight error-localized distillation framework with three components: (1) propagated-error localization identifies sensitive input channels; (2) quantized residual compensation strengthens recovery within expanded low-bit GEMMs, with Gradient-Guided Residual Compensation Allocation (GRCA) supporting lower-budget configurations; and (3) localized distillation trains only the selected corrective weights using compact teacher targets, avoiding full-vocabulary teacher-logit storage. Experiments across the Qwen and Llama model families span language modeling, zero-shot commonsense reasoning, broad-domain knowledge, and mathematical reasoning. LoDiQ surpasses state-of-the-art PTQ baselines in downstream accuracy under both MXFP4 and NVFP4 W4A4 quantization. On a matched reasoning evaluation, LoDiQ outperforms full-model QAD using only 1/100 of the distillation tokens and 4.67% of its trainable parameters, substantially reducing optimization workload and trainable-state memory. Inference benchmarks on RTX 5090 show up to 3.20× and 3.14× time-to-first-token (TTFT) speedups over BF16 in MXFP4 and NVFP4, respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Quantization, Quantization-Aware Distillation, Parameter-Efficient Training, Efficient Inference, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_NyUUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose a QAD framework for MXFP4 and NVFP4, the main idea is to keep majority of the model weight fixed while updating a only a small amount of important parameters for NVFP4/MXFP4. They also propose trainable residual compensation, that is an extra trainable weight set for important parameters of specific importants layers, after selecting those layers using a gradient-based heuristic under specific model budget.

Critical strengths
  1. The framework is practical, target a practical setting in quantization, for NVFP4/MXFP4

  2. Good empirical result. The proposed framework outperform recent PTQ baseline across Llama/Qwen models.

  3. Training efficiency. Updates only a small subset of parameters, significantly reduces trainable-state memory.

Critical weaknesses
  1. Limited technical novelty. Parameter-efficient QAT and selective weight training are not new ideas. ARCQuant already use residual compensation, while the metrics proposed for channel selection and allocation are quite simple.

  2. Unfair comparison. The paper propose a QAD framework, but the compared target are mostly PTQ method. PTQ often costs a lot less than QAD. And based on the ablation table, most of the improvement come from QAD. Would be better if we have some parameter-efficient QAD baseline

  3. Does the proposed channel selection metric work better than simple metrics, like random, or gradient-based selection?

  4. Why GRCA use NLL gradient but not KL despite the framework is trained with KL?

Questions for authors

Please answer Question in Weakness

2 comments

Thank you for recognizing LoDiQ’s practical relevance, accuracy improvements, and reduced trainable-state memory. We have strengthened the comparisons and clarified the contributions below.

1. Contribution beyond selective training and residual compensation

LoDiQ combines error-localized model-level distillation with explicitly controlled low-bit recovery capacity. Propagated-error analysis identifies the weight columns to update, independently trainable residual weights provide additional recovery capacity, and GRCA allocates that capacity under a global parameter budget.

The distinction from ARCQuant goes beyond making residual compensation trainable. ARCQuant uses an activation-magnitude threshold to determine a layer-dependent number of residual channels and duplicates their quantized weights. LoDiQ fixes the correction width at K=128K=128 and independently optimizes the primary and residual weight blocks through model-level distillation. Each compensated layer expands its GEMM reduction dimension by only 128, making additional storage and computation predictable and limiting expansion where threshold-based compensation would select more channels. GRCA further controls which layers receive this expansion.

The learned correction remains in the target FP4 format and is packed into expanded low-bit GEMMs without a separate high-precision adapter branch. Our contribution concerns this combination of localized learning, budgeted recovery capacity, and uniform W4A4 deployment, not selective updates or residual compensation alone.

2. Training-based comparisons

We agree that PTQ comparisons alone are insufficient. Tables 3 and 19 now compare FP4-adapted QLoRA and EfficientQAT on Llama-3.2-3B, Llama-3-8B, and Qwen3-8B under both formats. For Qwen3-8B MXFP4:

MethodWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
QLoRA (adapter)9.46069.29
QLoRA (merged)9.72268.39
EfficientQAT11.47567.54
LoDiQ-Max9.73069.75

Retained QLoRA adapters add high-precision branch computation; merging removes the branch but incurs requantization error. EfficientQAT is adapted from integer weight quantization to FP4 W4A4. MXFP4 scale optimization exhibits instability in our experiments, motivating validation-based checkpoint selection. Its training cost is substantial: Qwen3-8B MXFP4 requires 11.08 single-H20 hours, compared with 0.93 for LoDiQ. Section 4.3 and Appendix A.2 specify the configurations and timing scope.

These adapted baselines use next-token NLL and therefore constitute training-based comparisons, not parameter-efficient QAD baselines. We also extended full-parameter QAD to the original 800M-token scale using official code and data. On AceReason-Nemotron-1.1-7B, QAD obtains 61.89/66.67 on AIME25/26 using 495.13 GPU-hours; LoDiQ-Max obtains 62.33/67.67 with 1M tokens and 1.03 GPU-hours. Tables 4(b) and 9 report costs and accuracy at these respective budgets.

3. Comparisons with simpler reordering strategies

Table 7(a) compares random, activation-amax, and error-localized reordering followed by distillation. Each trains the last K=128K=128 weight columns after reordering and uses the full residual-compensation budget of LoDiQ-Max.

ModelReorderingWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
Qwen3-8BRandom10.31167.94
Amax9.70669.37
Error-localized9.73069.75
Llama-3-8BRandom7.81670.16
Amax7.75970.52
Error-localized7.69071.52

Error-localized reordering achieves the highest zero-shot accuracy on both models and lower perplexity on Llama-3-8B.

4. Why NLL gradients for GRCA instead of KL?

GRCA allocates residual capacity before distillation. NLL gradients provide a teacher-independent model-level sensitivity signal for this decision. The subsequent KL objective optimizes the allocated parameters to recover teacher predictions; the two objectives serve different purposes.

Appendix A.3, Table 11 compares NLL- and KL-based allocation at the same 50% residual-parameter budget:

ModelScoring objectiveWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
Qwen3-8BNLL9.73569.54
KL9.77869.69
Llama-3-8BNLL7.68271.16
KL7.73670.86

NLL yields lower perplexity on both models, with mixed zero-shot differences. NLL scoring uses a BF16 reference forward; KL scoring uses a quantized tail-only reference and a detached teacher. Thus, this compares two allocation implementations, not an isolated loss substitution. NLL remains our default because it provides effective allocation without requiring a teacher during scoring.

Thank you for the constructive suggestions.

Thank you for the rebuttal. The new experiments partly address my concerns. The channel-selection results show gains over random selection, but I would still like to see how the proposed score compares with using gradients to select which weight columns to train.

However, some of my concerns are not fully addressed. My concern about limited novelty remains partially solved. LoDiQ learns from the full-precision model’s predictions, while the new QLoRA and EfficientQAT baselines do not, so not exactly fair comparison. Therefore, I would like to keep my original rating.

Reviewer_LBZFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets accuracy recovery in W4A4 quantization while addressing the problems of significant accuracy drop and substantial training resource requirement. It proposes LoDiQ that preservers supervision from the teacher's final predictive distribution while updating only chosen weight columns, selected by propagated-error score. Also, a residual compensation is used to improve the accuracy recovery. Across several models their method generally improves downstream accuracy compared to the existing post training quantization methods. Through controlled comparison experiments the claim is that LoDiQ consistently improves the accuracy–training-cost trade-off over strong PTQ and QAD references while preserving efficient low-bit inference.

Critical strengths
  1. Practical Resource Balancing: The paper addresses a highly relevant problem in W4A4 quantization by connecting accuracy recovery to the training resources required, leveraging localized parameter updates to sidestep the heavy memory footprint of full-parameter training.

  2. Demonstrated Downstream Improvements: The proposed method achieves notable accuracy improvements across several models and benchmarks compared to existing Post-Training Quantization (PTQ) methods.

  3. Hardware Integration: The implementation integrates learned corrections directly into low-bit GEMMs, and runtime profiling on hardware demonstrates that inference efficiency is preserved under the evaluated configurations.

Critical weaknesses
  1. Under-Justified Selection and Allocation Mechanics: The specific benefits of the proposed channel selection and residual allocation rules are not fully established. To isolate their exact contributions, the evaluation should include direct comparisons against simpler alternatives—such as random or activation-based channel selection—under identical parameter and training budgets. Additionally, it remains unclear how well the proposed error scores correlate with actual post-training accuracy improvements.

  2. Inconsistent Baseline and Benchmark Context: While the method yields significant accuracy gains on several setups, the improvements are inconsistent across all benchmarks, and the claim of near-BF16 recovery remains inconclusive. Stronger evidence is required to confirm a clear advantage over full Quantization-Aware Fine-Tuning (QAD), specifically regarding baseline tuning details, convergence dynamics, and uncertainty estimates.

  3. Incomplete Efficiency and Validation Profiling: The efficiency evaluation relies on estimations rather than empirical measurements for full QAD memory requirements, and it omits end-to-end calibration and preparation overheads. Furthermore, while runtime is evaluated on hardware for a single model, the accuracy benchmarks rely on simulated quantization; the paper should explicitly verify that these accuracy profiles hold during native FP4 hardware inference across all evaluated settings.

Questions for authors

Please address the following:

  1. Ablation of Selection Rules: Could you provide comparative results where the proposed channel selection strategy is replaced by simpler baselines (e.g., random selection or purely activation-magnitude-based selection) restricted to the exact same parameter and training budgets?

  2. Native FP4 Verification: Since the downstream accuracy evaluation relies on simulated quantization, have you verified whether the reported accuracy recovery is fully maintained when executing native FP4 inference on actual hardware across the full suite of benchmarks?

  3. QAD Baseline and Resource Metrics: To substantiate the claimed advantages over full QAD, could you provide empirical memory measurements (rather than estimations) for the baseline execution? Please also provide details regarding the tuning, convergence criteria, and uncertainty estimates for the QAD references.

  4. End-to-End Overhead: Please clarify the additional time and computational costs introduced by the initial preparation, error scoring, and calibration phases. How do these pre-processing overheads scale with model size?

1 comment

Thank you for recognizing LoDiQ’s practical resource savings, accuracy recovery, and hardware integration. We have strengthened the reordering and allocation comparisons, reproduced QAD at its original training-token scale, and added measured preparation and training costs.

1. Contributions of reordering and residual allocation

Section 4.4, Table 7 now compares simpler alternatives under the corresponding trainable-parameter budgets.

For channel reordering, each strategy trains the last K=128K=128 weight columns after reordering and uses the full residual-compensation budget of LoDiQ-Max. Error-localized results reuse the main-table checkpoints.

ModelReorderingWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
Qwen3-8BRandom10.31167.94
Amax9.70669.37
Error-localized9.73069.75
Llama-3-8BRandom7.81670.16
Amax7.75970.52
Error-localized7.69071.52

Error-localized reordering achieves the highest zero-shot accuracy on both models and lower perplexity on Llama-3-8B.

For residual allocation, Table 7(b) compares GRCA with random allocation and layer-error reduction under the 50% residual-parameter budget of LoDiQ-Pro. Layer-error reduction measures the decrease in layer-output MSE without gradient weighting.

ModelAllocationWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
Qwen3-8BRandom9.85369.03
Layer-error reduction9.73869.41
GRCA9.73069.94
Llama-3-8BRandom7.86770.60
Layer-error reduction7.83270.41
GRCA7.69070.83

GRCA improves both metrics over both alternatives. These comparisons test the downstream utility of the proposed scores; they do not assume a direct numerical correlation between proxy scores and accuracy gains.

2. Native FP4 evaluation

We apologize that the distinction between training and inference was insufficiently clear. Simulated quantization is used during distillation on H20 GPUs. Our LoDiQ evaluation backend on RTX 5090 executes native FP4 operations with packed low-bit weights and format-specific scales.

3. Full-scale QAD and measured training costs

We extended the QAD reproduction to 800M training tokens using the official code and dataset at the source study’s training scale. Following its dense-model reasoning setting, we evaluate AceReason-Nemotron-1.1-7B on AIME25 and additionally include AIME26.

MethodTraining tokensAIME25 ↑\uparrowAIME26 ↑\uparrow
BF16—65.2269.22
QAD800M61.8966.67
LoDiQ-Max1M62.3367.67

Table 4(b) now provides empirical costs for these runs:

MethodH20 GPUsTraining GPU-hoursPeak memory per GPU (GiB)
QAD, 800M tokens8495.1367.65
LoDiQ-Max, 1M tokens11.0323.32

These are measured costs at the respective training-token budgets. Our reproduced QAD achieves 61.89 on AIME25, close to the original paper’s 62.0. The original study uses 48 sampling runs per problem, while we use 30 for both QAD and LoDiQ.

The official release does not provide validated training configurations and token budgets for the other model families in our evaluation. Given full-scale QAD’s computational cost, we prioritized its supported reasoning setting. To broaden the comparison, Tables 3 and 19 additionally evaluate FP4-adapted QLoRA and EfficientQAT across three models and both formats.

4. Preparation overhead and model scale

Appendix A.7 now compares preparation time at two model scales using identical calibration settings on one H20 GPU:

ModelPreprocessing (s)GRCA (s)
Qwen3-8B117.69216.86
Qwen3-32B395.39866.65

Preprocessing includes activation-amax collection, propagated-impact scoring, and reorder-index construction. GRCA includes gradient scoring and budget allocation and is required only by LoDiQ-Pro. Timings use CUDA synchronization and exclude model loading and checkpoint serialization.

From 8B to 32B, these stages increase by 3.36×3.36\times and 4.00×4.00\times, respectively. Training uses gradient checkpointing to reduce activation storage; the measured peak memory includes the remaining activation and runtime allocations.

Thank you for the constructive suggestions, which helped us strengthen both the experimental evidence and the implementation clarification.

Reviewer_eaNcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoDiQ, a lightweight distillation framework for W4A4 LLM quantization. It uses model-level teacher supervision to update weights associated with quantization-sensitive channels while freezing the remaining parameters. Quantized activation residuals provide additional compensation, with gradient-guided allocation controlling the compensation budget and compact teacher targets reducing storage costs. Experiments evaluate accuracy, training cost, and inference efficiency under MXFP4 and NVFP4.

Critical strengths
  1. LoDiQ improves downstream accuracy over several format-matched PTQ baselines while updating only a small fraction of model parameters. This provides a useful option for quantization recovery with limited trainable-state memory.

  2. The learned compensation remains in the target low-bit format and is integrated into expanded GEMMs. Experiments on an RTX 5090 demonstrate practical inference speed and memory benefits under both MXFP4 and NVFP4.

Critical weaknesses
  1. The ablations do not sufficiently establish whether the gains come from the proposed selection rules or mainly from distillation and additional compensation capacity. Table 5 shows substantial benefits from distillation, but does not compare simpler channel-selection and layer-allocation strategies under the same training and parameter budgets. Since selective updates and compensation are established approaches, these controls are important for assessing the contribution of the proposed designs.

  2. The comparisons with full-parameter QAD do not sufficiently establish the claimed accuracy–training-cost advantage. Table 7 compares accuracy on one model and two mathematical reasoning benchmarks, while the broader comparisons against strong PTQ baselines omit full QAD. Table 3 reports measured LoDiQ training costs but only an analytical memory estimate for full QAD. The results therefore leave unclear whether LoDiQ’s higher accuracy comes from parameter localization or differences in optimization and tuning, and do not jointly demonstrate its accuracy and training-cost advantages in the same setting.

Questions for authors
  1. Do the proposed channel-selection and compensation-allocation strategies outperform simpler alternatives under the same training and parameter budgets?

  2. Does the reproduced full-parameter QAD baseline match the original work’s performance? Can the accuracy and training costs of strong PTQ, full QAD, and LoDiQ be compared in the same setting?

2 comments

Thank you for recognizing the practical value of localized recovery and its low-bit deployment benefits. Your comments motivated additional reordering and allocation controls, together with a full-scale QAD reproduction and measured training costs. We address the two main concerns below.

1. Contributions of channel reordering and residual allocation

We have added comparisons with simpler alternatives in Section 4.4, Table 7.

Channel reordering. Table 7(a) compares random, activation-amax, and error-localized reordering followed by distillation under LoDiQ-Max. Each strategy selects the last K=128K=128 weight columns after reordering for training and uses the same full residual-compensation budget. The error-localized results reuse the checkpoints reported in the main tables.

ModelReorderingWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
Qwen3-8BRandom10.31167.94
Amax9.70669.37
Error-localized9.73069.75
Llama-3-8BRandom7.81670.16
Amax7.75970.52
Error-localized7.69071.52

Error-localized reordering achieves the highest zero-shot accuracy on both models. These comparisons support its benefit beyond distillation with randomly selected or activation-amax-selected channels.

Residual allocation. Table 7(b) compares GRCA with random allocation and layer-error reduction under LoDiQ-Pro. Layer-error reduction scores the decrease in layer-output MSE from residual compensation without gradient weighting. All strategies use the same 50% residual-parameter budget.

ModelAllocationWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
Qwen3-8BRandom9.85369.03
Layer-error reduction9.73869.41
GRCA9.73069.94
Llama-3-8BRandom7.86770.60
Layer-error reduction7.83270.41
GRCA7.69070.83

GRCA achieves lower perplexity and higher zero-shot accuracy than both alternatives on both models. Together, these results strengthen the evidence for error-localized reordering and budgeted allocation beyond simply adding trainable compensation capacity.

2. Full-scale QAD reproduction and measured training costs

We have reproduced QAD at the original 800M-token scale using the official code and dataset, following the published training recipe. We evaluate AceReason-Nemotron-1.1-7B on AIME25, following the original QAD study, and additionally include AIME26. Our reproduced AIME25 score is 61.89, close to the original study's 62.0. Our comparisons average 30 evaluation rounds for all methods; the source study uses 48 rounds.

Table 9 reports:

MethodTraining tokensAIME25 ↑\uparrowAIME26 ↑\uparrow
BF16—65.2269.22
RTN—59.7864.44
QAD1M59.8966.44
QAD800M61.8966.67
LoDiQ-Max1M62.3367.67

Table 4(b) now provides measured training costs on the same model and NVFP4 format:

MethodH20 GPUsTraining time (h)GPU-hoursPeak memory (GiB/GPU)
QAD, 800M tokens861.89495.1367.65
LoDiQ-Max, 1M tokens11.031.0323.32

LoDiQ-Max achieves higher AIME accuracy while using 1/800 as many training tokens and updating 4.67% as many parameters. These measurements jointly demonstrate accuracy and training cost in the same model setting, supplementing the original analytical memory estimate. Appendix A.2 specifies the timing scope.

The official QAD release does not provide validated adaptations, model-specific training configurations, or training-token budgets for the other models in our evaluation. Given the substantial cost of full-scale QAD, we prioritized reproducing its supported AceReason setting at the original token scale.

As a complementary comparison, we evaluated FP4-adapted QLoRA and EfficientQAT on Llama-3.2-3B, Llama-3-8B, and Qwen3-8B under both MXFP4 and NVFP4. Table 3 reports WikiText-2 perplexity and six-task zero-shot accuracy, while Table 19 includes MMLU, extending the training-based evaluation across models and tasks.

Thank you again for helping us strengthen the experimental evidence.

Thank you for the additional experiments. The full-scale QAD reproduction and measured training costs address several of my concerns, and the new ablations show modest gains from the proposed selection and allocation rules. I remain unsure how much these designs improve on a simpler parameter-efficient method under the same distillation setup. I will therefore keep my original rating of 2 (Weak Rejection).

Reviewer_8J68OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoDiQ, a parameter-efficient quantization-aware training (QAT) method to enhance 4-bit weight-activation quantized (W4A4) large language models (LLMs). To mitigate the large training cost from Quantization-Aware Distillation (QAD), LoDiQ selectively trains a small portion of weights. Based on the quantization impact on output perturbation, LoDiQ selects the salient weight channels, reorder each channel groups based on activation magnitudes, then selectively trains them. To futher improves accuracy, LoDiQ employs quantized residual compensation with Gradient-Guided Residual Compensation Allocation(GRCA) under the fixed trainable parameter budget. For the training objective, LoDiQ only uses teacher's top-16 token probabilities. Across Llama3, Llama3.2, Qwen3, and Moonlight models, LoDiQ shows higher accuracy than PTQ baselines, while significantly reduces the training cost than QAD.

Critical strengths
  • It is important to reduce training costs of QAD. While PTQ methods show clear limitation to improve accuracy under microscaled W4A4, QAD is expensive. I think this is a timely topic.
  • This paper is easy to follow, and the diagrams help to understand the overall method.
  • The proposed method is evaluated on not only simple question-answering task but also reasoning tasks.
  • The paper reports the training cost and the end-to-end GPU performance, enabling clearer understanding of this method's trade-offs.
Critical weaknesses
  • The novelty of this paper is limited. Defining a salient weight channel based on output perturbation (rather than activation magnitude itself) is widely used approach. Moreover, fine-tuning for thoes channels are already proposed in OWQ. But this paper did not cite or compare with that method.
  • I think this method should be categorized to parameter-efficient fine-tuning method rather than PTQ methods. However, the main evaluations are focused on comparing with the PTQ baselines. The paper should compare with PEFT methods (Quantization-aware LoRA variants, EfficientQAT, ...)
  • Reordering the channels based on activation magnitudes makes the acitvation distribution to be quantized friendly at inference, therefore already adapted on many PTQ methods. Please distangle the accuracy gain from reordering and fine-tuning. Adding experiments on LoDiQ w/o reordering would be helpful.
  • The overhead of Gradient-Guided Residual Compensation Allocation should be discussed. Especially, please compare the total calibration/fine-tuning time and the peak memory with other PTQ methods. Even with the trainable parameter is small, the intermediate activations are still needed, and they become larger when we increase context lengths or the batch sizes.
  • On NVFP4, the performance gain is limited, and perplexity is higher than PTQ methods in some columns. Also, there are only ~2% gap in AIME, which has 30 questions (one question takes 3.3% accuracy, is it really meaningful gap?). However, there is a still gap with BF16 baseline.
Questions for authors

Please see weaknesses.

2 comments
AuthorsOpenReview ↗

Part 2/2: Reordering Ablations, Preparation Costs, Activation Memory, and AIME Evaluation

3. Effect of reordering under distillation

Thank you for suggesting this comparison. Table 7(a) compares random, activation-amax, and error-localized reordering followed by distillation under LoDiQ-Max. Each strategy selects the last K=128K=128 weight columns after reordering for training and uses the same full residual-compensation budget. Random reordering provides a sensitivity-agnostic control, while amax tests whether activation magnitude alone is sufficient.

ModelReorderingWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
Qwen3-8BRandom10.31167.94
Amax9.70669.37
Error-localized9.73069.75
Llama-3-8BRandom7.81670.16
Amax7.75970.52
Error-localized7.69071.52

Error-localized reordering achieves the highest zero-shot accuracy on both models. These comparisons support the benefit of error-localized reordering beyond distillation with randomly selected or activation-amax-selected channels.

4. Preparation overhead and intermediate activation memory

Thank you for highlighting the importance of preparation costs and activation storage. Section 4.3 and Appendix A.2 now report preprocessing and GRCA time separately.

On Qwen3-32B under MXFP4, channel preprocessing takes 395.39 seconds. GRCA adds 866.65 seconds and is required only by LoDiQ-Pro. Lite and Max do not perform GRCA allocation. The measured training costs are:

ConfigurationPreprocessing (s)GRCA (s)Training (GPU-hours)Peak allocated memory (GiB)
LoDiQ-Lite395.39—4.4977.01
LoDiQ-Pro395.39866.654.5981.32
LoDiQ-Max395.39—4.6785.69

The preprocessing stage is shared across configurations. Training uses one H20 GPU and approximately 1M tokens. The stage timings exclude checkpoint transfer and downstream evaluation; Appendix A.2 specifies the measurement scope.

We agree that reducing trainable parameters does not eliminate intermediate activations or end-to-end backpropagation. Our implementation enables gradient checkpointing to reduce activation storage, and the reported peak memory includes the remaining activation and runtime allocations. Memory can still increase with sequence length and batch size. The primary memory saving from localization is in parameter gradients and optimizer states.

5. NVFP4 gains, the BF16 gap, and interpretation of AIME results

We agree that LoDiQ does not obtain the lowest perplexity in every NVFP4 comparison, and a gap to BF16 remains on some tasks. Our results demonstrate recovery across complementary metrics, not uniform superiority on every metric. For example, on Qwen3-8B, LoDiQ improves the zero-shot average over the strongest matched NVFP4 PTQ baseline by 1.14 percentage points. LoDiQ achieves strong overall recovery under NVFP4, with leading downstream accuracy across the evaluated models.

We apologize that the AIME evaluation protocol was not sufficiently clear. Following common practice in reasoning-model evaluation, we average accuracy over repeated samples to reduce sampling variability. Specifically, we use 30 evaluation rounds on the same benchmark questions. Consequently, the reported score is not restricted to increments of 100/30≈3.33100/30 \approx 3.33 percentage points, as it would be for one evaluation round.

We have now reproduced QAD at the original 800M-token training scale using the official code and dataset, following the published training recipe. We evaluate AceReason-Nemotron-1.1-7B on AIME25, following the model and benchmark used in the original QAD study, and additionally include AIME26. Under the same 30-round evaluation protocol used for LoDiQ, Table 9 reports:

MethodTraining tokensAIME25 ↑\uparrowAIME26 ↑\uparrow
BF16—65.2269.22
QAD800M61.8966.67
LoDiQ-Max1M62.3367.67

We appreciate your detailed feedback.

Reviewer_8J68OpenReview ↗

Thank you for the rebuttal

Thank you for the thoughtful rebuttal and the additional experiments. I appreciate the authors’ efforts to address my concerns. The preparation-cost breakdown is helpful, and I understand that the additional preprocessing and GRCA overhead is relatively modest.

Some concerns nevertheless remain. Retaining a small number of high-precision columns does not necessarily prevent efficient inference, as explored in prior work such as Atom (MLSys ’24). I also find the gains over activation-amax reordering and the adapted OWQ baseline relatively modest. Similarly, the NVFP4 results show some improvements over competing methods, but a noticeable gap to BF16 remains on some tasks.

While I recognize the practical value of the problem and appreciate the expanded evaluation, I still see the method as an incremental contribution, with empirical gains that are not yet strong enough to change my overall assessment. I therefore maintain my original rating, while acknowledging the helpful clarifications and additional evidence provided in the rebuttal.

Reviewer_7WrVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper addresses accuracy loss in W4A4 (MXFP4/NVFP4) LLMs, where PTQ lacks model-level supervision and full QAD is expensive.
  • It proposes LoDiQ, which selects error-sensitive channels and trains only those weight columns plus a quantized residual compensation under teacher KL, packed into a single low-bit GEMM.
  • The method achieves higher accuracy than strong PTQ baselines on five models with a speedup over BF16.
Critical strengths
  1. The method offers a practical middle ground between PTQ and full QAD. Training a few percent of weights clearly beats PTQ on five models, and the deployed model stays a single W4A4 GEMM with measured speedups.
  2. The paper reports costs concretely. Memory, GPU-hours and data ablations make the accuracy–cost trade-off easy to judge.
Critical weaknesses
  1. The localization claim is not tested. No control trains K random or amax-selected channels, and the residual part follows ARCQuant, so it is unclear whether the selection rule matters.
  2. The QAD comparison is weak. It covers one model and AIME only, and the reproduced QAD uses about 1/8 of the tokens in the source study.
Questions for authors
  1. How does LoDiQ compare with training K random channels or K activation-amax channels under the same setup?
  2. How does LoDiQ compare with QAD at the source study's token scale and on other models?
  3. Some Qwen3 results have lower WikiText-2 perplexity than BF16 (e.g., 7.34 vs 7.61 on Qwen3-32B NVFP4). Is the evaluation setup the same?
2 comments
AuthorsOpenReview ↗

Response to Reviewer: Channel Reordering, Full-Scale QAD, and Evaluation Consistency

Thank you for recognizing the practical value of localized recovery and the concrete reporting of training and deployment costs. Your comments motivated additional channel-reordering controls, a full-scale QAD reproduction, and broader training-based comparisons. We address each question below.

1. Comparison with random and activation-amax reordering

We have added comparisons with random and activation-amax reordering on Qwen3-8B and Llama-3-8B under MXFP4 (Section 4.4, Table 7(a)). All strategies select the last K=128K=128 weight columns after reordering for training and use the full residual-compensation budget of LoDiQ-Max. The error-localized results reuse the checkpoints reported in the main tables.

ModelReorderingWT2 PPL ↓\downarrow0-shot Avg. ↑\uparrow
Qwen3-8BRandom10.31167.94
Amax9.70669.37
Error-localized9.73069.75
Llama-3-8BRandom7.81670.16
Amax7.75970.52
Error-localized7.69071.52

Error-localized reordering achieves the highest zero-shot accuracy on both models, improving over amax by 0.38 and 1.00 percentage points, respectively. It also achieves lower perplexity on Llama-3-8B. These results support the overall benefit of error localization.

2. QAD at the original training-token scale and broader training-based comparisons

We have extended the QAD reproduction from 100M to 800M training tokens, using the official code and dataset at the source study’s token scale. Table 9 now reports the following results on AceReason-Nemotron-1.1-7B under NVFP4. Both AIME benchmarks are averaged over 30 evaluation rounds.

MethodTraining tokensAIME25 ↑\uparrowAIME26 ↑\uparrow
BF16—65.2269.22
RTN—59.7864.44
QAD1M59.8966.44
QAD800M61.8966.67
LoDiQ-Max1M62.3367.67

LoDiQ-Max exceeds the reproduced 800M-token QAD baseline by 0.44 and 1.00 percentage points while using 1/800 as many training tokens. We also added measured training costs in Table 4(b): QAD requires 495.13 GPU-hours on eight H20 GPUs, while LoDiQ-Max requires 1.03 GPU-hours on one H20. These costs correspond to the respective 800M- and 1M-token runs.

For the QAD comparison, we follow the source work’s dense-model and mathematical-reasoning setting, with AIME26 added as a newer benchmark. The official release does not provide validated model-specific training configurations and token budgets for the other model families in our evaluation. Given the computational cost of full-scale QAD, we prioritized reproducing this supported setting at its original token scale. We therefore limit the direct QAD conclusion to the evaluated reasoning setting.

To broaden the comparison beyond PTQ and mathematical reasoning, we additionally evaluated FP4-adapted QLoRA and EfficientQAT on Llama-3.2-3B, Llama-3-8B, and Qwen3-8B under both MXFP4 and NVFP4. Table 3 reports WikiText-2 perplexity and six-task zero-shot accuracy, and Table 19 provides the detailed results, including MMLU. QLoRA is evaluated both with retained high-precision adapters and after merging and requantization.

3. WikiText-2 perplexity below BF16

Thank you for raising this consistency check. We have checked that the BF16 and LoDiQ results use the same WikiText-2 evaluation protocol. The lower perplexity is not due to a different test split or scoring setup.

BF16 is a reference checkpoint, not a theoretical lower bound on held-out perplexity. Distillation changes the student’s predictions, and matching the teacher distribution does not require the student’s empirical next-token NLL to remain above the teacher’s on every evaluation corpus. A student can therefore obtain slightly lower WikiText-2 perplexity without being uniformly better across tasks.

We also observed this behavior in the additional FP4-adapted QLoRA experiments on Qwen3-32B: retaining the adapters yields perplexities of 7.395 under MXFP4 and 7.025 under NVFP4, compared with 7.610 for BF16. This indicates that the observation is not specific to LoDiQ. We interpret the lower perplexity as a corpus-specific result, not evidence that quantization universally improves the full-precision model.

We appreciate these suggestions, which have helped us strengthen the experimental controls and clarify the scope of our conclusions.

Thank you for the detailed response and the new experiments! The full-scale QAD reproduction, the added training-based baselines and the clarification on WikiText-2 perplexity address some of my concerns. However, as far as I can tell, the advantage of the proposed rule over simple activation-amax selection is small and mixed. It is also still unclear why the error-localized rule should work better. Therefore, I will keep my rating for now and take other reviewers view into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.