acceptodds
Withdrawn from ICLR 2027

EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Quantization has emerged as a mainstream approach for deploying Large Language Models (LLMs) on resource-constrained devices, yet reducing precision below 4-bit typically leads to severe performance degradation or prohibitive retraining costs. In this paper, we propose EdgeRazor, a lightweight framework for LLMs via Mixed-Precision Quantization-Aware Distillation. It contains three modules: Structural Quantization with Mixed Precision for fine-grained control over bit-widths, Layer-Adaptive Feature Distillation for dynamically selecting the most informative features for alignment, and Entropy-Aware KL Divergence for balancing forward and reverse KL to handle data from both human annotation and model distillation. With 1.88-bit weights, 8-bit activations, and 8-bit KV cache, EdgeRazor on Qwen3-0.6B outperforms the strongest 2-bit baseline by 11.40 points in average accuracy and the strongest 3-bit baseline by 4.38 points. In terms of training efficiency, EdgeRazor quantizes MobileLLM-350M with 4-10 fewer training tokens than the leading quantization-aware training method. On the deployment side, EdgeRazor achieves consistently higher compression ratios than existing methods across all bit-widths. Exported to the TQ2_0 quantization type supported by llama.cpp, the 1.58-bit Qwen3-0.6B quantized by EdgeRazor reduces the model size from 1.11 GB to 0.19 GB and decodes 15.16 faster than its 16-bit counterpart. Extensive experiments on the MobileLLM and Qwen families confirm the effectiveness and efficiency of EdgeRazor.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Lightweight, Quantization-Aware Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5MYKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

EdgeRazor combines structural mixed-precision quantization with feature- and logit-level distillation for low-bit LLM deployment. It reports strong sub-2-bit results on 0.35–1.7B models and demonstrates a compact Qwen3-0.6B deployment on an Apple M4 Pro. However, its scalability and computational efficiency (the central premises of a lightweight LLM compression method) are not adequately demonstrated.

Critical strengths
  1. Strong low-bit results on small edge models. EdgeRazor achieves substantial accuracy improvements over existing low-bit baselines on <2B models. The deployment of a 0.19GB Qwen3-0.6B model through llama.cpp also demonstrates practical benefits in a highly resource-constrained setting.
Critical weaknesses
  1. Scalability beyond small models is unsubstantiated. All sub-3-bit text-model experiments are limited to MobileLLM-350M, Qwen3-0.6B, and Qwen3-1.7B. The only 7B experiment uses 4-bit weights on a multimodal model. EdgeRazor additionally requires full-teacher feature and logit distillation over multiple training epochs, making its feasibility for standard 7B–70B text LLMs unclear. The current evidence supports small task-adapted edge models rather than general-purpose LLM compression.

  2. The lightweight and efficiency claims lack transparent, matched-budget evidence. The models are trained on eight A100-80GB GPUs, but the paper does not report important details such as total training time (in wall-clock and/or GPU-hours), peak training memory, teacher inference cost, data-generation cost, etc. The efficiency claim relies primarily on the number of training tokens used for MobileLLM-350M, although feature- and logit-level distillation can have substantially higher per-token costs than competing methods. It is thus unclear whether EdgeRazor is Pareto-optimal in model quality versus total compression cost under matched training conditions. Therefore, an efficient compression method should be evaluated for both the cost of producing the compressed model and the efficiency of deploying it, rather than only the latter.

Questions for authors
  • Larger-model scalability. Can the authors demonstrate sub-3-bit quantization on standard 7B-70B text LLM settings and report how training memory and computation scale with model size?

  • End-to-end compression cost. What are the wall-clock time, total GPU-hours, peak memory, training-token count, and teacher or data-generation costs for each model and bit width (Please provide all details of the setting and results for transparency and reproducibility)? How does the resulting quality-cost Pareto frontier compare with EfficientQAT, ParetoQ, BitDistiller, and more recent methods (if they exist) under matched (training budget and output model) conditions?

  • Compression versus task-specific training. How much of the reported improvement comes from the quantization method rather than supervised task adaptation? Could the authors compare against a BF16 model and the main QAT/QAD baselines trained with the same data, teacher signals, and compute budget?

My rating is 1 because results limited to models of at most 1.7B parameters do not establish a practically useful LLM quantization method. A scalable method should (at least) handle standard 7B–70B models at a clearly reported and usable cost: EfficientQAT, for example, quantizes 70B models in tens of GPU-hours, whereas EdgeRazor applies extensive teacher-feature and logit distillation only to small text models and does not report its GPU-hour cost at all. This fails the minimum evidence needed to support the paper’s central claim of lightweight and scalable LLM quantization.

Reviewer_5w1xOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EdgeRazor is a quantization-aware distillation framework combining periodic allocation of ternary and 4-bit weight channels, adaptive selection of teacher layers for feature distillation, and teacher-entropy-based mixing of forward and reverse KL divergence. The paper evaluates accuracy on MobileLLM and Qwen models across language and video-understanding benchmarks, provides component ablations, and reports training-token budgets and llama.cpp deployment measurements for ternary and 4-bit configurations.

Critical strengths

The framework demonstrates promising accuracy under aggressive quantization, with improvements over the evaluated low-bit baselines on held-out benchmarks for both Qwen3-0.6B and Qwen3-1.7B. These results provide empirical support that the combined approach preserves useful capabilities beyond the tasks whose training splits are included in distillation.

Critical weaknesses
  1. The deployment results do not validate the proposed mixed-precision format. The advertised “1.88-bit” average counts ternary weights as log₂(3) bits and excludes scale overhead, without specifying an actual packing scheme. No deployment results are provided for the mixed ternary/4-bit configurations. Instead, the llama.cpp experiments evaluate pure ternary models using TQ2_0, which uses 2-bit packing and requires 2.0625 bits per weight including scales. Thus, the advertised precision settings and measured deployment results do not correspond.
  2. The methodological gains are insufficiently established. Fixed periodic precision allocation does not address the changing channel importance used to motivate SQMP, and a matched random-allocation baseline is missing. EAKLD closely relates to existing confidence- and entropy-aware KL mixing, but its controlled ablation compares only against forward KL, leaving its advantage over existing or simpler alternatives unclear.
Questions for authors
  1. What exactly is deployed in the reported experiments? Please clarify the correspondence between the configurations evaluated for accuracy, compression, and runtime. How are the mixed ternary/4-bit weights packed, what is their actual storage cost including scales, and how do these configurations relate to the pure-ternary TQ2_0 deployment results?
  2. What is the principal methodological novelty of EdgeRazor? Please distinguish the proposed contributions from existing methods and identify the evidence supporting them. In particular, why should a fixed one-in-eight allocation of 4-bit channels address changing channel importance during training, and what substantive advance do the distillation components provide over prior adaptive distillation approaches?
Reviewer_Nx1WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a quantization-aware distillation framework for mixed-precision LLMs. It has three parts: a periodic allocation of ternary and 4-bit channels, feature supervision taken from teacher layers whose adjacent-layer cosine similarity is low, and a distillation loss that mixes forward and reverse KL using entropy-based weights. The authors evaluate on MobileLLM-350M, Qwen3-0.6B/1.7B and a 4-bit multimodal model. They report strong accuracy at low bit-widths, ablations for each module, smaller training-token budgets than prior work, and CPU inference numbers from llama.cpp on Apple Silicon.

Critical strengths
  1. The low-bit results are strong, and the held-out evaluation makes them more credible. In Table 2, Qwen3-0.6B at W1.88-A8-KV8 gets 41.76 average and 30.59 H-Average, whereas OmniQuant at W2-A8-KV8 gets 30.36 and 9.19. I appreciated that the held-out subset is reported separately: the training recipe uses the training splits of several of the evaluated tasks, so without it one could worry that the improvement simply reflects task supervision. The held-out numbers largely rule that out.
  2. The ablations and the deployment pipeline give the method real substance. Table 5 shows what is lost when the allocation scheme, the feature supervision, or the KL objective is swapped out. The supplementary material also contains scripts for training, evaluation, compression accounting and export/benchmarking. Taken together, this is more than a simulated-accuracy study of low-bit weights, which is what many papers in this area amount to.
Critical weaknesses
  1. The “lightweight training” claim rests mostly on token counts, not on comparable total compute. Figure 3 contrasts 1.2–3.1B student training tokens with 10–30B for ParetoQ. But distillation also requires teacher forward passes and handling of features/logits, so the token comparison alone does not show a 4–10× reduction in training cost. Similarly, Table 5 shows that each component helps locally, but it does not settle whether the advantage over other distillation recipes would hold once teacher access, data, optimization and total compute are matched. Since cheap training is one of the main selling points of the paper, I think this needs to be addressed.
  2. The best accuracy numbers and the deployment numbers come from different setups, and they have not yet been tied together at a single verified operating point. The main quality experiments use fake/structural precision with 4-bit embedding and output layers, while the headline speedup comes from a TQ2_0 export. Figure 5 compares that export against BF16, and Figure 4 credits a large part of the compression to quantizing the embedding/head parameters, which several baselines leave unquantized. These are real practical gains, but in the current comparison the effect of the algorithm is mixed up with differences in quantization coverage and in backend. To support the claimed quality–efficiency advantage, I would want to see accuracy measured on the exact exported model, compared against low-bit baselines that are optimized to the same degree, with actual packed sizes in bytes.
Questions for authors
  1. Could you report training FLOPs or GPU-hours that include the teacher, for EdgeRazor and for the strongest QAD/QAT baseline, with the same data and stopping criteria? It would be helpful to break this down into teacher inference, student training, and any preprocessing.
  2. What downstream accuracy does the exact TQ2_0 model used in Figure 5 achieve? Please also state the precision of the embedding/head, the activation and KV precision, the number of CPU threads, and whether the BF16 and quantized runs use identical CPU/backend settings.
  3. Would it be possible to compare against a BitDistiller/CAKLD recipe under matched settings, and to report variation across training seeds? The component gains are around 1–2 points, so seed variance matters. Appendix C explains the coefficient mismatch, but an end-to-end matched comparison is what is really needed to attribute the gain to EAKLD.
  4. How does the allocation behave at rho=0 and rho=1, and for ratios whose reciprocal is not an integer? I think Equation 2 and the description of the rounded super-groups should give the bit-width that is actually realized, along with the overhead from scales and packing.
  5. Missing related references: [1] "VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs". ACM Multimedia 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.