acceptodds
Under review as a conference paper at ICLR 2027

Channel-group Hierarchical Allocation for Mixed Precision

Abstract

Low-precision quantization can substantially reduce the memory and computation costs of Transformer inference, but aggressive 4-bit floating point (FP4) quantization often degrades model quality due to highly non-uniform quantization sensitivity across the model. Mixed-precision quantization addresses this challenge by assigning higher precision to sensitive regions while keeping less sensitive regions in lower precision. Existing fine-grained mixed-precision methods often select high-precision regions directly from local or global sensitivity scores, which can over-concentrate the high-precision budget in a subset of the model while leaving other sensitive regions insufficiently protected. We introduce CHAMP (Channel-group Hierarchical Allocation for Mixed Precision), a post-training framework combining FP4 and 8-bit floating point (FP8) that hierarchically distributes a global high-precision budget across model components and Transformer layers and constructs the resulting fine-grained channel-group precision mask using sensitivity-aware selection. CHAMP combines sensitivity- and size-aware module allocation, layer-specific adjustment, and local–global channel-group selection. Across language and video generation tasks, CHAMP provides the largest improvements when the FP8 budget is limited, outperforming prior fine-grained sensitivity-based mixed-precision allocation in this regime. CHAMP narrows the gap to full FP8 in our primary language-model settings, improves video frame-level fidelity, maintains comparable generation quality, and achieves higher theoretical throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training Quantization, Mixed-Precision Quantization, Low-Precision Inference, Large Language Models, Diffusion Transformers, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K2xuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CHAMP, a post-training mixed-precision framework that hierarchically distributes a global FP8 budget across modules, Transformer layers, and channel-groups using calibration-based sensitivity estimates. It constructs matched weight–activation precision masks and combines module-level allocation, layer-level adjustment, and local–global channel-group selection. The authors compare CHAMP with FGMP on Llama-3.1-8B, Wan 2.2, and HunyuanVideo, reporting improved LLM accuracy at low FP8 budgets and better video reconstruction fidelity relative to BF16. Ablations support the contributions of the allocation components. The efficiency results are based on the compute-weighted MXFP8 ratio and an idealized throughput model, rather than measurements with actual low-precision kernels.

Critical strengths
  1. The structured allocation of a limited high-precision budget is supported by concrete diagnostics. The oracle budget-swap diagnostics in the appendix show that allocations induced by global sensitivity ranking can be improved without changing the total budget. The leave-one-component-out ablation in Table 12 further supports module-level allocation, layer-level adjustment, and local–global selection. This provides stronger motivation than an allocation heuristic alone.

  2. There are identifiable quality gains in the low-budget regime. At r = 0.1, CHAMP improves average accuracy over FGMP by 1.07 percentage points, while video fidelity improves on two backbones. These results suggest practical potential when the high-precision budget is particularly constrained.

Critical weaknesses
  1. Deployment efficiency remains unvalidated, and theoretical throughput does not establish a practical advantage. Section 5.3 and Appendix B.4 explicitly use fake quantization, dequantizing tensors before computation. The throughput model assumes that halving bit-width doubles arithmetic throughput and omits the costs of channel-group execution, memory access, precision conversion, and other operators. The idealized gains over FGMP in Table 4 are already small—for example, from 3.10T to 3.20T on Llama—and actual execution overhead could change the conclusion. The current evidence therefore primarily supports precision-allocation quality rather than a demonstrated deployment acceleration.

  2. The generalization and statistical evidence for the quality gains are limited. The LLM evaluation covers only one 8B model. All quantitative and qualitative video results use eight Sora demonstration prompts and a fixed random seed of 997. At r = 0.2, the LLM improvement is only 0.44 percentage points, and the average VBench score on Wan is slightly below FGMP. Greater pixel-level similarity to BF16 does not directly establish better video generation quality. These results are insufficient to support broad advantages across models and workloads.

Questions for authors
  1. Deployment verification: Could you compare CHAMP and FGMP using at least one real kernel supporting the target numerical formats, reporting throughput, latency, peak GPU memory, and precision-conversion overhead? If this is not currently feasible, should the main claim be explicitly limited to allocation quality under a precision budget?

  2. Generalization verification: Do the main gains persist on a larger, independent video prompt set, across multiple random seeds, and on another LLM architecture? What are the confidence intervals for the differences observed with the eight-prompt evaluation?

  3. Mechanism clarification: After local–global channel-group selection redistributes the remaining budget across layers, how much of the original layer-level allocation is preserved in the final mask? Could you report the realized FP8 ratio, the compute-weighted MXFP8 ratio, and a controlled comparison against global ranking at the same granularity?

1 comment

We thank the reviewer for the careful assessment and for recognizing the experimental motivation behind our method, the value of our ablations, and the improvements of CHAMP. We give our response as follows:

Deployment verification and actual inference efficiency We agree that the theoretical throughput estimates should not be interpreted as demonstrated end-to-end inference acceleration. However, the theoretical analysis can serve a useful purpose: it can provide a consistent basis for comparing the compute-weighted high-precision budgets of different methods. For example, when r=0.2 on Llama-3.1-8B, CHAMP and FGMP achieves r^=0.25\hat{r}=0.25 and r^=0.29\hat{r}=0.29, respectively, corresponding to a better theoretical throughput estimates. These results show that our method can rr towards a different mix of computationally costly and cheap operations while maintaining improved model accuracy. However, understanding the theoretical benefits requires implementing specialized hardware kernels that would involve substantial engineering effort beyond the current scope of the work.

We will strengthen this clarification in the abstract, experimental discussion, and limitations section of the final manuscript.

Generalization, additional prompts, and statistical confidence we would like to highlight that our evaluation spans 7 language-understanding benchmarks (with Llama-3.1-8B), and two different video generation models (Wan 2.2-T2V-A14B and HunyuanVideo). We believe that these evaluations show the suitability of CHAMP in both autoregressive language models and diffusion-based video generation models.

We agree that improved frame-level similarity to BF16 does not automatically imply improved perceptual generation quality. These metrics measure different aspects of performance, and we intentionally report both. Regarding VBench, we would like to clarify that some of its dimensions have limited sensitivity. For example, motion smoothness scores are consistently close to 99, leaving little room to distinguish between methods. Therefore, we argue that small VBench differences should be interpreted cautiously, as they do not necessarily indicate meaningful differences. This, indeed, motivates using frame-level fidelity metrics to directly evaluate how our method compares against baselines in preserving BF16 outputs.

Preservation of layer-level allocation after local–global selection We thank the reviewer for highlighting an important aspect of the allocation mechanism. We clarify that for a layer initially allocated K FP8 channel-groups, the local selection stage reserves [K/2] groups for the most sensitive channel-groups within that layer. The remaining budget is then pooled across layers belonging to the same module and redistributed according to the channel-group sensitivity. This design choice (1) preserves a minimum local allocation, and (2) allows the rest of the budget to prioritize more sensitive channel-groups across layers of that module. Thus, the original layer-level allocations are not strictly preserved. Furthermore, Table 12 shows the value of combining local and global selection: comparing with full CHAMP, only local selection reduces accuracy by 0.27%, and using global selection reduces it by 0.50%.

Regarding realized FP8 ratios, Table 4 already shows the compute-weighted MXFP8 ratios used in our theoretical efficiency analysis. Similarly, our oracle experiments in Appendix B.5 investigate global ranking at the channel-group granularity and demonstrate that its allocation can be improved through even small budget reallocation.

Reviewer_bJanOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CHAMP, a hierarchical mixed-precision quantization framework that combines FP4 and FP8 for efficient Transformer inference. It allocates the high-precision budget across modules and layers, then uses a local-global strategy to select sensitive channel groups. Experiments on Llama-3.1-8B and two video generation models show that it improves over FGMP under a limited FP8 budget.

Critical strengths
  • Clear motivation. The paper identifies an important limitation of precision allocation based on global sensitivity and gives a sound motivation for hierarchical budget allocation.

  • Practical design. The proposed channel-group allocation uses matched precision for weights and activations, which can simplify hardware implementation compared with choosing their precisions independently.

Critical weaknesses
  • Limited methodological novelty. The method builds mainly on existing sensitivity-based mixed-precision quantization, especially FGMP. The proposed module-level allocation, hierarchical adjustment, and local-global selection are largely heuristic strategies. The methodological contribution seems incremental.

  • No real hardware evaluation. All experiments use simulated quantization, and the reported throughput improvements are theoretical. The paper does not demonstrate actual inference speedups, memory savings, or hardware efficiency. This is a major limitation for a method aimed at efficient inference.

Questions for authors
  • Relatively weak baselines. The experiments mainly compare CHAMP with FGMP. Comparisons with other recent mixed-precision methods, such as MicroMix and MixLLM, are missing. Stronger baselines would help establish the advantage of hierarchical allocation.

  • Limited and inconsistent improvements. At an FP8 ratio of 0.2, the accuracy gain on Llama-3.1-8B is only 0.44 percentage points, and at higher ratios CHAMP does not always outperform FGMP. In some settings, the VBench improvements on video generation are also marginal or even negative. The paper does not report variance across repeated runs or any statistical significance analysis to support these relatively small gains.

  • Limited experimental coverage. Only one 8B language model is evaluated, and the video experiments use only eight prompts. In addition, some attention tensors in HunyuanVideo are left unquantized. The ablation studies are conducted mainly on Llama-3.1-8B, so the contribution of each component to video generation remains unclear.

3 comments

Dear Authors,

Will you provide a rebuttal for your paper? I think you can try to make it better.

Reviewer

We thank the reviewer for their comments and for recognizing the importance of our motivation and its practical benefits. We give our response as follows:

Limited methodological novelty While CHAMP builds upon already well-established sensitivity-based quantization techniques, we respectfully believe that its contribution extends beyond a straightforward application of existing methods. Existing fine-grained methods, particularly FGMP, primarily rely on sensitivity-based selection, allowing the distribution of precision across modules and layers to emerge implicitly. This can lead to an excessive concentration of the high-precision budget, that could result in a sub-optimal solution (we showed this in Appendix B.5). Our method instead explicitly structures this distribution through hierarchical allocation, avoiding the excessive concentration effect. The effectiveness of these mechanisms is supported by our leave-one-component-out ablations in Table 12. Furthermore, CHAMP enforces matched weight–activation precision, avoiding the mismatched W4A8 and W8A4 computation paths that may arise when weights and activations are selected independently.

We therefore believe that CHAMP makes a meaningful contribution to fine-grained mixed-precision allocation by introducing a structured mechanism supported by empirical diagnostics and ablation studies. We will strengthen the presentation of these distinctions in the final manuscript to make the methodological contribution clearer.

Lack of real hardware evaluation We appreciate the reviewer’s concern regarding practical deployment efficiency. CHAMP's performance is evaluated through a controlled simulated quantization. The implementation of these techniques using actual hardware would require specialized kernels supporting the target formats, along with efficient precision conversion and execution scheduling. Developing and optimizing these kernels is a substantial engineering effort which is infeasible outside the scope of the current work. Section 5.3 explicitly identifies the efficiency analysis as theoretical.

We agree that hardware validation is an important direction as a future work. We will emphasize this limitation in the final manuscript. However, the absence of hardware implementation does not invalidate the precision-allocation and model-quality improvements of our experiments.

Missing comparisons with MicroMix and MixLLM Both methods are important related work, and both are discussed in Section 2. However, MixLLM primarily applies global mixed-precision allocation across output features using W4A8 and W8A8, whereas CHAMP considers matched W4A4/W8A8 configurations with channel-group allocation. In addition, MicroMix uses MXFP4/6/8 and specialized hardware kernels, whereas CHAMP focuses on hierarchical allocation between HiF4 and MXFP8. Therefore, directly comparing against these baselines, would introduce numerical format and granularity differences, making it difficult to isolate the effect of hierarchical allocation.

We will clarify the rationale for our baseline selection and the absence of these additional comparisons in the final manuscript.

Limited and inconsistent improvements We would like to emphasize that the amount of CHAMP’s improvements should be interpreted in the context of its intended operating regime, i.e., aggressive quantization, where a small fraction of channel-groups can only use FP8. For example, when r=0.1, CHAMP yields a 1.07% improvement over FGMP, substantially reducing the gap to full MXFP8 (57.3% of the accuracy lost). According to Table 9, it is crucial to note that at higher ratios, the benefit of mixed-precision allocation becomes less important, because more channel-groups can already receive higher precision. Finally, for video generation, our primary finding is improved frame-level fidelity relative to BF16. However, we do not claim that CHAMP improves every metric or every precision configuration.

Regarding statistical uncertainty, our evaluation follows a fixed experimental configuration, which enables direct comparisons under identical settings. Nevertheless, we agree that multiple random seeds and confidence intervals would improve the assessment of smaller performance differences.

Limited experimental coverage We appreciate the suggestion to expand experimental coverage. Nevertheless, we would like to highlight that our evaluation spans 7 language-understanding benchmarks (with Llama-3.1-8B), and two different video generation models (Wan 2.2-T2V-A14B and HunyuanVideo). Our experiments also cover multiple FP8 ratios, component ablations, and video fidelity and generation-quality measurements. We believe that these evaluations show the suitability of CHAMP in both autoregressive language models and diffusion-based video generation models.

Regarding HunyuanVideo, we clarify that the self-attention Q, K, and V tensors are kept unquantized consistently across the compared methods.

Dear authors,

Thank you for providing further explanation about my concerns. The explanation of hierarchical allocation and the differences from FGMP is helpful. However, I'd like to see the actual efficiency benefit and more evaluations about how these modules work in your method. Thus, I decided to keep my score.

Best regadrs,

Reviewer

Reviewer_hsDDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CHAMP, a hierarchical FP4/FP8 mixed-precision quantization framework. It allocates a FP8 budget across modules and layers before selecting sensitive channel-groups, using W4A4/W8A8 precision. Experiments on language and video generation models report quality improvements.

Critical strengths
  1. The paper identifies a limitation of flat global sensitivity ranking, which makes sense to me.
  2. It evaluates the proposed allocation against FGMP under matched FP8 ratios and includes component ablations on both language and video generation models.
Critical weaknesses
  1. Sensitivity-based mixed precision and budget allocation are well-established. CHAMP primarily introduces several rules for budget allocation. Their specific forms and hyperparameters are not sufficiently justified in the current version. Also, the hyperparameter-selection proxy minimizes uncovered additive sensitivity (Appendix B.2), whose unconstrained optimum is global Top-K ranking in my understanding (please correct me if that’s not the case). I understand that the authors used some constraints and won’t get to this optimum, but this is difficult to reconcile with the paper's current motivation.
  2. On Llama-3.1-8B, CHAMP improves average accuracy over FGMP by 0.44 points (Table 1). The advantage disappears or reverses at higher ratios (Appendix Table 9). Moreover, changing the hyperparameters improves accuracy from 63.19% to 63.85%, exceeding the reported 0.44-point improvement (Appendix Table 7). Although video frame-level fidelity improves, VBench gains are small or mixed (Table 3). More extensive evaluations and uncertainty estimates are needed.
Questions for authors
  1. Why is the proposed hierarchy necessary compared with simpler alternatives such as capped global ranking or minimum per-layer precision quotas?
  2. Can the authors provide broader model evaluations, uncertainty estimates, and measured inference efficiency using actual mixed-precision kernels?
1 comment

We would like to thank the reviewer for their thoughtful feedback. We give our response as follows:

Methodological novelty, allocation heuristics, and optimization proxy: We agree that sensitivity based mixed precision is a well-established research direction. Nevertheless, our framework addresses an important limitation of existing fine-grained methods, i.e., existing methods typically rely on global sensitivity ranking, which can produce sub-optimal solutions. CHAMP addresses this limitation by explicitly allocating the budget hierarchically across modules and layers. Our contribution indeed lies in explicitly separating hierarchical budget allocation from fine-grained channel-group selection. This design choice is motivated by empirical evidence rather than heuristics (Appendix B.5); We show that a global ranking leaves 42.4% of the linear module-layer pairs in HiF4 when FP8 ratio is 0.2. In addition, we show that reallocating only 4 channel-groups across modules improves the perplexity of WikiText-2.

Regarding the optimization proxy, we agree with the reviewer. However, we would like to clarify that the proxy is used as an inexpensive hyperparameter selection, not as the ultimate optimization objective of CHAMP. The actual objective is preserving downstream model quality, which is not an additive function of individual sensitivity scores due to interactions between quantization errors. We acknowledge that the relationship between the proxy and the final model-quality objective could be explained more clearly. We will clarify this distinction and the limitations in the final manuscript.

Limited improvements and hyperparameter sensitivity: We would like to emphasize that CHAMP is designed for scenarios where the high-precision budget is constrained. To show that our claim holds under these scenarios, notice that at r=0.1, CHAMP achieves 63.59% average accuracy compared with 62.52% for FGMP (an improvement of 1.07%), substantially reducing the gap to full MXFP8 (57.3% of the accuracy lost). In addition, when r=0.2, CHAMP improves the average accuracy by 0.44% and outperforms FGMP on 6/7 benchmarks.

Regarding the hyperparameter sensitivity, we clarify that we selected the reported hyperparameters using the calibration-based proxy rather than downstream benchmark accuracy (which is expensive and time-consuming to obtain). In addition, Table 7’s different best allocation cap highlights an important distinction between the effectiveness of CHAMP’s allocation strategy and the effectiveness of its hyperparameter selection proxy. Although the proxy optimization provides a cheap hyperparameter selection method, it does not always identify the optimal configuration. Notably, the higher accuracy achieved with an alternative cap indeed suggests that CHAMP’s allocation has room for further improvement via better hyperparameter selection. We acknowledge this limitation and will clarify it in the final manuscript.

Finally, for video generation, our primary finding is improved frame-level fidelity relative to BF16, rather than consistent improvements across every perceptual quality metric.

Why is the hierarchy necessary compared with simpler alternatives? The motivation for hierarchical allocation is to address sensitivity variation at multiple granularities while avoiding excessive concentration of the budget. As we showed in Appendix B.5, that a simple global ranking does not explicitly control module- or layer-level allocation, yielding an excessive concentration of the budget, that could result in a sub-optimal solution (we showed that redistributing some of the budget could be beneficial). This motivates a the CHAMP's strategy: allocating precision based on module sensitivity and size, then adjusts the allocation according to layer-level sensitivity, avoiding the excessive concentration effect. The effectiveness of these mechanisms is supported by our leave-one-component-out ablations in Table 12.

Broader evaluation, statistical uncertainty, and hardware efficiency We would like to highlight that our evaluation spans 7 language-understanding benchmarks, and two different video generation models. Our experiments also cover multiple FP8 ratios, component ablations, and video fidelity and generation-quality measurements. We believe this provides meaningful evidence that our method is effective across the evaluated language and video generation tasks, particularly in low-FP8-budget regime.

Regarding hardware efficiency, practical evaluation would require developing specialized mixed-precision kernels, that should support the target numerical formats and channel-group allocation. Such an implementation involves substantial engineering effort and computational resources, making comprehensive real-hardware benchmarking infeasible and out of the scope of this work.

We believe that our response addresses the reviewer's concerns. We welcome any follow-up discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.