acceptodds
Under review as a conference paper at ICLR 2027

MoE-PACT: Confidential MoE Inference via Searched Bit Partitioning and Optimal Clustering of Expert Tensors

Abstract

Mixture-of-Experts (MoE) architectures are widely deployed thanks to their low training and inference compute cost, but their large parameter counts impose severe memory and host-to-device (H2D) transfer bottlenecks. These bottlenecks are most severe on cloud server nodes whose H2D bandwidth is restricted by NVIDIA Confidential Computing (CC) mode. Existing solutions for memory and bandwidth overhead come at the price of degraded accuracy, or of expensive retraining or auxiliary memory for mixed-precision weight storage. We present MoE-PACT, an MoE inference system that reduces memory footprint and H2D traffic under Confidential Computing. We first systematically analyze NVIDIA CC mode to characterize the PCIe bottlenecks introduced by encryption and decryption, achieving a bandwidth gain through vector AES and multiprocess transfers. We then propose an exhaustive per-tensor search over base-delta bit partitions with optimal 1D -means clustering, and a fused GPU kernel that performs bit-unpacking, lookup-table (LUT) base-delta reconstruction, and dequantization on the fly. Against the strongest baselines under CC, MoE-PACT stores 25% fewer expert bytes and cuts H2D traffic by up to half, accelerating decoding by up to at batch size one and at large batches, with negligible accuracy degradation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture of Experts, Confidential Computing, Offloading

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BGPtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoE-PACT, a system for efficient MoE inference under Confidential Computing. It combines optimized encrypted CPU–GPU transfers, searched base-delta bit partitioning, and fused GPU kernels for weight reconstruction. Experiments on three MoE models show reduced memory usage and improved decoding throughput with limited accuracy degradation.

Critical strengths

1.The paper addresses a practical bottleneck in confidential MoE inference. The measurements on actual Confidential Computing hardware make the results more convincing. 2.The system integrates several useful optimizations, with promising throughput improvements across different MoE models and batch sizes.

Critical weaknesses

1.The contribution of the bit-partition search is not clearly isolated. Section 4 shows that nearly all tensors select the same partition, which is then fixed in the implementation. A comparison with a fixed bit split or a simpler 6-bit quantization baseline would help establish whether the search and clustering are actually necessary. 2.The baseline comparisons need stronger support. EARTH and MoE-APEX are reimplemented, and EARTH's accelerator design is not included. Moreover, the batch-size-one CC measurements use single, short runs for Mixtral. This makes it difficult to assess the reliability of the reported speedups. 3.The confidentiality implications of routing-dependent transfers are not sufficiently evaluated. At batch size one, delta fetching exposes transfer sizes and timing to the untrusted host, as the authors acknowledge. The paper does not quantify this leakage or evaluate the performance cost of preventing it.

Questions for authors

1.How much of the performance gain remains compared with a fixed 6-bit quantization scheme using the same caching policy and GPU kernels? 2.Can the authors provide repeated CC throughput measurements and evaluate the security-performance trade-off of fixed-size delta fetching?

3 comments

Thank you very much for reviewing our paper and for your comments on how to improve it. We have tried to address them as best we can.

EARTH and MoE-APEX are our reimplementations because their authors have not released their code.

The batch-size-one measurements in CC mode were run once because they take significantly more time. They also show less variability in decode latency, because H2D transfers are an even larger bottleneck under CC. We therefore limited them to a shorter decode length and a single run. In normal mode, where we ran three prompts per configuration, throughput varied by a median of about 3% on the Mixtral models.

The security implications will be studied in future work, as this paper focuses on the performance bottlenecks of NVIDIA CC mode and on designing a system that achieves higher performance under it.

Regarding the bit-partition search, every other partition has a larger quantization error, which is the objective the search minimizes. Even the second-best partition has a median of 15% (Mixtral-8×7B), 12% (Qwen3), and 6% (Mixtral-8×22B) higher error per tensor than the selected one. Larger error results in higher perplexity: on Mixtral-8×7B the conventional high/low split used by EARTH gives a perplexity of 4.2450, against 3.8530 for our searched split.

We have uploaded the updated PDF, with all changes shown in blue. If you have further comments, we would be happy to hear them and address them as best we can.

Thank you for the response and the additional details. The quantization error and perplexity comparison against the conventional high/low split clearly demonstrates the value of the searched bit partitioning, and your clarifications regarding the CC measurement variance and paper scope make sense. I will maintain my positive rating.

Thank you for your assistance in improving our paper.

Reviewer_jSKGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

There is a bottleneck in model deployment which happens due to the NVIDIA Confidential Computing mode. In this work, a new method is proposed to help with this bottleneck called MoE-PACT. MoE-PACT is a MoE inference system that reduces the memory footprint of mixture-of-experts models. Empirically the framework is demonstrated using two Mixtral models and a Qwen3 model.

Critical strengths

Full disclosure this paper is not my area of expertise. I do like that the authors have made a commitment in the paper to open source their work. I think for such a complicated hardware/software setup, is great to have some commitment to reproducibility.

Critical weaknesses

There are two major issues with the paper.

Issue #1: Scope. I am not sure this paper is appropriate for ICLR. While it is true that this paper is about machine learning, it is heavily focused on the computation and architecture speed up side of things. More specifically, this work is focused on one NVIDIA specific technology (the NVIDIA Confidential Computing). I am not sure if that scope is appropriate for a research paper. I will say that I am by no means an expert in this domain, so I am open to hearing other opinions on this issue.

Issue #2: Poor writing quality. A lot of the paper is hard to understand. For example, when I was trying to write a summary of the paper and skimming through its contents this issue became very clear. I could not figure out what dataset the paper used. Are the datasets written in the abstract? No. Are the datasets written in the experimental setup in section 6.1? No. As far as I can tell the datasets are not directly mentioned in the conclusion either. Such a major oversight in a top tier submission is hard to understand. Maybe the data is irrelevant in this application? But I would find that hard to believe. Likewise, as someone who is not a domain expert, I found figures 3 and 4 almost incomprehensible. In Figure 6 it says in the caption, “Bars as in Figure 5.”…what does that even mean? Lastly, in the experimental section there is no big overview to try and explain how the results are organized. It made interpreting things much more difficult than necessary. It would be great if the authors could include one paragraph at the start of the experimental section that breaks down how the results in the rest of this section are going to be disseminated. Overall, I think in terms of presentation the authors could do a much better job organizing the paper.

Questions for authors
  1. Can you address my question about the scope of your work being appropriate and not too “single technology” focused?

  2. Can you fix the paper’s presentation in terms of writing and figures based on what I mentioned in “Issue #2” in my weaknesses section?

2 comments

Thank you very much for reviewing our work and for your suggestions on how to improve it. We have tried to address your comments as best we can.

Regarding Issue #1 (scope), we note that ICLR 2027 has the primary area "infrastructure, software libraries, hardware, systems, etc.", under which we submitted this paper. Our paper studies NVIDIA Confidential Computing (CC), a feature of Hopper and Blackwell, NVIDIA's two most recent generations of data-center GPUs, which are widely deployed in cloud environments. We investigate the performance bottlenecks of CC on an H200 GPU and propose and evaluate an inference system designed for it. Our design also shows significant improvements in normal mode, without CC, so it applies to standard deployments as well. Following your comment, we now state more clearly in the conclusion that the improvements are not restricted to confidential deployments.

Regarding Issue #2 (presentation), we added an overview at the start of the evaluation section and now clearly explain what each experiment measures, how it is run, and which datasets are used. This information was previously mostly in the appendix. We also clarified the Figure 6 caption, which now describes the bars directly instead of referring to Figure 5. We greatly appreciate you pointing out these problems and helping us address them.

We have uploaded the updated PDF, with all changes shown in blue. If you have further comments, we would be happy to hear them and address them as best we can.

Hello,

Thank you again for your time and thoughtful feedback. As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. If anything remains unclear or you would like further details, we would be happy to clarify.

Reviewer_F7A2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses MoE inference under private NVIDIA Confidential Computing mode. They first find out using CPU vector AES can improve the communication speed over baseline and that can be further improved by 4 concurrent transfers. Then they further improve the transfer speed by decomposing weights per tensor to base delta. Base is 1 bit sign + clustering next 4 bits to 3 (using min quantization error), delta is clustering 3 bits to 2. Then they show transfer speeds based cache expert ratio being held in HBM and show improvement of decoding speed on Mixtral and Qwen

Critical strengths
  • Finding out vector AES + parallelization over the standard approach on NVIDIA CC is clever
  • Clustering 4 bits to 3 for base and 3 to 2 for delta is a very interesting approach
  • The performance numbers look very good
Critical weaknesses

This paper even though looks interesting has lost the main plot, which it has to show the MoE inference system improves the accuracy vs latency gap. They have only focused on latency and just one number on GSM8k and TruthfulQA is reported. Just focusing on speed is not important because you might as well not transfer anything or have a faulty compression and be super fast. So at least these two baseline methods should be compared with them:

Also the evaluations on speed has a critical issue: they have not mentioned which dataset, how many tokens, etc this methods have been used?

Questions for authors

Look at weaknesses section please.

2 comments

Thank you very much for reviewing our paper and helping us address the issues you identified.

First, we acknowledge that we did not include enough information about the benchmarks, and most of it was in the appendix. We have now added which datasets and configurations each benchmark uses, including the number of generated tokens.

We also thank you for pointing out that accuracy and throughput were not clearly compared together. We updated the evaluation section to show that, at batch size one, all systems are calibrated to the same perplexity and reach the same downstream task accuracy, and their throughputs are compared in that setting. Our system therefore achieves higher throughput than the existing batch-size-one systems at equal accuracy, while transferring less data.

None of the batch-size-one systems runs at full INT8 accuracy: each keeps two precision levels and uses a threshold to trade accuracy for transferred bytes. EARTH mixes 4- and 8-bit weights, MoE-APEX mixes 2- and 8-bit weights, and MoE-PACT mixes 4- and 6-bit weights. Even though our highest precision is only 6 bits, we match their accuracy and still transfer fewer bytes per token (5–26% fewer than EARTH and 30–67% fewer than MoE-APEX), which is where our speedup comes from.

This shows that a more complex weight encoding can improve throughput. EARTH explicitly chooses a simple encoding to avoid reconstruction overhead; our fused kernel and memory layout make the more complex encoding cheap enough to decode. At large batch sizes, our system achieves higher throughput than INT8 at the cost of a small perplexity increase (+0.30% on Mixtral-8×7B and +0.25% on Qwen3), which is expected since it uses only 6 bits. The perplexity and downstream accuracy numbers are in the evaluation section and the table in the appendix.

Regarding FP8 and NVFP4: FP8 weights take 8 bits, the same as our INT8 baseline, so they would transfer the same number of bytes. Since decoding in our setting is bound by H2D transfers, FP8 would reach about the same throughput as INT8. NVFP4 is a Blackwell format and is not supported by the tensor cores of our H200 (Hopper) GPU.

We have uploaded the updated PDF, with all changes shown in blue. If you have further comments, we would be happy to hear them and address them as best we can.

Hello,

Thank you again for your time and thoughtful feedback. As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. If anything remains unclear or you would like further details, we would be happy to clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.