acceptodds
Withdrawn from ICLR 2027

Federation of Experts: Communication Efficient Distributed Inference for Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts. Between clusters, a sum synchronizes the post-attention residuals, which then drives routing and dispatch for the next MoE block. In a single-node setting, FoE completely eliminates all-to-all communication as all experts within a group are contained on the same GPU. In multi-node settings, FoE confines all-to-all communication to the intra-node fabric, thus significantly reducing communication overhead. An implementation of FoE finds that on LongBench, FoE significantly improves inference throughput and latency in both single-node and multi-node settings, reducing the end-to-end forward-pass latency by up to 5.2x, TTFT by 3.62x, and TPOT by 1.95x. It does so while achieving comparable generation quality to a mixture of experts model of the same size and training configuration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Mixture of Experts, Communication, Distributed Inference, Efficiency

Related papers

Discussion (0)

Sign in to comment.