acceptodds
Under review as a conference paper at ICLR 2027

TAP-MoE: Task-Aware Profiling for Memory-Efficient Mixture-of-Experts Deployment

Abstract

Mixture-of-Experts (MoE) models scale capacity through sparse activation, yet their large expert pools impose substantial storage and weight-transfer costs that hinder efficient edge deployment. When experts cannot all reside in accelerator memory, deployment must address both compression-induced quality loss and expert-loading overhead. We propose TAP-MoE, a task-aware framework that connects fine-grained expert behavior to task-level deployment decisions. It captures expert activation and sensitivity patterns through shared class-conditional statistics. Reweighting these statistics by the target task's token-class distribution yields a workload-specific profile. This profile guides sensitivity-aware mixed-precision allocation, task-conditioned expert prefetching and residency, and risk-aware output compensation. Precision allocation determines both expert sizes and quantization residuals, linking memory planning and error correction to the same task profile. The framework separates offline calibration from online execution. It preserves the pretrained routing rule and shared experts without model retraining. Our analysis bounds profile-transfer error and derives optimality conditions for a continuous precision-allocation surrogate. We evaluate TAP-MoE on OLMoE-1B-7B, Qwen1.5-MoE-A2.7B, and DeepSeek-MoE-16B-Base across three task domains under matched storage and runtime memory budgets. Compared with BF16, TAP-MoE reduces routed-expert weight storage by 75% while maintaining near-BF16 quality. Against the strongest evaluated SOTA baseline for each metric, TAP-MoE achieves 1.0-5.0% relative reductions in perplexity and 1.25-1.75× decoding speedups.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, Task-Aware Profiling, Mixed-Precision Quantization, Expert Offloading, Memory-Efficient Deployment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_mb9cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TAP-MoE, a task-aware framework for MoE compression and inference under memory constraints. Its core idea is to analyse how different token categories relate to expert-level statistics, then use the token-category distribution of a target workload to estimate expert sensitivities and formulate a precision-allocation optimisation problem. The paper provides theoretical bounds for the statistical transfer and an optimisation analysis of the resulting allocation. It further introduces expert access and prefetching strategies, together with output compensation for quantisation errors (Section C).

Experiments on three MoE models and three workloads (general text, code, and mathematics) show improved perplexity over the evaluated quantisation baselines, alongside reported decoding speedups of 1.25–1.75× on an NVIDIA RTX A6000.

Critical strengths
  1. Well-motivated insight and coherent analysis. The paper formulates task-aware expert compression through category-conditioned statistics and workload-specific sensitivity estimation. The resulting optimisation objective has a clear mathematical formulation, and the statistical transfer bounds and optimisation analysis provide a coherent theoretical foundation.

  2. Practical deployment extensions. Beyond precision allocation, the paper introduces expert access and prefetching strategies and output compensation for quantisation errors (Section C). These components address practical deployment requirements and are evaluated through experiments and ablations.

Critical weaknesses

none

Questions for authors

Estimate in different context:The method estimates target-task expert sensitivities by reweighting category-conditioned gradient statistics collected from shared calibration data. However, tokens from the same category may exhibit different gradients and routing behaviour in different contexts. Have the authors evaluated whether the estimated expert sensitivities remain stable when the token-category proportions are similar but the within-category context distribution changes? Such an evaluation would help clarify the reliability and scope of the proposed statistical transfer.

Reviewer_wH79OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a novel method TAP-MoE, which introduces task-level mixed precision of experts (i.e. lower expert accuracy if the expert is not relevant for a task), combined with task-aware prefetching and (cache) residency, as well as risk-aware quantization error output compensation. The authors show speedups on OLMoE, Qwen1.5 and DeepSeek on an NVIDIA RTX A6000 GPU.

Critical strengths

The authors show that their method yields the expected improvements, yielding better token generation rate vs accuracy trade-offs than other deployment strategies.

Critical weaknesses
  1. Central to the paper is the concept of a 'task' or a 'token class'. However, these concepts are not defined anywhere, making it hard to understand what it is exactly the authors aim to achieve.

  2. More generally the paper uses a lot of specific terms that are not introduced without further explanation. E.g., in the contributions on lines 66-79, 'token-class statistics', 'target distribution', 'profile-transfer bound', 'continuous allocation analysis', 'class-conditional co-activation', 'target-workload reuse', 'activation-weighted task means'. Being nonstandard, these concepts should be defined clearly upon first usage, or replaced by more standard usage where applicable.

  3. The paper introduces some notation without explanation. E.g., in equation 3, what do the hats indicate? In equation 4 what does the difference f - \hat{f} indicate? etc.

  4. The paper compares to very few relevant baselines. Several baselines introduced in literature as comparable to the paper's approach (e.g. AWQ, OmniQuant, MxMoE, QMoE, MoEQuant for quantization/compression; eMoE, ST-MoE, EARTH, MoE-OPU, PagedWeight, ADEPT, DyMoE for expert offloading and access; KBVQ-MoE for quantization-error correction) are omitted in

  5. It is unclear what the benefit of adapting to tasks is vs other dynamic compression methods. This point is taken as a given (introduction line 49: "Aggregating each quantity over a mixed calibration pool can obscure differences betweent tasks" -- why only look at tasks?), but not motivated. Since the authors do not define a task, other than briefly on line 120 as a "[inducing] a distribution over input documents and token positions", without references to prior literature, this choice gives the impression of being arbitrary.

5.a. Why 14 unique tasks? Where does this number come from?

  1. Additionally, the method has many bells and whistles, leading to a complicated set of offline and online stages to run through before claimed benefits can be realized.

  2. The authors should compare their approach to other works that dynamically compress MoEs, e.g., SliceMoE [1], which dynamically chooses a bitwidth for each token-expert, or instruction-following pruning [2], which, based on an input prompt decides which subnetworks to activate.

  3. The author's error compensation is somewhat reminiscent of The authors should add a comparison to this paper in the literature and the results.

  4. The authors should plot the various trade-offs on tok/s incurred by the proposed method and the different baselines the authors compare against

I hope the authors can perform a significant rewrite of the paper, focusing strongly on clarity (particularly in the method exposition), as well as better comparison to baseline methods. In the current stage I consider it a "reject", but the resuls seem promising.

[1] SliceMoE [2] Instruction-following pruning [3] MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE

Questions for authors
  1. Could you clarify the non-standard jargon mentioned above?
  2. How does the performance of the method change at longer generation tasks, e.g., document summarization? Deployment decisions are made offline, implying that a 'mistake' in deployment decision is non-recoverable.
  3. How does the performance change with different GPUs? I would expect higher FLOPs or higher shared memory bandwidth to both signficantly affect the measured KPIs
Reviewer_68k6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TAP-MoE, a task-aware profiling framework for memory-efficient MoE deployment. It leverages token-class statistics and task-specific token distributions to guide expert quantization, prefetching, and output compensation. Experiments across multiple MoE models and task domains demonstrate reduced expert storage, improved perplexity over evaluated baselines, and faster decoding.

Critical strengths
  1. The paper presents a unified framework that integrates expert quantization, task-aware prefetching, and output compensation, addressing multiple aspects of memory-efficient MoE deployment.
  2. The proposed task-aware profiling strategy is technically reasonable and provides a practical way to exploit differences in expert usage across tasks.
  3. The experimental evaluation is relatively comprehensive, covering multiple MoE models, task domains, and evaluation metrics, with useful ablation studies.
Critical weaknesses
  1. TAP-MoE estimates task-specific expert statistics by reweighting class-conditional statistics using the target task's token-class distribution. This relies on the assumption that expert behavior within each token class transfers across workloads. Although the paper provides a theoretical transfer-error analysis, experiments on only General, Code, and Math workloads provide limited evidence under more substantial distribution shifts.
  2. The 14 token classes are primarily based on token character patterns, which may overlook contextual differences. For example, alphabetic tokens in natural language, code identifiers, and mathematical expressions may exhibit different expert-routing patterns despite belonging to the same class. The paper does not evaluate whether finer-grained or context-aware classification could improve profiling accuracy.
  3. Although the paper reports peak GPU memory usage, a more detailed breakdown across routed-expert weights, shared parameters, and KV cache would help clarify how the reduction in routed-expert weight storage translates into overall memory savings in practical deployment.
Questions for authors

Please refer to the weaknesses above.

Reviewer_DzNwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TAP-MoE targets inference for mixture-of-experts (MoE) models whose routed experts do not fit in accelerator memory, so experts live in host memory and are copied to the GPU on demand. The paper builds a task-conditioned expert profile. Each token is mapped to one of 14 character-based classes, and four per-class statistics are collected once on a shared calibration pool: each expert's selection probability, a gradient-energy sensitivity (the per-token squared gradient of the expert's weights, including its routing weight), the mean input to the expert's down projection, and adjacent-layer co-activation counts. Reweighting these statistics by a target task's class distribution gives the profile without new gradient passes. The profile drives three offline decisions:

  • Task-aware mixed precision. Bit widths from a {3, 4, 6, 8} palette are allocated under a storage budget by minimizing Σj wj · 2−2bj, where wj is the normalized task sensitivity raised to a contrast power ρ. Theorem 2 gives the continuous optimum; a greedy search builds candidate maps, and full-model calibration NLL picks one.
  • Task-conditioned prefetching and residency. Once layer l−1 has routed, layer-l experts are ranked by smoothed, class-conditioned co-activation counts and admitted by byte cost. A knapsack picks a base preload set, and a request-local cache keeps packed or decoded weights.
  • Risk-aware output compensation. Each expert adds a constant vector ΔWj μj (the mean down-projection quantization error over the inputs that reach it), shrunk toward a pooled version. Task, pooled or no correction is chosen offline using paired confidence bounds.

The router, shared experts and expert count are unchanged, and nothing is retrained. Theorem 1 splits profile-transfer error into class-mix estimation, per-class estimation and within-class (conditional) shift. The evaluation uses OLMoE-1B-7B, Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base on general text (WikiText-2), code (MBPP) and math (MATH). It runs at a 4-bit routed-expert average on one RTX A6000 with host offloading. The paper reports 1.0–5.0% lower perplexity than the best matched-storage baseline, 1.25–1.75× faster decoding than eMoE, and gains over GPTQ 4-bit on HumanEval, GSM8K and MMLU.

Critical strengths
  • Clean architectural co-design across compression and execution. Using the offline bit-allocation map bτ as a structural bridge—simultaneously defining byte footprints sj for knapsack/prefetch admission and weight residuals ΔWj for output bias correction—is well-motivated and avoids treating quantization, caching, and error compensation as isolated heuristics.

  • Zero-retraining, zero-router-modification deployment contract. Preserving native top-K routing and BF16 shared experts avoids expert-dropping quality collapse and makes access-policy changes strictly output-invariant for a fixed quantized checkpoint (Proposition A.8), allowing clean isolation of systems latency from quantization error.

Critical weaknesses

The headline numbers follow exact ratios

  • Table 2: every speedup over eMoE is a multiple of 0.05 and every TTFT multiple of 10
  • Table D.3: the correction cost ms/token is 0.150, 0.250 or 0.350 ms/token, also multiples of 0.05.

The speedups are not attributable to the method, and baseline implementations are undocumented.

  • Table D.4: with no cache and no prefetching, TAP-MoE is already 1.22–1.55× faster than eMoE (OLMoE general: 58.39 vs. 90.71 ms/token). Caching and prefetching add only 1.5–22%.
  • At the same 4-bit average, GPTQ 4-bit takes 156.20 ms/token versus 58.39 for cache-free TAP-MoE, a 2.7× gap. That points to different runtimes or kernels, not to the proposed policies.
  • The paper doesn't say how the baselines were run: official code or reimplementation, kernels, transfer overlap or memory budgets. EARTH (an accelerator design) and MoE-OPU (an FPGA overlay) are reported at 584–1,170 ms/token in the A6000 setup. These are presumably software emulations, which say little about the original systems.
  • The paper picks a different "strongest baseline" for each metric. Speed is compared against eMoE, whose perplexity is far worse (10.65 vs. 7.88 on OLMoE general). Quality is compared against GPTQ 4-bit and EdgeMoE, which are 1.5–2.9× slower than TAP-MoE in these tables.

Weak evidence for the token-class premise.

  • The choice of 14 hand-designed classes is never ablated.
  • Table D.1: the pooled global routing predictor does worse than a uniform guess in 5 of 9 settings. A uniform guess scores 6.00 bits for 64 experts and 5.91 for 60, while OLMoE general scores 6.414. The claimed 0.40–0.78-bit class-information gain is therefore measured against a weak baseline.
  • Within-class shift is large. Same-source class fits beat pooled class fits by 0.04–0.49 bits. On math, that is 75–97% of the class-conditioning gain. On OLMoE math, a same-source global predictor (5.654 bits) beats the pooled class predictor (5.726).
  • Direct target-specific estimation is worse than synthesis in every setting (Table D.2), and the paper doesn't explain why. If this reflects small target calibration sets, the paper should say so and control for it.
Questions for authors
  • Please share raw logs, scripts and code for Tables 1–2, C.3–C.8, and D.2–D.5. Why is every speedup over eMoE is a multiple of 0.05 and every TTFT multiple of 10?

  • Baseline implementations and uncached runtime gap: How were Pre-gated MoE (which requires trained pre-gates unavailable for these frozen checkpoints), EARTH (an ASIC), and MoE-OPU (an FPGA processor) executed on an RTX A6000? Why is TAP (No cache) 2.68× faster than GPTQ4 (58.39 vs. 156.20 ms/token on OLMoE) at the same 4-bit weight budget before any caching or prefetching is enabled?

  • Is Figure 2 computed from real data? Which tokenizer and corpus produced its token shares?

  • Sensitivity scale invariance and SwiGLU input drift: Why does Eq. (5) omit each expert's squared weight quantization scale κj from Eq. (A.13), leaving γj sensitive to function-preserving SwiGLU rescaling?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.