acceptodds
Under review as a conference paper at ICLR 2027

REPS: Compressed Expert Replacement for Multiversion MoE Serving

Abstract

Serving expert-local customizations of a mixture-of-experts (MoE) model requires a representation that supports both local updates and shared execution. We propose Replaceable Expert Patches for Serving (REPS), which uses the same compressed records for a parent and its complete expert replacements. Two sign bytes encode each width-eight block through a complete Hadamard transform. This structure gives independent scalar assignment at fixed scale and integer reconstruction without codeword lookup. A version-aware operator selects parent or patch rows and consumes these records directly during prefill. Qwen3-30B-A3B Code updates improve target perplexity while keeping non-target and general-text degradation within 1% in three fixed-initialization data orders. On one of these exact updates, integer reconstruction reduces synchronized backend first-token time by 3.51 and 4.45 at batches one and eight relative to our lookup-table (LUT) decoder, with unchanged weights and decode. In a nine-format storage control, sharing reduces REPS weight allocation by 48.9% and 73.4% at two and four structural versions.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture-of-experts, model serving, expert versioning, efficient inference, quantization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_8yUPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies multiversion MoE serving, where customized versions of a shared backbone must be stored compactly and executed jointly. The authors propose REPS, a compressed expert representation. It encodes weights without fitting a codebook and reconstructs them without codeword lookup. Experiments show that REPS matches AQLM-level quality while reducing TTFT and weight allocation.

Critical strengths
  1. The paper addresses multiversion MoE serving under GPU memory constraints, an important problem, and solves it effectively.

  2. The evaluation is rigorous

Critical weaknesses
  1. At equal budgets, AQLM achieves lower PPL; REPS's quality advantage comes from a larger continuation budget. AQLM also has lower ITL than REPS. REPS therefore yields neither a quality nor an overall throughput advantage — its gains are limited to encoding speed and prefill TTFT
Questions for authors
  1. Could you provide an experimental or analytical comparison against ExpertWeave at the same number of versions and the same workloads?
Reviewer_mBZnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

REPS is a 2-bit weight format for MoE expert FFNs for expert-local fine-tuned use case with compressed parent weights. The quantization is based on a tile size of 1x8. Quantized values are stored as two sign bytes plus one BF16 scale per projection. Different fine-tuned versions replace four complete experts per layer, stored as records in the same format. A version-aware grouped prefill operator selects parent or patch rows per (request version, routed expert), so requests for different versions share one batch. Encoding is a FWHT followed by per-coordinate rounding (lines 209–210), and decoding can be done by XOR instead of a lookup table. In Table 1, the authors compared the quality performance of the proposed method with 8 other approaches and shows superior perplexity performance. Evaluation also shows that integer reconstruction lowers prefill TTFT. It also shows that sharing one parent reduces weight allocation by 48.9%/73.4% at 2/4 versions. A synthetic single-GPU serving study in which mixed-version batching meets both SLOs for 8-request bursts at 1–2 req/s while copies/separate batching attain 0%, and no Poisson configuration reaches 95% attainment.

Critical strengths
  1. The problem is well-defined. Serving versioned expert-replacements at scale is a realistic problem. The paper described a whole pipeline, including encoder, fixed-stride patch records, version-aware grouped kernel with CUDA-graph capture etc.
  2. The paper reports its own negative results: AQLM is better at equal training budget as mentioned in the paper: At equal budgets, AQLM has lower PPL, so Table 1 does not establish a quantization-quality advantage for REPS. For some adaptation, REPS worsens both held-out targets, while LoRA improves them.
Critical weaknesses
  1. Most comparison that favors REPS is confounded. REPS gets 8 continuation stages, 106.8M targets with C4 and PTB training text in the final stage, and ends up beating BF16 on Wiki-2 (7.20 vs 7.26), PTB (10.64 vs 12.43) and PubMed (5.45 vs 5.78). AQLM, the only trained baseline, is the authors own reimplementation at 49.152M targets, and the authors concede AQLM wins at equal budget. The remaining baselines appear to be PTQ only. The LLVQ row (5.18 bpp, worse than 4.13-bpp NF4 on every corpus) is anomalous.

  2. Memory evaluation: the 48.9%/73.4% savings are arithmetic of replacing 4/128 experts (4 copies × 9.62 GiB vs 9.62 + 3 × 0.21 GiB) and hold for any fixed-stride format. All low-bit curves overlap (REPS 63.49 vs AQLM 63.79 GiB at 256 versions). ExpertWeave, which already serves complete expert replacements over a shared base, is never compared against.

  3. For the table 1 result, BF16/NF4 run eagerly while REPS uses captured CUDA graphs?

Questions for authors
  1. Why does Table 1 show the 49M target AQLM rather than the equal-budget endpoint from S7.1?

  2. Which Table 1 baselines received any fine-tuning (QTIP and QuIP# have their own fine-tuning stages)? What is REPS vs AQLM PPL at equal continuation budget, and REPS PPL without PTB/C4 text in the final stage?

  3. For the table 1 result, BF16/NF4 run eagerly while REPS uses captured CUDA graphs?

Reviewer_3qr5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposed a new MoE model that uses expert patching and encoding plus shared parents to reduce memory pressure and increase tok/sec for settings where multiple versions of an MoE may want to be used. The authors propose using sparse expert patches that can be applied to a shared parent, allowing more expert-specific localization if experts are fine tuned for specific tasks. The authors compare against a set of compression methods for two models, and show generally better perplexity, memory, TTFT, and encoding time.

Critical strengths

This is an interesting compression method for MoEs, that is somewhat orthogonal to typical quantization methods. The results in Table 1 look promising, and this could be a good direction to explore for MoE-specific compression, particularly for small models where on device compute is highly constrained.

Critical weaknesses
  1. This paper references a missing appendix a fair amount.
  2. I would have some difficulty reproducing what the authors did, or even how what generally went into the results in Table 1. (Data, implementation, etc) As such, while the numbers look good, it is very hard to verify that it is apples-to-apples. Perplexity for REPS being substantially better than baseline in a few cases despite REPS being a compression method suggests that what is being displayed needs more context and explanation.
  3. This method is sold around domain adaptation, but the domain adaptation story is fairly weak. Ablations on how much patches add across a set of models and benchmarks would be useful, along with what happens when patches are knocked out, and varying numbers of patches.
Questions for authors
  1. Why are perplexity values in table 1 better for REPS compared to BF16 on Wiki-2, PubMed, and PTB? It feels like the uncompressed baseline model should be better. Is there tuning to those datasets? If so, can we do an apples-to-apples comparison with the BF16 model?
  2. Where is the appendix? It is referenced a few times and seems to be missing.
  3. What are the axes on Figure 3A, specifically the y-axis? If it is just the model, that is rather confusing and could use a different data viz method.
  4. How does quality and inference time/memory scale across tasks as the number of expert patches increases?
Reviewer_1xgoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

REPS supports multi-version MoE serving by sharing a compressed base model and storing only modified experts for each customized version. It introduces a unified low-bit format and a version-aware operator that selects base or replacement experts, using integer reconstruction to reduce weight reconstruction overhead during prefill.

Critical strengths

S1: The work presents a concrete design connecting independently replaceable compressed expert records with a unified prefill operator, supporting shared execution across customized model versions.

S2: The same-weight, same-decode comparison between integer reconstruction and lookup-table decoding provides useful evidence for the proposed reconstruction mechanism’s contribution to prefill performance.

Critical weaknesses

W1: The significance and generality of the target scenario are insufficiently established. The approach assumes multiple customizations that modify selected experts while preserving shared parameters. The paper provides limited evidence that this update pattern is representative of practical workloads or preferable to alternative adaptation approaches.

W2: The central insight and its distinction from prior work are difficult to identify. The presentation intertwines quantization, adaptation, weight sharing, and serving. Although related work is discussed, the additional contribution over ExpertWeave and existing structured quantization methods needs sharper explanation, including which principles remain useful beyond the particular encoding and kernel implementation.

W3: The experiments do not sufficiently establish the complete system’s advantage. Several comparisons use different training budgets and execution paths, while a direct end-to-end comparison with ExpertWeave is missing. Quality assessment relies heavily on perplexity, and the reported throughput reversal for longer outputs raises questions about how broadly the prefill improvements translate into practical serving benefits.

Questions for authors

See my weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.