- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the challenges of large expert parameter storage overhead, low computational efficiency, and high data movement costs when deploying Mixture-of-Experts (MoE) models on resource-constrained edge devices. To overcome these challenges, the authors propose shifting the traditional computation-centric inference paradigm toward a retrieval-centric approach.
Critical strengths
1.Novel Retrieval-Centric MoE Inference Paradigm. The paper introduces a retrieval-centric MoE inference framework that replaces conventional online expert computation with a Semantic-Conditioned Lookup Table (SC-LUT). This approach provides a new perspective on efficient MoE inference for resource-constrained edge devices.
2.Comprehensive System Optimization and Experimental Validation. The paper integrates NF4 quantization, a fused retrieval kernel, and asynchronous prefetching to jointly optimize storage, computation, and data movement overhead. Extensive comparisons with multiple baselines and ablation studies demonstrate the effectiveness of the proposed approach, achieving up to a 31.1× improvement in inference throughput.
Critical weaknesses
1.Limited Semantic Context Modeling. The proposed approach primarily relies on fixed bigram and trigram patterns to construct semantic conditions, which may not adequately capture long-range dependencies and dynamic contextual semantics. Therefore, its generalization capability on complex language tasks requires further investigation.
2.Scalability Concerns of SC-LUT Storage. Despite employing NF4 quantization for compression, the SC-LUT still requires substantial external storage space (9.93 GB and 6.81 GB for the two evaluated model scales, respectively). As model size and the number of semantic patterns increase, the storage overhead and offline construction costs of the lookup table may limit the scalability of the proposed approach to larger MoE models.
Questions for authors
How does RcMoE handle long-range semantic dependencies beyond bigram and trigram patterns?
How does the storage overhead of SC-LUT scale as the model size increases?
Response to Reviewer xwaM
We sincerely thank you for your thoughtful review and for recognizing the novelty and effectiveness of RcMoE. We would like to address your concerns point by point below.
1. Clarification on Semantic Context Modeling. In RcMoE, NPMI-selected bigram and trigram patterns provide local semantic conditions for Retrieval Expert lookup. Different patterns associated with the same token generate distinct semantic-conditioned retrieval representations and Retrieval Expert outputs to distinguish different local semantic contexts.
Clarification on Long-Range Semantic Dependency. RcMoE preserves and utilizes the Transformer contextual hidden state through three original MoE pathways to handle long-range dependencies and dynamic contextual semantics. Specifically, the residual connection carries the contextual hidden state forward; the shared expert performs online computation on the hidden state; and the router computes aggregation weights from the contextual hidden state, controlling Retrieval Expert contributions to the final MoE representation.
Validation on Long-Context Tasks. By preserving contextual information through these pathways and using local patterns to refine Retrieval Expert outputs, RcMoE maintains competitive long-context performance. Specifically, we evaluate RcMoE on eight LongBench tasks under a zero-shot protocol with an 8192-token input. As reported in Table 2, RcMoE achieves an average LongBench score of 11.44 at the 1.7B scale, outperforming MoLE (10.85). Detailed results are provided in Appendix D (Table 9).
Overall, RcMoE preserves contextual modeling through the original MoE pathways while using local semantic patterns to condition Retrieval Expert lookup, achieving competitive long-context performance.
2. Concerns regarding the Scalability of SC-LUT Storage. SC-LUT exhibits predictable storage scaling with vocabulary sizes and MoE dimensions. External storage avoids keeping the full table in device memory. We discuss its scalability from two perspectives:
Storage Scaling with Model and Vocabulary Sizes. As described in Section 3.3, SC-LUT scales as (∣V∣+∣M∣+1)×L×Nr×d, where ∣V∣ and ∣M∣ denote token and semantic pattern vocabulary sizes, L the number of MoE layers, Nr the number of Retrieval Experts per layer, and d the hidden dimension. Storing token and pattern residual entries separately yields additive scaling with ∣V∣ and ∣M∣, rather than multiplicative scaling, and linear scaling with L, Nr, and d. NF4 quantization reduces storage by approximately 4×, while kernel fusion and asynchronous prefetching mitigate I/O overhead.
Practicality of External Storage Footprint and Offline Construction. SC-LUT is constructed once offline, outside the online inference process. In a server-to-edge deployment, it can be generated and compressed on the server before being transferred to external storage for edge inference, avoiding construction overhead during online deployment. The NF4 SC-LUTs for RcMoE-1.7B and RcMoE-360M occupy 6.81 GB and 9.93 GB, respectively, and are stored on external SSDs for on-demand access. Unlike conventional in-memory MoE execution, RcMoE stores precomputed Retrieval Expert outputs in external storage and replaces online expert computation with lookups.
Validation on Scalability. Appendix E (Tables 14–16) evaluates the impact of ∣M∣ on model quality and storage, and Nr on storage, construction time, and inference latency. We briefly summarize the results in the following table.
SC-LUT Scalability with Retrieval Expert Count.
As shown above, storage and offline construction time grow near-linearly with Nr, with construction taking 499.0 s at Nr=32 on an RTX 4090. Using the RcMoE-1.7B configuration, increasing Nr from 3 to 128 corresponds to an equivalent total parameter capacity of approximately 154.9B. Across this range, NF4 SC-LUT storage increases from 6.81 to 290.48 GB, approximately 4× smaller than BF16. On an 8 GB Jetson Orin Nano, RcMoE latency increases from 4.60 to 18.22 s, compared with 4.61 to 32.70 s for the original BF16 MoLE baseline, achieving up to a 1.79× speedup.
We have clarified long-range contextual modeling in Section 3.2 (Retrieval Expert Modeling, Lines 192–197) and expanded the scalability analysis in Appendix E (Tables 14–16).