acceptodds
Under review as a conference paper at ICLR 2027

RcMoE: Semantic-Conditioned Retrieval for Efficient MoE Inference on Commodity Edge Devices

Abstract

Mixture-of-Experts (MoE)-based large language models (LLMs) leverage sparse activation to increase model capacity without proportionally increasing per-token computation. However, their large parameter footprint and storage demands hinder deployment on resource-constrained edge devices. Existing on-device MoE approaches remain largely computation-centric, limiting the efficiency benefits of sparse activation under tight edge resource constraints. Although lookup-based MoE methods can replace expert computation with precomputed expert outputs, existing token-indexed schemes limit retrieval expressiveness and shift the bottleneck to storage overhead, fragmented execution, and data movement. To address these challenges, we propose RcMoE, a retrieval-centric on-device MoE inference framework that constructs a semantic-conditioned lookup table (SC-LUT) of precomputed expert outputs, thereby improving retrieval expressiveness. RcMoE further develops a lightweight retrieval execution pipeline with SC-LUT compression, a fused retrieval kernel, and asynchronous prefetching to reduce storage footprint, execution fragmentation, and data movement overhead. Experiments on commodity edge devices demonstrate that RcMoE achieves up to 30.6 the throughput of DMoE while maintaining competitive accuracy under edge resource constraints.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
On-Device Inference, Mixture-of-Experts, Large Language Model Serving, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xwaMOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the challenges of large expert parameter storage overhead, low computational efficiency, and high data movement costs when deploying Mixture-of-Experts (MoE) models on resource-constrained edge devices. To overcome these challenges, the authors propose shifting the traditional computation-centric inference paradigm toward a retrieval-centric approach.

Critical strengths

1.Novel Retrieval-Centric MoE Inference Paradigm. The paper introduces a retrieval-centric MoE inference framework that replaces conventional online expert computation with a Semantic-Conditioned Lookup Table (SC-LUT). This approach provides a new perspective on efficient MoE inference for resource-constrained edge devices.

2.Comprehensive System Optimization and Experimental Validation. The paper integrates NF4 quantization, a fused retrieval kernel, and asynchronous prefetching to jointly optimize storage, computation, and data movement overhead. Extensive comparisons with multiple baselines and ablation studies demonstrate the effectiveness of the proposed approach, achieving up to a 31.1× improvement in inference throughput.

Critical weaknesses

1.Limited Semantic Context Modeling. The proposed approach primarily relies on fixed bigram and trigram patterns to construct semantic conditions, which may not adequately capture long-range dependencies and dynamic contextual semantics. Therefore, its generalization capability on complex language tasks requires further investigation.

2.Scalability Concerns of SC-LUT Storage. Despite employing NF4 quantization for compression, the SC-LUT still requires substantial external storage space (9.93 GB and 6.81 GB for the two evaluated model scales, respectively). As model size and the number of semantic patterns increase, the storage overhead and offline construction costs of the lookup table may limit the scalability of the proposed approach to larger MoE models.

Questions for authors

How does RcMoE handle long-range semantic dependencies beyond bigram and trigram patterns?

How does the storage overhead of SC-LUT scale as the model size increases?

1 comment
AuthorsOpenReview ↗

Response to Reviewer xwaM

We sincerely thank you for your thoughtful review and for recognizing the novelty and effectiveness of RcMoE. We would like to address your concerns point by point below.

1. Clarification on Semantic Context Modeling. In RcMoE, NPMI-selected bigram and trigram patterns provide local semantic conditions for Retrieval Expert lookup. Different patterns associated with the same token generate distinct semantic-conditioned retrieval representations and Retrieval Expert outputs to distinguish different local semantic contexts.

Clarification on Long-Range Semantic Dependency. RcMoE preserves and utilizes the Transformer contextual hidden state through three original MoE pathways to handle long-range dependencies and dynamic contextual semantics. Specifically, the residual connection carries the contextual hidden state forward; the shared expert performs online computation on the hidden state; and the router computes aggregation weights from the contextual hidden state, controlling Retrieval Expert contributions to the final MoE representation.

Validation on Long-Context Tasks. By preserving contextual information through these pathways and using local patterns to refine Retrieval Expert outputs, RcMoE maintains competitive long-context performance. Specifically, we evaluate RcMoE on eight LongBench tasks under a zero-shot protocol with an 8192-token input. As reported in Table 2, RcMoE achieves an average LongBench score of 11.44 at the 1.7B scale, outperforming MoLE (10.85). Detailed results are provided in Appendix D (Table 9).

Overall, RcMoE preserves contextual modeling through the original MoE pathways while using local semantic patterns to condition Retrieval Expert lookup, achieving competitive long-context performance.

2. Concerns regarding the Scalability of SC-LUT Storage. SC-LUT exhibits predictable storage scaling with vocabulary sizes and MoE dimensions. External storage avoids keeping the full table in device memory. We discuss its scalability from two perspectives:

  • Storage Scaling with Model and Vocabulary Sizes. As described in Section 3.3, SC-LUT scales as (∣V∣+∣M∣+1)×L×Nr×d(|V| + |M| + 1) \times L \times N_r \times d, where ∣V∣|V| and ∣M∣|M| denote token and semantic pattern vocabulary sizes, LL the number of MoE layers, NrN_r the number of Retrieval Experts per layer, and dd the hidden dimension. Storing token and pattern residual entries separately yields additive scaling with ∣V∣|V| and ∣M∣|M|, rather than multiplicative scaling, and linear scaling with LL, NrN_r, and dd. NF4 quantization reduces storage by approximately 4×4\times, while kernel fusion and asynchronous prefetching mitigate I/O overhead.

  • Practicality of External Storage Footprint and Offline Construction. SC-LUT is constructed once offline, outside the online inference process. In a server-to-edge deployment, it can be generated and compressed on the server before being transferred to external storage for edge inference, avoiding construction overhead during online deployment. The NF4 SC-LUTs for RcMoE-1.7B and RcMoE-360M occupy 6.81 GB and 9.93 GB, respectively, and are stored on external SSDs for on-demand access. Unlike conventional in-memory MoE execution, RcMoE stores precomputed Retrieval Expert outputs in external storage and replaces online expert computation with lookups.

Validation on Scalability. Appendix E (Tables 14–16) evaluates the impact of ∣M∣|M| on model quality and storage, and NrN_r on storage, construction time, and inference latency. We briefly summarize the results in the following table.

SC-LUT Scalability with Retrieval Expert Count.

NrN_rBF16 (GB)NF4 (GB)Construction (s)RcMoE (s)MoLE (s)
327.236.8144.04.604.61
872.6218.16117.85.225.65
16145.2436.31234.46.087.46
32290.4872.62499.07.8111.06
64580.97145.24913.611.2918.28
1281161.94290.481642.318.2232.70

As shown above, storage and offline construction time grow near-linearly with NrN_r, with construction taking 499.0 s at Nr=32N_r=32 on an RTX 4090. Using the RcMoE-1.7B configuration, increasing NrN_r from 3 to 128 corresponds to an equivalent total parameter capacity of approximately 154.9B. Across this range, NF4 SC-LUT storage increases from 6.81 to 290.48 GB, approximately 4× smaller than BF16. On an 8 GB Jetson Orin Nano, RcMoE latency increases from 4.60 to 18.22 s, compared with 4.61 to 32.70 s for the original BF16 MoLE baseline, achieving up to a 1.79×1.79\times speedup.

We have clarified long-range contextual modeling in Section 3.2 (Retrieval Expert Modeling, Lines 192–197) and expanded the scalability analysis in Appendix E (Tables 14–16).

Reviewer_ND12OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to make memory-constrained devices run MoE more efficiently. The authors trained a class of experts that can calculate outputs in advance, storing the results corresponding to individual tokens and common phrases in tables, and querying the tables during inference. Routing and shared experts that rely on complete context still perform online calculations. Subsequently, they reduced storage and execution costs through compression of tables, fusion of computations, and prefetching, and compared speed and answer quality on edge devices.

Critical strengths

S1. The design stores the parts that can be looked up in tables, while retaining online calculations that depend on the context. This design is quite clear. Phrase results are saved using the difference from the output for a single token, which is also convenient for combining during runtime.

S2. The compression experiments not only report how much space is saved, but also report that compression alone slows down the speed. The subsequent execution optimization can compensate for most of the costs, and this comparison is quite persuasive.

S3. The authors also added the cost of building tables, energy consumption, and results on another type of device. These experiments help determine what costs need to be paid when the solution is actually deployed.

Critical weaknesses

W1. Table 3 has already shown that the quality of the answers has improved after adding phrase information. However, MoLE did not use the same compression, fused computation, and prefetching methods, so it is not clear whether RcMoE is still more cost-effective in terms of quality, speed, and storage under the same optimization conditions.

W2. The speed results in Tables 2 and 3 are consistent, but in Figure 7, the throughput of the larger model when generating 64 tokens is significantly lower. The paper did not explain the differences in the corresponding test conditions, which affects the judgment of the speedup.

W3. The data that is read in advance may already be in the system cache. Specifying two prefetch slots alone does not show how much was actually read from the SSD. Therefore, it is not clear how much of the speed improvement comes from prefetching and how much depends on data that has already been cached.

Questions for authors

Q1. Can you apply the same compression, kernel fusion, and prefetching to MoLE, then compare its speed, storage requirements, and answer quality with RcMoE?

Q2. Can you explain why the throughput differs between Figure 7 and Table 2 when the larger model generates 64 tokens? Please clarify any differences in test conditions.

Q3. Can you compare performance with and without cached data, and report the actual SSD read volume and memory usage, including the file cache?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ND12

We sincerely thank you for recognizing the clarity of our presentation and for your insightful comments. We address these concerns by comparing MoLE under the same optimizations, clarifying the evaluation conditions, and analyzing caching and prefetching.

1. Comparison under the same optimization conditions. Recall that our research objective lies in making MoE-based LLM inference practical on resource-constrained edge devices (e.g., 8GB Jetson Orin Nano), whereas MoLE is not designed for edge deployment. To achieve this, our proposed optimization pipeline (compression, kernel fusion, and prefetching) serves as a core contribution of RcMoE. Although MoLE cannot directly apply this pipeline, we invested additional effort to adapt it to MoLE, denoted as MoLE (Opt.). Using this adapted baseline, we evaluate the overall cost-effectiveness under comparable LUT storage (using a 1K pattern vocabulary for RcMoE).

Performance Comparison of MoLE, MoLE (Opt.), and RcMoE.

ModelMethodAccuracy (%)LongBenchLUT Storage (GB)Latency (s)
1.7BMoLE63.4710.8513.507.14
1.7BMoLE (Opt.)63.2110.743.436.31
1.7BRcMoE63.5211.123.446.67

MoLE (Opt.) reduces storage and latency but incurs a slight accuracy drop. This indicates that our SC-LUT design is essential for the retrieval-centric architecture to achieve better performance. By capturing richer contextual semantics, RcMoE overcomes this degradation and surpasses the MoLE while using about 1/4 of the storage (e.g., reaching 63.52% Accuracy and 11.12 LongBench at 1.7B). Moreover, although online semantic matching requires more operations than static token lookup, our pipeline keeps the latency overhead minimal. Overall, co-designing SC-LUT and our optimization pipeline achieves a better trade-off among accuracy, storage, and latency on resource-constrained edge devices.

2. Clarification of evaluation conditions in Figure 7 and Table 2. The original Figure 7 used a Prompt Length of 512 tokens on the NVIDIA Jetson Orin Nano, whereas Table 2 used the default 16 tokens. The different prompt lengths result in different prefill workloads and end-to-end inference latencies, accounting for the observed throughput discrepancy. We apologize for not specifying this difference. We have redrawn Figure 7 using previously obtained 16-token measurements and updated Section 4.2 (Output Length Scalability).

To examine the impact of prompt length, we evaluate 16-2,048 input tokens with 64 generated tokens. The 1.7B results are shown below.

Performance of the 1.7B model across Prompt Lengths

Prompt (tokens)MoLE Latency (s)MoLE Throughput (tokens/s)RcMoE Latency (s)RcMoE Throughput (tokens/s)
167.148.966.649.64
647.188.916.689.58
1287.228.866.729.52
2567.628.407.069.07
51212.455.1411.685.48
1,02414.854.3113.884.61
2,048OOM--19.393.30

Throughput is relatively stable through 256 input tokens but falls as prefill cost rises. RcMoE has lower latency and higher throughput wherever both methods complete inference. At 2,048 tokens, MoLE encounters OOM, whereas RcMoE achieves 3.30 tokens/s. Both scales are reported in Appendix D (Prompt Length Scalability, Table 10).

3. The analysis of page caching and prefetching. Page caching reduces SSD reads, while asynchronous prefetching provides additional performance benefits under hot-cache conditions. RcMoE accesses SC-LUT files through Linux mmap, with page residency managed by the page cache. Asynchronous prefetching uses two alternating prefetch slots and background workers to issue prefetch requests for next-layer SC-LUT entries while the current layer executes.

We evaluate the 1.7B model with 64 input and 64 generated tokens using the low-repetition input. Cold-cache runs have no resident LUT pages, while hot-cache runs follow a warm-up generation. Results average three runs and each pair denotes Prefetch Off / On.

Performance, SSD reads, and memory of the 1.7B model (Prefetch Off / Prefetch On).

CacheLatency (s)Throughput (tokens/s)SSD Read (MB)Peak RSS (MB)Peak Cached (MB)
Cold6.33/6.3010.11/10.1611.42/9.381122.50/909.702273.00/2473.00
Hot6.55/6.189.78/10.350.31/0.59999.30/1130.102349.00/2352.00

Hot-cache reads (0.31-0.59 MB) are far below cold-cache reads (9.38-11.42 MB), showing that page caching reduces SSD I/O. Even under hot-cache conditions, asynchronous prefetching reduces latency from 6.55 to 6.18 s despite minimal SSD reads. We have clarified the system implementation in Appendix B (Asynchronous SC-LUT Prefetch) and added the complete experimental results in Appendix E (Inference Performance under Different Cache Conditions, Table 18).

Thank you for the additional experiments. They address my main concerns, and I will raise my score.

Dear Reviewer ND12,

Thank you for your support. We appreciate your recognition of the clarity of our design and the value of our additional experimental analyses. Have a nice day!

Thanks again,

The authors

Reviewer_KrNYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes RcMoE, a retrieval-centric MoE framework for edge inference. It extends token-indexed lookup with NPMI-selected local patterns and introduces an execution pipeline combining LUT compression, fused kernels, and asynchronous prefetching. Experiments demonstrate competitive model quality and substantial inference improvements on resource-constrained devices.

Critical strengths

The overall approach is reasonable and technically coherent. Extending token-only retrieval with local contextual information is a natural design choice, and the experiments provide empirical support for its effectiveness. The paper also presents a relatively complete system implementation, with useful comparisons and ablations covering model quality, storage, and inference performance.

Critical weaknesses

none

Questions for authors

Could the authors provide some further discussion or analysis of how the NPMI-selected local patterns relate to the contextual distinctions needed for expert retrieval? While the overall design is reasonable, a clearer interpretation of what these patterns capture and why they are useful for retrieval would help strengthen the modeling motivation and provide more insight beyond the empirical improvements.

3 comments
AuthorsOpenReview ↗

Response to Reviewer KrNY

We sincerely thank you for your thoughtful review and for recognizing the reasonableness of our overall design. Below, we clarify what the NPMI-selected local patterns capture for retrieval, analyze how they provide contextual distinctions for expert retrieval, and discuss the insights beyond empirical improvements.

1. Interpretation of what NPMI-selected local patterns capture and why they are useful for retrieval. Local patterns selected by NPMI capture recurring bigrams and trigrams with statistical associations between neighboring tokens. These patterns are useful for retrieval because they provide discrete lookup conditions that distinguish different local contexts, allowing the same token to have different retrieval representations depending on its surrounding words.

2. Analysis of how NPMI-selected local patterns relate to the contextual distinctions needed for expert retrieval. NPMI-selected bigrams and trigrams distinguish local contexts associated with the same token, providing finer-grained conditions for expert retrieval. RcMoE combines the token embedding with each matched pattern embedding, so the same token can produce different precomputed Retrieval Expert outputs across local contexts. Meanwhile, local patterns only guide Retrieval Expert lookup without replacing the full contextual hidden state. The Router computes aggregation weights from the full contextual hidden state, the Shared Expert takes the full hidden state as input for online computation, and the residual connection keeps the original context.

To measure Retrieval Expert output differences across local patterns, we test the RcMoE-360M checkpoint on 60 WikiText-2 validation articles using BF16 Retrieval Experts.

Analysis of Retrieval Expert output differences under different semantic patterns.

MetricValue (%)
Pair-weighted symmetric relative L214.69
Mean relative L2 (Semantic-Conditioned vs. Token-Indexed)10.37

As shown in the table above, matching different patterns for the same token creates a 14.69% relative L2 difference in Retrieval Expert outputs. Compared with token-indexed lookup using only token identity, semantic-conditioned lookup with local patterns also brings a 10.37% mean relative L2 difference. Together, these results show that incorporating NPMI-selected local patterns makes Retrieval Expert outputs sensitive to local contexts beyond token identity alone.

3. Further discussion and insights beyond the empirical improvements. The primary goal of our work is to make MoE-based LLM inference practical on resource-constrained edge devices (e.g., Jetson Orin Nano 8G). Retrieval-centric inference offers a promising approach to reducing the cost of expert access on resource-constrained devices. However, preserving model quality in a retrieval-centric architecture requires more than retrieving expert outputs based solely on token identity, as the same token may require different Retrieval Expert outputs in different contexts. The key insight of RcMoE is that effective expert retrieval should account for contextual distinctions beyond token identity, while maintaining sufficient reusability for offline precomputation. By combining semantic-conditioned precomputed retrieval with contextual routing, RcMoE brings together the local-context sensitivity of expert computation and the efficiency of table lookup, providing a practical design for efficient MoE-based LLM deployment on commodity edge devices.

Overall, these analyses clarify what local patterns capture, how they introduce local contextual distinctions into Retrieval Expert outputs, and why semantic-conditioned retrieval is useful for preserving model quality in retrieval-centric architectures. We have incorporated these clarifications into Section 3.2 (Semantic-Conditioned Representation, Lines 177–182) and added the analysis in Appendix E (Pattern Effects on Retrieval Expert Outputs, Table 13).

Thank you for the clarification and the additional analysis. I agree that incorporating local contextual information is a reasonable design choice for expert retrieval, and the new results provide sufficient support for this motivation. A deeper mechanistic analysis of what these patterns capture could further strengthen the paper, but the current treatment is acceptable. I will maintain my original score.

Dear Reviewer KrNY,

Thank you for your continued support. We are encouraged by your recognition of our additional analysis and design motivation. Have a nice day!

Thanks again,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.