acceptodds
Under review as a conference paper at ICLR 2027

FRAME: Routing-Aware Expert Placement for CPU–CXL MoE Inference

Abstract

Mixture-of-experts (MoE) models activate few experts per token, yet serving requires access to the full expert set, so expert-weight memory requirements scale with total rather than active parameters. Keeping all weights in GPU memory is costly, while offloading can stall decoding with weight transfers. CPU servers with DDR5 and CXL memory expand available capacity and allow CPU cores to execute experts directly from either tier. Efficient placement remains challenging: DDR5 capacity is limited, expert activation patterns change across requests, and migration into DDR5 competes with in-place expert reads for CXL bandwidth. We present FRAME, a routing-aware expert-placement system for CPU–CXL MoE inference. FRAME separates expert ranking from DDR5 admission so that changing routing estimates need not trigger immediate migrations. An online adaptive mechanism estimates residency benefits from recent expert activations, while a residency controller selects between frequency-based and adaptive rankings according to how well historical frequencies capture current routing patterns. Bounded admission and eviction regulate the resulting migrations, enabling placement adaptation while controlling its CXL traffic cost. We evaluate FRAME on a real CPU–CXL system across three MoE architectures and controlled and held-out workloads. Following a routing-distribution shift introduced without notifying the runtime, FRAME achieves 17.09 tokens/s on a cross-domain request sequence, outperforming the strongest adapted baseline by 17.6%. It also achieves 16.92 tokens/s on 24 held-out LongBench requests spanning six tasks. Mechanism analysis shows how ranking and admission jointly determine the trade-off between reducing subsequent CXL reads and incurring migration traffic.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture-of-experts, CXL memory, CPU inference, expert placement, memory tiering, expert migration, offline/online residency control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ygzeOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FRAME, an expert placement system for MoE inference in CPU–CXL environments. In this setting, CPUs can directly execute CXL-resident experts without migrating their weights to DDR5. However, direct CXL access and expert migration compete for the same CXL bandwidth, making efficient residency management essential. FRAME combines three mechanisms: (1) online adaptive ranking based on recent expert activations and routing information, (2) a routing-aware selector that switches between frequency-based and adaptive rankings, and (3) bounded admission and residency management to control migration traffic. A key design principle is to separate expert ranking from actual DDR5 admission so that changes in ranking do not immediately trigger migrations. The authors evaluate FRAME on a real CPU–CXL server using Qwen3-Next-80B, Qwen3-30B, and OLMoE-1B-7B. Following a routing shift, FRAME achieves 17.09 tokens/s on the 80B model, outperforming the strongest adapted external baseline, Pre-Gated, by 17.6%. It also achieves 16.92 tokens/s on 24 held-out LongBench requests. Additional analyses examine the relationships among ranking accuracy, ready recall, migration volume, and CXL read traffic.

Critical strengths

Strength 1: Empirical Characterization of the Migration–Residency Trade-off in CPU–CXL Systems An important strength of this paper is its characterization of the costs and benefits of expert migration on real CPU–CXL hardware. Because CPUs can directly execute CXL-resident experts, migration becomes an optional optimization whose cost must be amortized through subsequent expert reuse. The paper identifies bandwidth contention between direct CXL reads and expert migration as a key system-level optimization problem. In particular, the break-even microbenchmark in Appendix A.2, the routing-dependent throughput and migration analysis in Figure 4, and the frequency-core sensitivity analysis in Appendix D.3 quantitatively demonstrate this trade-off. These findings provide valuable insights into when expert migration translates into actual performance improvements, beyond simply improving cache hit rates. I consider this an important empirical contribution that supports acceptance.

Strength 2: Controlled Experimental Design and Detailed Validation The paper makes a substantial effort to evaluate the proposed placement policy under controlled experimental conditions. As described in Section 4.1 and Appendix A.1, the compared policies share the same CPU–CXL execution engine, model, NUMA binding, copy path, and memory budget. The dedicated-core control in Appendix A.1 further verifies that contention between the migration worker and inference threads does not determine the main performance ordering. Appendices B.1 and B.3 provide repeated measurements and paired statistical comparisons. Appendix B.2 additionally checks output equivalence between the modified execution engine and a reference implementation, although the authors appropriately acknowledge that this does not establish token-level identity or task accuracy. While these controls do not constitute complete reproductions of the original baseline systems, they provide a useful basis for evaluating the relative effectiveness of different placement mechanisms under the same CPU–CXL environment. The explicit documentation of baseline adaptations, implementation differences, and experimental limitations further strengthens the transparency and credibility of the evaluation.

Critical weaknesses

Weakness 1: Insufficient Isolation of the Independent Contributions of Ranking and Admission The central contribution of FRAME lies in separating routing-aware expert ranking from bounded DDR5 admission to control migration traffic. However, related ideas such as activation-aware expert caching and residency management have been explored in prior work. For instance, MoE-Infinity and ProMoE investigate activation-aware caching and predictive expert management, respectively, while FluxMoE addresses expert residency management in a different execution environment. Furthermore, Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU–NDP Systems considers related problems involving CXL-based expert execution and workload-aware placement, although its execution architecture differs from FRAME's CPU-only setting. These prior studies do not invalidate FRAME's contribution. In particular, directly executing CXL-resident experts on CPUs introduces a different cost structure from conventional GPU offloading, potentially motivating specialized placement policies. Nevertheless, the paper should more clearly establish the additional benefits of its specific ranking and admission mechanisms over simpler adaptive caching approaches. Section 5.1 and Appendix C compare predictor recall–turnover characteristics, Appendix D.1 analyzes online feedback, and Appendix D.3 extensively examines the impact of the frequency-core size on migration traffic. However, the individual end-to-end contributions of the adaptive ranking signals, the admission bound in Eq. (8), and the engine-side eviction hysteresis remain insufficiently isolated. Although online feedback reduces migration traffic, its measured throughput improvement is relatively small, reaching only 0.075 tokens/s in the 10 GiB configuration. Consequently, it remains unclear how much additional benefit FRAME provides over a simpler recent-frequency ranking with the same admission controller, or over an adaptive ranking with a different admission policy. This is not merely a request for additional experiments. It directly affects the assessment of the proposed controller's independent contribution and whether its design complexity is justified.

Weakness 2: Limited Evidence for the Benefits and Generalizability of the Routing-Aware Selector One of FRAME's central mechanisms is a routing-aware selector that chooses between frequency-based and adaptive rankings according to changes in the routing distribution.

Figure 3 demonstrates that this mechanism can respond effectively to a routing shift. However, Figure 3(b) and Table 6 show that fixed adaptive achieves higher post-shift throughput than FRAME (17.31 vs. 17.09 tokens/s). Furthermore, Section 4.2 reports that FRAME maintains throughput within 0.3% of fixed adaptive over the full stream while reducing completed migration by approximately 7.0%. This is a meaningful reduction, but it also indicates that the selector's measured benefit primarily lies in migration efficiency. FRAME does improve post-shift throughput over fixed frequency, so these results do not imply that the selector is unnecessary. The main concern is whether this throughput–migration trade-off remains beneficial under more diverse routing dynamics. The main 80B experiment considers a transition from a stable workload to a shifted workload. In contrast, Appendix B.5 reports that the 30B model remains in adaptive mode throughout the experiment. Appendix E.2 further explains that repeated candidate resets in the 80B domain-drift experiment prevent the controller from returning to frequency mode. Therefore, while the ability to switch from frequency to adaptive ranking is demonstrated, the effectiveness of the reverse transition and its contribution to migration reduction are less clearly established. Additionally, the routing-shift threshold in Appendix A.4 is calibrated using a preceding pass over the same 14 request identities used in the main evaluation. The authors provide threshold sensitivity analyses and held-out experiments, so this observation does not invalidate the reported results. However, the generalizability of the threshold across model geometries and routing distributions remains an open question. For example, routing distances in the 30B experiment consistently exceed the threshold. As the authors explain, this may reflect the absence of a stable frequency regime for that model rather than incorrect selector behavior. Nevertheless, the current evidence does not clearly establish how reliably the mode-selection mechanism operates across different routing conditions. Overall, the experiments demonstrate that FRAME performs well under several evaluated workloads, but they provide limited evidence of the general robustness of its automatic mode-selection mechanism.

Questions for authors

*Q1. Independent Contributions of Ranking, Admission, and the Controller FRAME presents the separation of ranking and admission, together with bounded migration through Eq. (8), as a central design principle. Appendix D.1 evaluates the effect of disabling online feedback, but does not independently isolate the contributions of ranking-weight updates and the admission bound. Are results available comparing (1) a simple ranking such as recent-window frequency, (2) the full adaptive ranking without the feedback-based admission bound in Eq. (8), and (3) the complete FRAME controller, under the same execution environment? Additionally, does the benefit of bounded admission remain when the engine-side eviction hysteresis is held fixed? Such comparisons would help establish the independent contribution of the proposed ranking–admission separation and justify the combination of multiple control mechanisms.

Q2. Selector Robustness and Threshold Generalizability Figure 3 shows that FRAME maintains throughput comparable to fixed adaptive while reducing migration traffic following a routing shift. However, Appendix B.5 reports that the 30B model remains in adaptive mode throughout the experiment, while Appendix E.2 indicates that repeated candidate resets prevent a return to frequency mode in the 80B domain-drift experiment. Are there evaluated workloads where the selector successfully transitions from adaptive back to frequency mode and reduces migration traffic? Could the authors also provide per-request mode decisions for the held-out LongBench evaluation in Appendix B.4, ideally together with the corresponding throughput and migration volume? Furthermore, given that the threshold in Appendix A.4 was calibrated on the 80B workload, what evidence supports its applicability across different model geometries without additional tuning? These results would help assess whether the selector can consistently select and maintain appropriate residency policies across diverse routing distributions.

Q3. Generation Length and Evaluation-Time Residency Freeze The main experiments in Section 4.1 use a 32-token generation cap, while the authors note that actual generations in the LongBench experiments frequently terminate around 32 tokens despite a 120-token cap. What motivated the choice of these relatively short generation lengths? In particular, was this setting intended to match the evaluation protocols of prior baselines, or was it selected for other experimental or system-level reasons? Appendix A.3 also explains that the reported FRAME configuration includes an evaluation-time residency freeze after 30 issue calls, which is not part of the actual deployment rule in Table 16. I understand that this freeze affects only the last few tokens in the reported experiments and that the corresponding ablation does not show a statistically distinguishable throughput difference. Nevertheless, longer generations may alter the amortization of migration costs, the evolution of routing distributions, and the effectiveness of residency adaptation. If measurements are available for longer generations using the original deployment rule without the evaluation-time freeze, could the authors clarify whether the relative throughput and migration trends remain consistent? This question is intended to clarify the scope of generalization supported by the current evaluation rather than to request a comprehensive evaluation of production serving scenarios.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors(1/3)

We thank the reviewer for the detailed technical assessment. We agree that the key questions are whether FRAME's mechanisms have separable system-level value, when automatic mode selection is useful, and whether results hold under longer generation without evaluation-time shortcuts. We added matched controls and revised Sections 3.3, 4.1–4.4, 5.1–5.2, Appendices A.3–A.4, B.2/B.5, C.1, D.1/D.4, and Limitations. Revisions are highlighted in blue.

W1/Q1. Independent contributions of ranking, admission, and eviction. Activation-aware caching and prediction are established in MoE-Infinity, ProMoE, FluxMoE, and related work; context-aware CXL GPU–NDP inference explores a different near-memory path. FRAME addresses a narrower setting: on a CPU-only CXL host, nonresident experts can execute directly from CXL, so DDR5 migration is optional and costly because copies contend with direct reads. We therefore separate which experts to rank from whether to admit DDR5 copies, rather than attributing the overall gain to a novel predictor.

First, a three-arm end-to-end control uses six unseen GSM8K/WMT16 requests (80B model, 10 GiB pool, both arm orders). Model, outputs, token counts, stopping conditions, copy queues, ranking cadence, CPU bindings, and eviction rules are matched (Appendix D.1, Table 24).

PolicyGeneration (tok/s)Completed copies (GB)
A: FRAME adaptive ranking + bounded admission17.09068.209
B: Recent-window frequency + same admission16.89569.597
C: Recent-window frequency + unrestricted policy admission14.476126.659

A vs. B isolates the composite adaptive ranker: +1.15% throughput with slightly fewer copies. B vs. C holds ranking fixed and isolates bounded admission: +16.72% throughput. A vs. C yields +18.06% throughput and 46.15% fewer copies. “Unrestricted” removes only the policy replacement bound; all arms retain the same physical 10 GiB capacity. These conditional contrasts must not be added.

Second, matched studies hold adaptive ranking fixed to isolate the four-replacement ceiling, Eq. (8) hit-feedback refinement, and eviction protection (Appendix D.1, Tables 21–22).

ComparisonThroughput gainCopy reduction
Eq. (8) + cap4 vs. unrestricted admission+19.01%53.06%
Fixed cap4 vs. unrestricted admission+18.89%52.05%
Eq. (8) + cap4 vs. fixed cap4+0.10%2.10%
Current-generation protection on/off, bounded admission+4.77%38.38%
Current-generation protection on/off, full admission+0.44%−0.66%

The fixed ceiling explains most of the throughput gain; Eq. (8) further reduces traffic, tightening the cap in 720 of 6,809 admission opportunities. The earlier feedback-on/off difference of only 0.075 tok/s does not measure removing admission bounds: these new controls distinguish feedback refinement, a fixed cap, and unrestricted admission. A 2×2 admission-by-eviction control finds admission gains of 19.91% with and 14.96% without protection. It changes only current-generation protection under same-kind LRU; reader safety, capacity, and copy queues are retained, while the policy-side retention bonus and legacy page-migration trigger are disabled. Protection matters more under bounded admission, so its effect is not an independent additive gain.

Third, predictor-level evidence shows why additional ranking signals help, but should not be mistaken for large system speedups. In causal replay over 36,864 decode-layer events at P=107 (Appendix C.1, Table 18), composite adaptive/window candidate recall is 86.43%/73.91%. At equal logical migration (2.040 GB per 1,000 events), admitted recall is 81.73%/72.40% (+9.33 points). Yet the matched end-to-end gain is only 1.15%: resident copies may serve experts no longer in the candidate list, new copies may arrive too late, and missing experts execute directly from CXL. Freezing vs. updating ranking weights changes native throughput by only −0.24% to +0.50% across four paired controls. Thus, the composite adaptive ranker improves ranking quality, but its measured end-to-end contribution is modest; these controls do not assign separate speedups to every routing, transition, or gate signal. The principal benefit comes from migration-limited admission, with a conditional benefit from eviction protection. This decomposition—not prediction accuracy alone—motivates the controller's components.

AuthorsOpenReview ↗

Official Comment by Authors(2/3)

W2/Q2. Automatic selector, threshold transfer, and reverse transitions. The original post-shift result (FRAME 17.09 vs. fixed ADAPT 17.31 tok/s) does not show a throughput advantage over always-ADAPT. Across the original stream, FRAME stays within ~0.3% of fixed ADAPT while reducing copies by ~7%; its demonstrated value is migration efficiency.

We froze the entry threshold (0.770115) and tested it on 64 unseen LongBench requests across two independently calibrated overlap regimes (Appendix A.4, Table 7): HotpotQA/QMSum (Jaccard 0.127) and GovReport/MultiNews (0.473). Each pair has A4–B4–A4 and B4–A4–B4 streams plus same-task controls. Residency persists within each sequence; the selector receives no task-boundary signal. We use official generation caps, natural EOS, and both execution orders.

OverlapPhaseFixed FREQFixed ADAPTFRAME
LowerInitial16.29217.05517.195
LowerShifted15.69017.03116.997
LowerReturned task15.47917.07917.056
HigherInitial17.88617.81317.793
HigherShifted17.14917.63417.594
HigherReturned task17.19317.59817.572

Throughput in tok/s; phases denote workload changes, not mode switches. Pair Jaccard differs from the selector's request-level displacement score.

Across complete lower-/higher-overlap streams, FRAME achieves 17.030/17.691 tok/s, vs. 15.842/17.512 for fixed FREQ and 17.020/17.714 for fixed ADAPT.

OverlapThroughput vs. FREQCopies vs. FREQThroughput vs. ADAPTCopies vs. ADAPT
Lower+7.50%+64.68%+0.06%−0.49%
Higher+1.02%+115.08%−0.13%−0.12%

FRAME-relative changes; positive copy changes mean more migration. Completed copies include prefill.

After the first transition, throughput gains over FREQ are 9.36%/2.40%, with 83.32%/123.04% more copies. This is a throughput–migration trade-off, not migration reduction. Against fixed ADAPT, FRAME preserves nearly the same operating point, with 0.49%/0.12% fewer copies and negligible throughput differences. These 64-request results differ from the ~7% fewer copies in the controlled 14-request stream and do not establish migration reduction in both overlap regimes. FREQ is slightly faster in the higher-overlap initial phase; later phases favor ADAPT.

Per-request decisions and limitations. The earlier 24-request LongBench expansion used an adaptive-only build and cannot evidence selector switching. In the new tests, all eight sequences enter ADAPT at cold start (0.862–0.897 displacement, above threshold); all recorded modes remain ADAPT. Of 104 post-first decisions, 82 reset candidate-frequency state; only 14/96 valid paired comparisons meet the return-hit condition, with no consecutive qualifying wins (Appendices A.4, E.2). This supports selected-state performance, not bidirectional-switching generalization.

OverlapSequenceModes in request order (both orders)
LowerA–B–AD D D - D D D D D D - D
LowerB–A–BD D D D D - D D D D D D
LowerA onlyD - D D
LowerB onlyD D D D
HigherA–B–AD D D D D D D D D D D D
HigherB–A–BD D D D D D D D D D D D
HigherA onlyD D D D
HigherB onlyD D D D

D = ADAPT; “−” = unrecorded mode.

Two controls delimit threshold transfer. On unseen GSM8K→MMLU→HumanEval requests, post-change displacement is 0.628–0.687, below threshold, so FREQ persists despite task changes. On 40 shared 30B/80B requests, the same thresholds work without retuning and both geometries retain ADAPT: 18.365/17.624 tok/s vs. 18.135/17.649 for fixed ADAPT (Appendix B.5). This supports threshold transfer and selected-state performance, not universally correct switching: 30B displacement (0.79–0.86) remains above the entry threshold. The separate candidate-reset freshness threshold is also geometry-dependent (Appendix E.2), so these runs do not establish cross-geometry reverse-transition reliability.

For ADAPT→FREQ, a development return-only control switches at request 9, reducing copies 4.69–4.71% at 0.19–0.32% throughput cost. An eight-request refined-selector test enters ADAPT at request 6 and returns to FREQ at request 8, reducing copies 8.53–8.57% at 0.05–0.95% throughput cost (Appendix D.4). On repeated GSM8K, fixed FREQ beats fixed ADAPT by 4.35% throughput and 44.00% fewer copies. These are refinement/control results, not evidence that the original frozen rule independently performs reverse switching; the Limitations section now states this explicitly.

AuthorsOpenReview ↗

Official Comment by Authors(3/3)

Q3. Generation length and evaluation-time freeze. The 32-token cap was chosen to constrain decode-work variation in the controlled shift comparison, not to match another system's evaluation configuration. Importantly, the main 14-request stream does not freeze residency updates. A 30-issue-call freeze applied only to earlier pilot and 24-request LongBench settings; the distinction is now explicit in Section 4.1 and Appendix A.3.

We added a new 32-request, eight-task LongBench evaluation using official generation caps, natural EOS, and the deployed continuously updating placement rule without a freeze (Appendix B.2, Table 10). Actual outputs range from 4 to 512 tokens, including eight requests exceeding 400 tokens. Across both execution orders, FRAME, fixed adaptive, and recent-window frequency match in exact tokens, stopping conditions, output text, and official task scores.

PolicyGeneration (tok/s)Completed copies (GiB/block)
FRAME17.028426.999
Fixed adaptive17.042427.732
Recent-window frequency17.118429.043

FRAME is 0.08% slower than fixed adaptive and 0.53% slower than recent frequency, with 0.17% and 0.48% fewer completed copies, respectively. Throughput covers decoding; whole-request copy totals include prefill, so these small differences should not be interpreted as a decode-only migration-amortization measurement. The near-tie is consistent with recent frequency already capturing much of the reusable demand in this workload; it does not demonstrate persistent speedups for every long generation. Separate shifted-routing controls show where adaptive ranking helps (+9.36%/+2.40% over cumulative frequency after transitions), and the isolated admission comparison shows a larger +18.06% throughput / −46.15% copying relative to unrestricted recent-window admission. These results establish continuous-update behavior at longer output lengths while identifying the workload conditions and limits of the benefits.

Reviewer_joe3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FRAME, a routing-aware expert placement framework for CPU–CXL Mixture-of-Experts (MoE) inference. Unlike conventional offloading systems, where nonresident experts must be transferred before execution, CPU–CXL systems allow experts to be executed directly from CXL memory. FRAME exploits this distinction by treating expert migration as a performance optimization rather than an execution requirement.

The proposed framework separates expert ranking, routing-aware mode selection, and residency control. It dynamically selects between frequency-based and adaptive expert rankings based on observed routing changes, while controlling migration traffic to reduce contention for CXL bandwidth.

Critical strengths

1. Well-motivated problem formulation and system design. The paper identifies an important architectural distinction between CPU–CXL inference and conventional expert offloading. Since nonresident experts can be executed directly from CXL memory, the problem shifts from ensuring expert availability to deciding when migration into limited DDR5 capacity is beneficial. This makes the trade-off between memory bandwidth, expert reuse, and migration cost particularly relevant. The decomposition into expert ranking, routing-aware selection, and residency control is also a well-motivated design choice that separates prediction from actual placement decisions.

2. Evaluation on real CPU–CXL hardware. A significant strength is the experimental validation on a real CPU–CXL platform rather than relying solely on simulation or analytical models. Performance in this setting depends on hardware characteristics such as CXL bandwidth, memory access latency, and contention between expert reads and migration. The real-system evaluation and migration analysis therefore provide useful evidence supporting the practical relevance of the proposed approach.

Critical weaknesses

1. Limited fidelity of the baseline comparisons. The paper compares FRAME against mechanisms associated with MoE-Infinity, ProMoE, ExpertFlow, Fiddler, and other approaches. However, Appendix A.1 states that these are adaptations of the respective mechanisms to the CPU–CXL runtime rather than complete reproductions of the original systems. Some adaptations omit components of the original approaches, which could affect their performance. Although evaluating all methods within a common runtime is useful for controlled comparisons, it remains unclear whether these adaptations represent sufficiently strong baselines. This limits how conclusively the reported performance improvements can establish FRAME's advantage over existing methods.

2. Limited independent validation of the routing-aware selector threshold. The routing-aware selector uses a threshold of approximately 0.77, calibrated from an earlier execution pass involving the same 14 request identities used in the main controlled evaluation (Appendix A.4). Although the authors provide threshold-sensitivity analysis and evaluate held-out LongBench requests without retuning, the principal routing-shift experiment is not an independent validation of the selected threshold on unseen routing transitions. Consequently, it is unclear how reliably the selector would detect routing changes with different characteristics, particularly when the stable and shifted routing distributions are less clearly separated. This limits the evidence for the generalizability of the proposed mode-selection mechanism.

Questions for authors

Q1. Baseline fidelity (Appendix A.1): Since the compared methods are adaptations of existing mechanisms rather than complete reproductions, could the authors clarify how the omitted components may affect baseline performance? In particular, what evidence supports the claim that these adaptations provide sufficiently competitive comparisons under the CPU–CXL setting?

Q2. Routing-shift threshold generalization (Section 3.3 and Appendix A.4): The routing-aware selector threshold is calibrated using the same request identities as the main controlled evaluation. Could the authors provide additional evidence that this threshold generalizes to independent routing-shift sequences? In particular, how does the selector perform when the routing distributions before and after a shift overlap more substantially? Providing selector decisions on the held-out LongBench workload would also help address this concern.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the careful assessment. The concerns about baseline fidelity and independent threshold validation helped us distinguish mechanism-level evidence from claims about complete systems. We have strengthened the CPU–CXL baseline implementations, added frozen-threshold evaluations and per-request selector summaries, and revised Sections 4.1–4.2, Table 5, Appendices A.1/A.4 and D.4, and the Limitations section. All revisions are highlighted in blue in the revised manuscript.

W1/Q1. Fidelity and competitiveness of baseline adaptations. We agree that running methods on a common backend is not sufficient if important parts of their placement mechanisms have been omitted. Accordingly, the revised comparisons incorporate stronger CPU–CXL adaptations (Table 5; Appendix A.1):

  • ProMoE: Learned top-10 expert prediction, asynchronous LOW/HIGH queues, current-layer demand preemption, and LRU replacement. Predictor execution is included in the serving configuration; the reconstruction choices for unpublished training settings and the disjoint training/evaluation request groups are documented.
  • ExpertFlow: CPU-side online RPP prediction, locality ranking, LRU, and route correction. Unlike an offline-trace approximation, the online prediction cost is included in generation measurements.
  • EAMC-C2: Activation-pattern ranking with history updates and score-based eviction. Adding these controls strengthens the adaptation by 37.64–38.21% in throughput, with 44.33–44.60% fewer completed copies relative to its earlier adaptation.
  • Fiddler: Hot/cold placement with reuse-based copy admission. A reuse cutoff of 11, supported by measured operator and copy costs, improves the adaptation by 5.49–6.02% over cutoff 4 while reducing copies by 17.17–17.53%.

The adaptations yield the following paired CPU–CXL comparisons:

AdaptationFRAME throughput gainCompleted copies: FRAME / reference
ProMoE+13.31%29.675 / 97.460 GB
ExpertFlow+10.87%67.08 / 167.32 GB
EAMC-C2+61.81%70.47–70.48 / 437.56–438.22 GB
Fiddler+23.94%FRAME copies 3.69–3.71× more

Each baseline is compared with FRAME using identical requests and execution settings. Completed-copy totals include prefill. The Fiddler result is informative: less copying does not necessarily give higher throughput if useful experts are less often resident. We also retain the MoE-Infinity EAM mechanism-level comparison; the held-out predictor study reports 89.63% versus 80.97% candidate Recall@107 for FRAME ranking and EAM (Table 3), which is distinct from the native serving comparisons.

We also clarify what was not reproduced. Original GPU-specific execution ordering, GPU memory management, and batch/token scheduling are outside this fixed CPU-only backend. Their omission can affect overlap and the original systems' absolute performance. We therefore claim advantages over strengthened prediction and residency mechanisms within matched CPU–CXL execution, not superiority over the full original GPU systems. The anonymous implementation and scripts make the adaptations and their deviations inspectable.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W2/Q2. Independent routing-overlap and frozen-threshold evidence. The reviewer is correct that the original 14-request experiment does not independently validate its entry threshold: 0.770115 was selected using an earlier pass over the same request identities. We therefore froze this threshold and evaluated 64 unseen LongBench requests without retuning (Appendix A.4, Table 7).

We selected two task pairs using separate calibration routes: HotpotQA/QMSum (lower top-107 expert-set overlap; Jaccard 0.127) and GovReport/MultiNews (higher overlap; 0.473). Each pair includes A4–B4–A4 and B4–A4–B4 sequences, plus two four-request same-task controls, for 32 requests per pair. Each sequence preserves its own residency history; the selector receives neither task labels nor change-point information. All policies use the same model, 10 GiB pool, official generation caps, natural EOS, and both execution orders.

OverlapPhaseFixed FREQFixed ADAPTFRAME
LowerInitial16.29217.05517.195
LowerShifted15.69017.03116.997
LowerReturned task15.47917.07917.056
HigherInitial17.88617.81317.793
HigherShifted17.14917.63417.594
HigherReturned task17.19317.59817.572

Generation throughput in tok/s. “Returned task” identifies the workload phase, not a selector transition. Pair-level Jaccard is distinct from the request-to-frequency displacement score used by the selector.

Across the complete lower-/higher-overlap streams, FRAME achieves 17.030/17.691 tok/s, compared with 15.842/17.512 for fixed FREQ and 17.020/17.714 for fixed ADAPT. Relative to fixed FREQ, FRAME changes throughput by +7.50%/+1.02% and completed copies by +64.68%/+115.08%. From the first task transition onward, throughput gains are +9.36%/+2.40%, with +83.32%/+123.04% more copies. Thus, FRAME’s advantage over fixed FREQ is a throughput–migration trade-off, not migration reduction.

Relative to fixed ADAPT, throughput changes are +0.06%/−0.13%, while copies change by −0.49%/−0.12%. FRAME therefore preserves nearly the same operating point as fixed ADAPT in these streams. These results are distinct from the approximately 7% fewer copies observed against fixed ADAPT in the controlled 14-request stream; they do not establish migration reduction across both overlap regimes. In the higher-overlap initial phase, fixed FREQ is slightly faster, while later phases favor ADAPT.

Actual selector decisions. All eight sequences start in ADAPT because cold-start request-to-frequency displacement is 0.862–0.897, above the unchanged entry threshold. Across both execution orders, every recorded decision remains ADAPT. Of 104 post-first decisions, 82 reset the candidate-frequency state; only 14 of 96 valid paired comparisons meet the return-hit condition, with no consecutive qualifying wins. Accordingly, the 64-request experiment validates selected-state performance on independent requests, but does not demonstrate that the frozen selector detects each task transition or returns to FREQ.

OverlapSequenceRequest-ordered modes (same in both execution orders)
LowerA–B–AD D D - D D D D D D - D
LowerB–A–BD D D D D - D D D D D D
LowerA onlyD - D D
LowerB onlyD D D D
HigherA–B–AD D D D D D D D D D D D
HigherB–A–BD D D D D D D D D D D D
HigherA onlyD D D D
HigherB onlyD D D D

D = ADAPT; “−” = no recorded mode. Positions follow request order within each sequence.

We added an independent low-displacement task-shift control using six previously unused GSM8K requests followed by four unused MMLU and four unused HumanEval requests. Despite the task changes, post-change displacement remains 0.628–0.687, below the same frozen threshold, and the selector retains FREQ from request 3 onward. This indicates that the selector responds to measured routing displacement rather than task identity alone.

Finally, separately evaluated development refinements—not the original frozen rule—demonstrate ADAPT→FREQ recovery and reduce copies by 4.69–8.57%, at small throughput costs (Appendix D.4). We distinguish these refinement results from the frozen-rule evaluation and do not use them as evidence that the original selector successfully detected transitions.

AuthorsOpenReview ↗

Follow-up to Reviewer joe3

We sincerely thank the reviewer for the thoughtful feedback. In response, we strengthened the CPU–CXL baseline implementations and clarified their comparison scope. We also added an independent evaluation on 64 unseen LongBench requests with the selector threshold fixed, reporting per-request decisions and explicitly clarifying the limits of bidirectional switching. The corresponding revisions are highlighted in the updated manuscript.

We would greatly appreciate your feedback on whether these additional experiments and clarifications have addressed your concerns. If so, we would be grateful if you could consider updating your assessment in light of the revised evidence.

Reviewer_ueu7OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FRAME, a CPU-CXL system target for MoE inference. They first characterize the execution-migration trade-off. Then, they design an adaptive expert-placement strategy to optimize overall system performance. Finally, they evaluate the framework on the real system.

Critical strengths
  1. The experiments are performed on the real hardware system.
  2. The CPU-CXL system is a very important platform for LLM model deployment.
Critical weaknesses
  1. The comparison with prior studies is not adequate. For example, TPP (ASPLOS23) and MEMTIS (SOSP23) have already discussed several methods for a heterogeneous memory hierarchy. Although these works do not emphasize MoE models, their methods may originally support expert selection.
  2. I think the biggest issue is that the selected batch size (request size) is too small in this work (16). Since, under a large batch size, all experts will be selected and overall performance will be bounded by DDR5/CXL bandwidth, the proposed gating method may have little impact on overall performance.
Questions for authors
  1. DDR5 is not a SOTA DRAM on the CPU side. Can you provide an analysis of DDR6?
  2. More analysis of why the proposed method is better than other adaptive methods should be included.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the thoughtful comments, which helped us strengthen the hardware characterization and the controlled evaluation. We have revised the related-work discussion, system evaluation, and mechanism analysis (Sections 2, 4.1–4.2, 4.4, and 5.2; Appendices A.1–A.2, B.4, and D.1). All revisions are highlighted in blue in the revised manuscript .

W1. Comparison with TPP and MEMTIS. We agree that general-purpose tiered-memory policies are important references because CPU execution can access slow-tier data without first migrating it. We added CPU–CXL adaptations of the core migration rules, not full kernel implementations (Appendix B.4, Table 14). TPP uses repeated-reference promotion and inactive/LRU demotion; MEMTIS uses 16-bin hotness classification, capacity-based hot/warm thresholds, cooling, and warm-page retention. Both use request-batched PEBS observations, aligned 2 MiB tracking groups, and physical 4 KiB page movements. The comparisons use the same 80B model, 10 GiB fast-tier budget, CPU/NUMA bindings, and request stream; both execution orders are evaluated, and all 84 outputs, token counts, and stopping conditions match.

PolicyGeneration (tok/s)Promoted/copied (GB)Demoted (GB)
TPP adaptation12.9786.09–6.270.14–0.30
MEMTIS adaptation13.5533.71–3.773.71–3.77
FRAME17.09025.140.00

FRAME improves post-shift throughput by 31.69% over TPP and 26.10% over MEMTIS, even though it copies more bytes. The distinction is not that general tiering cannot run on CXL, but that page-access history and MoE expert reuse provide different placement signals. FRAME ranks experts using routing information and controls their DDR5 admission separately. Because a nonresident expert can still execute directly from CXL, ranking changes need not cause immediate migration. The results support the importance of selecting useful expert residency rather than minimizing copied bytes alone. Our claim is limited to these matched policy adaptations, not complete TPP or MEMTIS kernel reproductions.

W2. Larger physical batches and bandwidth. The previously reported 16-request predictor traces consist of independent requests; they are not a physical batch of 16. The original main stream has one active decode sequence. We therefore added genuine concurrent-decoding runs at 2, 4, 8, 16, 32, 64, and 128 active sequences sharing one 10 GiB expert pool (Appendix B.4, Table 15). Within each width, we hold ranking, eviction, model, and generated work fixed, changing only bounded versus unrestricted policy admission; both execution orders preserve outputs and stopping conditions. Reported throughput is total decode tokens divided by cumulative decode time across orders.

Active sequencesCXL RX (GB/s)Bounded throughput (tok/s)Gain vs. full admissionCopy reduction
169.9716.40+8.41%39.33%
328.9120.33+4.54%36.16%
647.7126.20+3.79%36.50%
1286.2830.51+2.34%39.70%

At width 128, the per-layer expert union has a median of 269, a 95th percentile of 337, and a maximum of 398 out of 512 experts. Thus, the measured working set remains selective; the assumption that every expert is selected is not borne out at the tested widths. We agree that the throughput advantage decreases with concurrency. Nevertheless, bounded admission still reduces completed copying by 39.70% and improves throughput by 2.34% at width 128. These are within-width admission ablations, not claims that the entire FRAME policy will retain the same benefit under arbitrary concurrency or a fully saturated future CXL configuration.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q1. Sensitivity to faster local DRAM. We added a microbenchmark-based sensitivity analysis (Appendix A.2, Table 6). At 24 threads, an expert-sized CXL read takes 137.52 µs, the corresponding DDR5 read 12.08 µs, and CXL-to-DDR5 copying 140.58 µs. The continuous migration break-even is therefore 1.12 future complete reuses. Hypothetically doubling local-memory read bandwidth while holding measured CXL read/copy costs fixed changes this to 1.07; both require at least two complete subsequent reads to amortize migration. Faster local DRAM therefore does not remove the central migration–reuse decision. This is a sensitivity study, not a measurement of DDR6 hardware or a prediction of absolute DDR6 throughput.

Q2. Why FRAME improves adaptive placement. We agree that prediction alone is insufficient to explain end-to-end gains: copying competes with in-place CXL reads, and replacing a copy before it is reused can waste migration bandwidth. We therefore added matched admission and eviction ablations with adaptive ranking fixed (Appendix D.1, Tables 21–22):

Controlled comparisonThroughput gainCopy reduction
Fixed four-replacement cap vs. unrestricted policy admission+18.89%52.05%
Eq. (8) + cap4 vs. fixed cap4+0.10%2.10%
Current-generation eviction protection, with bounded admission fixed+4.77%38.38%

The four-replacement ceiling accounts for the principal throughput benefit; the online refinement in Eq. (8) further suppresses copying, tightening the ceiling in 720 of 6,809 measured admission opportunities. Current-generation protection prevents premature replacement under the same LRU and reader-safety rules. These are conditional effects, not additive percentage gains. In paired CPU–CXL comparisons, FRAME also improves throughput over strengthened ProMoE by 13.31% and online ExpertFlow by 10.87% (Appendix A.1, Table 5). The collective evidence points to migration-aware residency control, rather than predictor accuracy by itself, as the main source of the benefit in this CPU–CXL setting.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary for the AC and reviewers: each concern and where the revision answers it (2/2)

4. Independent selector evaluation and generalization boundaries (joe3, ygze).

We evaluated the frozen routing-shift threshold on 64 unseen LongBench requests across task pairs with lower and higher expert-routing overlap, without retuning or supplying task boundaries.

FRAME improves throughput over fixed frequency by 7.50% and 1.02%, respectively, but incurs 64.68% and 115.08% more copying. All recorded decisions remain ADAPT. These results establish the performance of the selected adaptive state on independent requests, rather than successful detection of subsequent mode transitions.

A separate low-displacement control retains FREQ despite semantic task changes. We also report development controls demonstrating ADAPT-to-FREQ recovery and its migration benefits, while distinguishing these refinements from the original frozen rule.

We have clarified that reliable bidirectional mode selection across routing distributions and model geometries remains incompletely validated. The experimental evidence and corresponding limitations are documented in Appendix A.4, Appendix D.4, and the Limitations section.

5. Expanded concurrency, generation length, and hardware analysis (ueu7, ygze).

We added physical concurrent-decoding experiments with 2–128 active sequences sharing a 10 GiB expert pool. With ranking and eviction fixed, bounded admission improves throughput by 8.41% at 16 sequences and 2.34% at 128 sequences, while reducing completed copying by 39.70% at 128. The throughput benefit decreases with concurrency, and we do not extrapolate beyond the evaluated range.

We also added a 32-request, eight-task LongBench evaluation with 4–512 generated tokens, including eight requests exceeding 400 tokens. Placement updates remain active throughout, without an evaluation-time residency freeze. FRAME performs within 0.08% of fixed adaptive and 0.53% of recent frequency, with slightly less copying. These measurements establish near-equivalent performance under longer continuous generation, not universal throughput gains.

Finally, we clarified the residency-freeze configurations and added a faster-local-memory sensitivity analysis, explicitly distinguishing the latter from DDR6 hardware measurements. These results appear in Section 4.1 and Appendices A.2–A.3, B.2, and B.4.

Overall, the revised experiments establish two complementary results. First, bounded admission provides the principal system-level benefit by limiting migration traffic that competes with direct CXL expert reads. Second, the new matched-cap comparisons show a repeatable 2.45–2.66% throughput advantage for the complete FRAME policy over simpler bounded-admission alternatives under the evaluated stable and shifted workloads.

We have also narrowed our claims where the evidence is less conclusive, particularly for automatic selector generalization and long-generation speedups. We respectfully hope that these targeted experiments, revised claims, and explicit limitations address the central concerns and assist the reviewers and Area Chair in assessing the revised submission.

AuthorsOpenReview ↗

Summary for the AC and reviewers: each concern and where the revision answers it (1/2)

Summary for the AC and reviewers: concerns addressed in the revision

Summary of Revisions and Additional Evidence

We thank the reviewers for their careful assessment and constructive feedback. Their comments have helped us strengthen the experimental evaluation, clarify the contributions of individual mechanisms, and better define the scope of our claims. We have revised the manuscript accordingly, with earlier changes highlighted in blue and the latest additions addressing Reviewer ygze's follow-up highlighted in green. We summarize the main revisions and supporting evidence below.

1. Direct comparison against simpler bounded-admission policies (ygze).

To address the reviewer's remaining concern, we added direct comparisons between FRAME and recent-window frequency under identical replacement caps (2 or 4), memory budgets, eviction rules, current-generation protection, and execution settings. We evaluate both stable and shifted routing workloads using the 80B model and a 10 GiB DDR5 expert pool.

Each configuration includes six paired repetitions, three in each execution order. FRAME achieves 2.45–2.66% higher decode throughput across these comparisons and is faster in every paired repetition. All reported paired 95% confidence intervals exclude zero (November 16 follow-up). At cap2, completed copying differs by only 0.17% and 0.02% on the shifted and stable workloads, respectively. Thus, the incremental throughput improvement cannot be explained simply by reducing total migration volume.

These matched-cap experiments provide direct evidence of the full policy's additional value over a simpler bounded-admission reference. They do not, by themselves, isolate the independent contributions of ranking and mode selection. The results have been added to Section 4.4, Table 2, and Appendix D.1, Table 25.

2. Independent contributions and revised system claims (ygze, ueu7).

We added controlled ablations separating expert ranking, bounded admission, online feedback, and eviction protection:

  • Ranking: Adaptive ranking improves throughput by 1.15% over recent-window frequency with admission and eviction fixed.
  • Admission: Bounded admission improves throughput by 16.72% with window-frequency ranking fixed. Separately, fixed cap4 improves throughput by 18.89% and reduces copying by 52.05% over unrestricted admission with adaptive ranking fixed.
  • Online feedback: Eq. (8) adds 0.10% throughput and reduces copying by 2.10% beyond fixed cap4.
  • Eviction protection: Current-generation protection improves throughput by 4.77% and reduces copying by 38.38% under bounded admission.

These are controlled, conditional effects and are not additive. The results identify bounded admission as the principal throughput contributor, while adaptive ranking and eviction protection provide additional, workload-dependent benefits.

We have accordingly refocused the Abstract, Introduction, methodology overview, Section 4.4, and Conclusion on separating expert ranking from physical DDR5 admission. We no longer attribute the overall performance improvement primarily to adaptive prediction or automatic mode selection. Detailed comparisons are reported in Appendix D.1.

3. Strengthened baselines and fair comparison scope (joe3, ueu7).

We strengthened the CPU–CXL adaptations of ProMoE, ExpertFlow, EAMC, and Fiddler by incorporating their key prediction, scheduling, replacement, and reuse mechanisms. Online prediction overhead is included where applicable. FRAME achieves 13.31% and 10.87% higher throughput than the strengthened ProMoE and ExpertFlow adaptations, respectively, in separate matched comparisons.

We also added TPP and MEMTIS tiering-policy adaptations. FRAME improves post-shift throughput by 31.69% and 26.10%, respectively, on the same CPU–CXL backend.

The revised paper explicitly distinguishes mechanism-level CPU–CXL adaptations from complete reproductions of the original GPU systems or OS kernels. We document the implemented mechanisms, controlled configurations, and comparison limitations in Sections 2 and 4.1–4.2, Appendix A.1, and Appendix B.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.