- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FRAME, an expert placement system for MoE inference in CPU–CXL environments. In this setting, CPUs can directly execute CXL-resident experts without migrating their weights to DDR5. However, direct CXL access and expert migration compete for the same CXL bandwidth, making efficient residency management essential. FRAME combines three mechanisms: (1) online adaptive ranking based on recent expert activations and routing information, (2) a routing-aware selector that switches between frequency-based and adaptive rankings, and (3) bounded admission and residency management to control migration traffic. A key design principle is to separate expert ranking from actual DDR5 admission so that changes in ranking do not immediately trigger migrations. The authors evaluate FRAME on a real CPU–CXL server using Qwen3-Next-80B, Qwen3-30B, and OLMoE-1B-7B. Following a routing shift, FRAME achieves 17.09 tokens/s on the 80B model, outperforming the strongest adapted external baseline, Pre-Gated, by 17.6%. It also achieves 16.92 tokens/s on 24 held-out LongBench requests. Additional analyses examine the relationships among ranking accuracy, ready recall, migration volume, and CXL read traffic.
Critical strengths
Strength 1: Empirical Characterization of the Migration–Residency Trade-off in CPU–CXL Systems An important strength of this paper is its characterization of the costs and benefits of expert migration on real CPU–CXL hardware. Because CPUs can directly execute CXL-resident experts, migration becomes an optional optimization whose cost must be amortized through subsequent expert reuse. The paper identifies bandwidth contention between direct CXL reads and expert migration as a key system-level optimization problem. In particular, the break-even microbenchmark in Appendix A.2, the routing-dependent throughput and migration analysis in Figure 4, and the frequency-core sensitivity analysis in Appendix D.3 quantitatively demonstrate this trade-off. These findings provide valuable insights into when expert migration translates into actual performance improvements, beyond simply improving cache hit rates. I consider this an important empirical contribution that supports acceptance.
Strength 2: Controlled Experimental Design and Detailed Validation The paper makes a substantial effort to evaluate the proposed placement policy under controlled experimental conditions. As described in Section 4.1 and Appendix A.1, the compared policies share the same CPU–CXL execution engine, model, NUMA binding, copy path, and memory budget. The dedicated-core control in Appendix A.1 further verifies that contention between the migration worker and inference threads does not determine the main performance ordering. Appendices B.1 and B.3 provide repeated measurements and paired statistical comparisons. Appendix B.2 additionally checks output equivalence between the modified execution engine and a reference implementation, although the authors appropriately acknowledge that this does not establish token-level identity or task accuracy. While these controls do not constitute complete reproductions of the original baseline systems, they provide a useful basis for evaluating the relative effectiveness of different placement mechanisms under the same CPU–CXL environment. The explicit documentation of baseline adaptations, implementation differences, and experimental limitations further strengthens the transparency and credibility of the evaluation.
Critical weaknesses
Weakness 1: Insufficient Isolation of the Independent Contributions of Ranking and Admission The central contribution of FRAME lies in separating routing-aware expert ranking from bounded DDR5 admission to control migration traffic. However, related ideas such as activation-aware expert caching and residency management have been explored in prior work. For instance, MoE-Infinity and ProMoE investigate activation-aware caching and predictive expert management, respectively, while FluxMoE addresses expert residency management in a different execution environment. Furthermore, Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU–NDP Systems considers related problems involving CXL-based expert execution and workload-aware placement, although its execution architecture differs from FRAME's CPU-only setting. These prior studies do not invalidate FRAME's contribution. In particular, directly executing CXL-resident experts on CPUs introduces a different cost structure from conventional GPU offloading, potentially motivating specialized placement policies. Nevertheless, the paper should more clearly establish the additional benefits of its specific ranking and admission mechanisms over simpler adaptive caching approaches. Section 5.1 and Appendix C compare predictor recall–turnover characteristics, Appendix D.1 analyzes online feedback, and Appendix D.3 extensively examines the impact of the frequency-core size on migration traffic. However, the individual end-to-end contributions of the adaptive ranking signals, the admission bound in Eq. (8), and the engine-side eviction hysteresis remain insufficiently isolated. Although online feedback reduces migration traffic, its measured throughput improvement is relatively small, reaching only 0.075 tokens/s in the 10 GiB configuration. Consequently, it remains unclear how much additional benefit FRAME provides over a simpler recent-frequency ranking with the same admission controller, or over an adaptive ranking with a different admission policy. This is not merely a request for additional experiments. It directly affects the assessment of the proposed controller's independent contribution and whether its design complexity is justified.
Weakness 2: Limited Evidence for the Benefits and Generalizability of the Routing-Aware Selector One of FRAME's central mechanisms is a routing-aware selector that chooses between frequency-based and adaptive rankings according to changes in the routing distribution.
Figure 3 demonstrates that this mechanism can respond effectively to a routing shift. However, Figure 3(b) and Table 6 show that fixed adaptive achieves higher post-shift throughput than FRAME (17.31 vs. 17.09 tokens/s). Furthermore, Section 4.2 reports that FRAME maintains throughput within 0.3% of fixed adaptive over the full stream while reducing completed migration by approximately 7.0%. This is a meaningful reduction, but it also indicates that the selector's measured benefit primarily lies in migration efficiency. FRAME does improve post-shift throughput over fixed frequency, so these results do not imply that the selector is unnecessary. The main concern is whether this throughput–migration trade-off remains beneficial under more diverse routing dynamics. The main 80B experiment considers a transition from a stable workload to a shifted workload. In contrast, Appendix B.5 reports that the 30B model remains in adaptive mode throughout the experiment. Appendix E.2 further explains that repeated candidate resets in the 80B domain-drift experiment prevent the controller from returning to frequency mode. Therefore, while the ability to switch from frequency to adaptive ranking is demonstrated, the effectiveness of the reverse transition and its contribution to migration reduction are less clearly established. Additionally, the routing-shift threshold in Appendix A.4 is calibrated using a preceding pass over the same 14 request identities used in the main evaluation. The authors provide threshold sensitivity analyses and held-out experiments, so this observation does not invalidate the reported results. However, the generalizability of the threshold across model geometries and routing distributions remains an open question. For example, routing distances in the 30B experiment consistently exceed the threshold. As the authors explain, this may reflect the absence of a stable frequency regime for that model rather than incorrect selector behavior. Nevertheless, the current evidence does not clearly establish how reliably the mode-selection mechanism operates across different routing conditions. Overall, the experiments demonstrate that FRAME performs well under several evaluated workloads, but they provide limited evidence of the general robustness of its automatic mode-selection mechanism.
Questions for authors
*Q1. Independent Contributions of Ranking, Admission, and the Controller FRAME presents the separation of ranking and admission, together with bounded migration through Eq. (8), as a central design principle. Appendix D.1 evaluates the effect of disabling online feedback, but does not independently isolate the contributions of ranking-weight updates and the admission bound. Are results available comparing (1) a simple ranking such as recent-window frequency, (2) the full adaptive ranking without the feedback-based admission bound in Eq. (8), and (3) the complete FRAME controller, under the same execution environment? Additionally, does the benefit of bounded admission remain when the engine-side eviction hysteresis is held fixed? Such comparisons would help establish the independent contribution of the proposed ranking–admission separation and justify the combination of multiple control mechanisms.
Q2. Selector Robustness and Threshold Generalizability Figure 3 shows that FRAME maintains throughput comparable to fixed adaptive while reducing migration traffic following a routing shift. However, Appendix B.5 reports that the 30B model remains in adaptive mode throughout the experiment, while Appendix E.2 indicates that repeated candidate resets prevent a return to frequency mode in the 80B domain-drift experiment. Are there evaluated workloads where the selector successfully transitions from adaptive back to frequency mode and reduces migration traffic? Could the authors also provide per-request mode decisions for the held-out LongBench evaluation in Appendix B.4, ideally together with the corresponding throughput and migration volume? Furthermore, given that the threshold in Appendix A.4 was calibrated on the 80B workload, what evidence supports its applicability across different model geometries without additional tuning? These results would help assess whether the selector can consistently select and maintain appropriate residency policies across diverse routing distributions.
Q3. Generation Length and Evaluation-Time Residency Freeze The main experiments in Section 4.1 use a 32-token generation cap, while the authors note that actual generations in the LongBench experiments frequently terminate around 32 tokens despite a 120-token cap. What motivated the choice of these relatively short generation lengths? In particular, was this setting intended to match the evaluation protocols of prior baselines, or was it selected for other experimental or system-level reasons? Appendix A.3 also explains that the reported FRAME configuration includes an evaluation-time residency freeze after 30 issue calls, which is not part of the actual deployment rule in Table 16. I understand that this freeze affects only the last few tokens in the reported experiments and that the corresponding ablation does not show a statistically distinguishable throughput difference. Nevertheless, longer generations may alter the amortization of migration costs, the evolution of routing distributions, and the effectiveness of residency adaptation. If measurements are available for longer generations using the original deployment rule without the evaluation-time freeze, could the authors clarify whether the relative throughput and migration trends remain consistent? This question is intended to clarify the scope of generalization supported by the current evaluation rather than to request a comprehensive evaluation of production serving scenarios.
6 comments
Official Comment by Authors(2/3)
W2/Q2. Automatic selector, threshold transfer, and reverse transitions. The original post-shift result (FRAME 17.09 vs. fixed ADAPT 17.31 tok/s) does not show a throughput advantage over always-ADAPT. Across the original stream, FRAME stays within ~0.3% of fixed ADAPT while reducing copies by ~7%; its demonstrated value is migration efficiency.
We froze the entry threshold (0.770115) and tested it on 64 unseen LongBench requests across two independently calibrated overlap regimes (Appendix A.4, Table 7): HotpotQA/QMSum (Jaccard 0.127) and GovReport/MultiNews (0.473). Each pair has A4–B4–A4 and B4–A4–B4 streams plus same-task controls. Residency persists within each sequence; the selector receives no task-boundary signal. We use official generation caps, natural EOS, and both execution orders.
| Overlap | Phase | Fixed FREQ | Fixed ADAPT | FRAME |
|---|---|---|---|---|
| Lower | Initial | 16.292 | 17.055 | 17.195 |
| Lower | Shifted | 15.690 | 17.031 | 16.997 |
| Lower | Returned task | 15.479 | 17.079 | 17.056 |
| Higher | Initial | 17.886 | 17.813 | 17.793 |
| Higher | Shifted | 17.149 | 17.634 | 17.594 |
| Higher | Returned task | 17.193 | 17.598 | 17.572 |
Throughput in tok/s; phases denote workload changes, not mode switches. Pair Jaccard differs from the selector's request-level displacement score.
Across complete lower-/higher-overlap streams, FRAME achieves 17.030/17.691 tok/s, vs. 15.842/17.512 for fixed FREQ and 17.020/17.714 for fixed ADAPT.
| Overlap | Throughput vs. FREQ | Copies vs. FREQ | Throughput vs. ADAPT | Copies vs. ADAPT |
|---|---|---|---|---|
| Lower | +7.50% | +64.68% | +0.06% | −0.49% |
| Higher | +1.02% | +115.08% | −0.13% | −0.12% |
FRAME-relative changes; positive copy changes mean more migration. Completed copies include prefill.
After the first transition, throughput gains over FREQ are 9.36%/2.40%, with 83.32%/123.04% more copies. This is a throughput–migration trade-off, not migration reduction. Against fixed ADAPT, FRAME preserves nearly the same operating point, with 0.49%/0.12% fewer copies and negligible throughput differences. These 64-request results differ from the ~7% fewer copies in the controlled 14-request stream and do not establish migration reduction in both overlap regimes. FREQ is slightly faster in the higher-overlap initial phase; later phases favor ADAPT.
Per-request decisions and limitations. The earlier 24-request LongBench expansion used an adaptive-only build and cannot evidence selector switching. In the new tests, all eight sequences enter ADAPT at cold start (0.862–0.897 displacement, above threshold); all recorded modes remain ADAPT. Of 104 post-first decisions, 82 reset candidate-frequency state; only 14/96 valid paired comparisons meet the return-hit condition, with no consecutive qualifying wins (Appendices A.4, E.2). This supports selected-state performance, not bidirectional-switching generalization.
| Overlap | Sequence | Modes in request order (both orders) |
|---|---|---|
| Lower | A–B–A | D D D - D D D D D D - D |
| Lower | B–A–B | D D D D D - D D D D D D |
| Lower | A only | D - D D |
| Lower | B only | D D D D |
| Higher | A–B–A | D D D D D D D D D D D D |
| Higher | B–A–B | D D D D D D D D D D D D |
| Higher | A only | D D D D |
| Higher | B only | D D D D |
D = ADAPT; “−” = unrecorded mode.
Two controls delimit threshold transfer. On unseen GSM8K→MMLU→HumanEval requests, post-change displacement is 0.628–0.687, below threshold, so FREQ persists despite task changes. On 40 shared 30B/80B requests, the same thresholds work without retuning and both geometries retain ADAPT: 18.365/17.624 tok/s vs. 18.135/17.649 for fixed ADAPT (Appendix B.5). This supports threshold transfer and selected-state performance, not universally correct switching: 30B displacement (0.79–0.86) remains above the entry threshold. The separate candidate-reset freshness threshold is also geometry-dependent (Appendix E.2), so these runs do not establish cross-geometry reverse-transition reliability.
For ADAPT→FREQ, a development return-only control switches at request 9, reducing copies 4.69–4.71% at 0.19–0.32% throughput cost. An eight-request refined-selector test enters ADAPT at request 6 and returns to FREQ at request 8, reducing copies 8.53–8.57% at 0.05–0.95% throughput cost (Appendix D.4). On repeated GSM8K, fixed FREQ beats fixed ADAPT by 4.35% throughput and 44.00% fewer copies. These are refinement/control results, not evidence that the original frozen rule independently performs reverse switching; the Limitations section now states this explicitly.
Official Comment by Authors(3/3)
Q3. Generation length and evaluation-time freeze. The 32-token cap was chosen to constrain decode-work variation in the controlled shift comparison, not to match another system's evaluation configuration. Importantly, the main 14-request stream does not freeze residency updates. A 30-issue-call freeze applied only to earlier pilot and 24-request LongBench settings; the distinction is now explicit in Section 4.1 and Appendix A.3.
We added a new 32-request, eight-task LongBench evaluation using official generation caps, natural EOS, and the deployed continuously updating placement rule without a freeze (Appendix B.2, Table 10). Actual outputs range from 4 to 512 tokens, including eight requests exceeding 400 tokens. Across both execution orders, FRAME, fixed adaptive, and recent-window frequency match in exact tokens, stopping conditions, output text, and official task scores.
| Policy | Generation (tok/s) | Completed copies (GiB/block) |
|---|---|---|
| FRAME | 17.028 | 426.999 |
| Fixed adaptive | 17.042 | 427.732 |
| Recent-window frequency | 17.118 | 429.043 |
FRAME is 0.08% slower than fixed adaptive and 0.53% slower than recent frequency, with 0.17% and 0.48% fewer completed copies, respectively. Throughput covers decoding; whole-request copy totals include prefill, so these small differences should not be interpreted as a decode-only migration-amortization measurement. The near-tie is consistent with recent frequency already capturing much of the reusable demand in this workload; it does not demonstrate persistent speedups for every long generation. Separate shifted-routing controls show where adaptive ranking helps (+9.36%/+2.40% over cumulative frequency after transitions), and the isolated admission comparison shows a larger +18.06% throughput / −46.15% copying relative to unrestricted recent-window admission. These results establish continuous-update behavior at longer output lengths while identifying the workload conditions and limits of the benefits.
Official Comment by Authors(1/3)
We thank the reviewer for the detailed technical assessment. We agree that the key questions are whether FRAME's mechanisms have separable system-level value, when automatic mode selection is useful, and whether results hold under longer generation without evaluation-time shortcuts. We added matched controls and revised Sections 3.3, 4.1–4.4, 5.1–5.2, Appendices A.3–A.4, B.2/B.5, C.1, D.1/D.4, and Limitations. Revisions are highlighted in blue.
W1/Q1. Independent contributions of ranking, admission, and eviction. Activation-aware caching and prediction are established in MoE-Infinity, ProMoE, FluxMoE, and related work; context-aware CXL GPU–NDP inference explores a different near-memory path. FRAME addresses a narrower setting: on a CPU-only CXL host, nonresident experts can execute directly from CXL, so DDR5 migration is optional and costly because copies contend with direct reads. We therefore separate which experts to rank from whether to admit DDR5 copies, rather than attributing the overall gain to a novel predictor.
First, a three-arm end-to-end control uses six unseen GSM8K/WMT16 requests (80B model, 10 GiB pool, both arm orders). Model, outputs, token counts, stopping conditions, copy queues, ranking cadence, CPU bindings, and eviction rules are matched (Appendix D.1, Table 24).
A vs. B isolates the composite adaptive ranker: +1.15% throughput with slightly fewer copies. B vs. C holds ranking fixed and isolates bounded admission: +16.72% throughput. A vs. C yields +18.06% throughput and 46.15% fewer copies. “Unrestricted” removes only the policy replacement bound; all arms retain the same physical 10 GiB capacity. These conditional contrasts must not be added.
Second, matched studies hold adaptive ranking fixed to isolate the four-replacement ceiling, Eq. (8) hit-feedback refinement, and eviction protection (Appendix D.1, Tables 21–22).
The fixed ceiling explains most of the throughput gain; Eq. (8) further reduces traffic, tightening the cap in 720 of 6,809 admission opportunities. The earlier feedback-on/off difference of only 0.075 tok/s does not measure removing admission bounds: these new controls distinguish feedback refinement, a fixed cap, and unrestricted admission. A 2×2 admission-by-eviction control finds admission gains of 19.91% with and 14.96% without protection. It changes only current-generation protection under same-kind LRU; reader safety, capacity, and copy queues are retained, while the policy-side retention bonus and legacy page-migration trigger are disabled. Protection matters more under bounded admission, so its effect is not an independent additive gain.
Third, predictor-level evidence shows why additional ranking signals help, but should not be mistaken for large system speedups. In causal replay over 36,864 decode-layer events at P=107 (Appendix C.1, Table 18), composite adaptive/window candidate recall is 86.43%/73.91%. At equal logical migration (2.040 GB per 1,000 events), admitted recall is 81.73%/72.40% (+9.33 points). Yet the matched end-to-end gain is only 1.15%: resident copies may serve experts no longer in the candidate list, new copies may arrive too late, and missing experts execute directly from CXL. Freezing vs. updating ranking weights changes native throughput by only −0.24% to +0.50% across four paired controls. Thus, the composite adaptive ranker improves ranking quality, but its measured end-to-end contribution is modest; these controls do not assign separate speedups to every routing, transition, or gate signal. The principal benefit comes from migration-limited admission, with a conditional benefit from eviction protection. This decomposition—not prediction accuracy alone—motivates the controller's components.